Zodiac alignment can augment base damage either positively or negatively by 25%, 50%, or not at all. As such, assuming two characters have the exact same base damage despite differences in equipment, class, level, or otherwise and are attacking the same enemy, the character with poor zodiac alignment can do as little as 1/3 the damage of a character with good zodiac alignment.
x = base damage
1.5x = damage with best zodiac alignment
0.5x = damage with worst zodiac alignment
However, while people often confer all numerical anomalies to the complicated Zodiac system, there is a second, much more likely reason for your discrepancy: Katana damage (Samurai) is a function of Bravery while Ninja Sword damage (Ninja) is a function of speed.
Katana ........... damage = [(PA * Br) / 100] * WP XA = [(PA * Br) / 100]
Ninja Sword ...... damage = [(PA + Sp) / 2] * WP XA = [(PA * Sp) / 2]
Actual damage is typically
PA * WP along with a single modifier. You can see here that Bravery directly reduces your Samurai's damage unless you have exactly 100 Bravery. Ninjas, on the other hand, can very easily use equipment to increase their Speed to improve their actual damage.
If the large discrepancy you witnessed occurred only once or just occasionally, you could attribute it to Zodiac alignment, and therefore luck. If you're consistently getting poor performance from your Samurai, it has everything to do with his stats.
That being said, the Samurai class is very demanding. You need a character with high Bravery and you need gear to support both Physical (PA) and Magic (MA) attack power; Faith bears little weight beyond resistance rolls for a Samurai. If you focus on high PA your Katana Draws will suffer, and if you focus on high MA your physical strikes will suffer.
Source: Aerostar's Battle Mechanics Guide v6.5