Assuming the mechanics are the same as they've been across the series...
The major difference is that axes do bash damage while swords do cut damage. One thing to remember is that with exceedingly rare exception, there's no such thing as a "neutral" damage. All damage is of an element, either magical or physical. And so enemies have varying tolerances - some things are easier to slash than to smash, and vice versa. For example, rocky or shell type enemies tend to either be weak to or take normal damage from bash type, but would take reduced damage from cut and stab type.
To that point, if I recall correctly, axes and staves are the only default-to-bash type weapons in the game. Whilst swords, whips, and katanas are the default-to-cut type weapons. As such, be cognizant of your desired elemental coverage - axes are pretty much the major source of bash damage for standard physical fighters, barring the traditional combat medic builds.
Past that, the other main difference is in the skillsets of the class in question. Usually the attack sets of the two skill trees for any given multi-weapon class tend to present two wildly different attack strategies. As such, in some ways, skill access is a more deciding factor than raw attack power, at least late game.