I'm not exactly sure of the math behind the "magic" attacks, but I do know they are quite powerful. Magic is extremely useful with very large groups of enemies, as you can knock them all back with one blow. Holding the magic button for only a short period of time will do almost no damage, but it gives you time to re-attack anyway you like.
Many areas in the game have the enemies attacking from a short distance away, but at a distance none the less. Use this to your advantage, as you will be able to charge your magic attacks for quite a bit of time before they are in attack range. A fully charged magic attack can kill quite a few enemies around you, if not all of them with one single blow.
This is countered by some groups having ranged weapons like rifles which will sometimes interrupt your charge.
To answer your question: Magic is very useful in Fable III, but it wouldn't benefit you to only use magic the entire time.
Ivo Flipse is correct in saying that you cannot "die" in Fable III. So if you want to only use magic, you will still be able to complete the game. You might get some scars on your character, but no one really cares about that.
Hope this helps.