Short answer: spell-casting will get you through the game on normal difficulty just fine, but ultimately falls far short of physical weapons if you really want to push the envelope. I haven't played on master difficulty, so I can't comment there.
Long answer: was really long, snipped.
Medium answer: without recursively buffing alchemy and enchanting, if you max alchemy, enchanting, and smithing, plus get all appropriate crafting perks and weapon skill perks, a crafted and improved daedric sword can get over 300 damage per hit, while the best I can see from a maxed destruction spell is around 250 damage per hit.
(To do that, enchant a set of armor to add +100% alchemy and +100% smithing, then craft the best +smithing potion you can make with that armor, then use the armor plus potion to improve the sword. Then use the +40% weapon skill enchants on your regular armor to double or triple the improved weapon's damage. Note this isn't exploiting recursion in any way.)
Combine the extra damage with the fact that you can dual-wield the swords, and swords hit much more quickly than spells, and you've already got far more damage output with swords than spells. Then, use your alchemy to make a +weapon skill potion, and you can increase output even further for a minute at a time (again, no recursion exploits). Finally, throw in some sneaky-sneaky and hit 2k on a backstab, and mages can only dream of such damage.
If you want to go full out and exploit recursive synergies (which seems to be intended behavior, not a bug exploit), you can hit almost 1200 damage per hit, per sword. Combine these with the orc's berserker rage, and you can hit 2400 legitimately, with a sneak attack of over 14k. Not sure if it's possible, but if you can backstab with both swords at once, you'd be hitting nearly 30k in one hit -- something that would take 100 spells to match.
That said, none of this is at all required on normal difficulty, and many are considering it boringly-overpowered even on master difficulty, so just because physical weapons far overpower magic when maximizing character potential, it doesn't mean you can't play an effective pure-mage class on the game.
Source: the Bethesda forums.