Light Armor provides mainly Bonuses to Magicka and Spell Penetration, Medium Armor to crit and Stamina regeneration. When going with a setup that uses instant casts and Dual Wield to kill your foes, which one of the two would be more beneficial? Seeing that a Mage has not a lot of Stamina regenerating abilities ( I am Breton too, to make things worse ), I wonder which Armor would be better to use.
There's no definitive yes or no here, since it's mostly personal choice.
Think this also depends on how intensive you're using your weapon skill over magic. The more magic you're using, the more damage you'll lose by using medium armor. On the other hand, using medium armor will lower your spell efficiency.
I think the best question to ask is: Are you running low on stamina or magicka (more often)?
If you're lacking magicka, grab one or two more pieces of light armor. The same for stamina and medium armor. This will force you to spec into both armor skill trees, but I'd say this is OK since you won't use all passives of both lines anyway.