The beauty of Skyrim, is that you can do whatever you want (of course there are consequences for some actions), but with that said you can take whatever route you want, if you find that at the lower level you are just running out of magicka, then grab a dagger (if you want to stick with small weapons).
It really comes down to how you want to play. Typically it is more difficult playing as a 'true' mage class (in most games, not just Skyrim) as the character usually wears light armor or no armor at all, you have to depend on your spells more and more.
Also the 'College of Winterhold' is just something you can visit, but you can pick up spells as you slowly work your way through the (very very) large world of Skyrim.
With all that said, there really is no right answer to your question, if you want to go to the college, go for it, i you want to invest in some armor / weapons, go for it. You can do what you like.
My character is a heavy armor wearing archer (I just fell in love with bows after awhile), plus I fall back to shield and sword when close quarters.