I would work on restoration to 60, to provide additional magicka regen, and cheap Close Wounds. The spells and perks beyond 60 are very much optional.
Alteration is indeed useful, as without damage reduction you will be very squishy at later levels. The entire tree is pretty good, with very useful perks - but it does need a lot of them.
Conjuration is useful as a means of providing a wall between you and enemies. However, it may not be to your playstyle and you can certainly do just fine without it.
Illusion is good. Invisibility is not perfect, and you can still be detected but you can use it to infiltrate at a short to medium range. The CC spells of Illusion require high investment in the relevant perks to remain effective end-game, but you can focus on a particular effect (e.g. calm) to save a few perk points. On the other hand, Aspect Of Terror also boosts your destruction fire damage by 10 points.
I would consider alchemy too, as one of the few ways to reliably boost destruction damage. (glowing mushrooms with nightshade are probably the most common ingredients that offer the effect).
Enchant reduced cost destruction equipment, and use remaining slots to boost magicka regen for use on the other schools. Magic resistance is useful, but with Alteration perks and any base Racial trait (are you a Breton?) one piece should be fine.
As mentioned by others, a character planner can help ensure you have enough perk points. Magic can be perk intensive!
Some choices, in response to a comment:
1) A perk planner spreadsheet by Shiloh hosted at SkyrimNexus
2) Graphical Perk Planner by PsychoHampster hosted at SkyrimNexus
3) There are also some excellent online ones, I believe one by IGN as well. But I use offline ones so I don't have links.