Increasing your skills makes you more effective in their use. The way in which this happens vary by skill.
In the case of the various schools of magic, it comes in the form of reducing the Magicka cost of higher level spells, allowing you to cast more powerful alternatives to your earlier weaker ones.
Most of the weapon/armor skills manifest as direct damage/damage reduction increases that you'll see when you attempt to equip the related items. .
Increasing crafting skills increases the potency of the product created. More powerful potions/enchants, and better smithing upgrades at a workbench.
Increasing Speechcraft and Sneak, and Pickpocket improves your chance of success when using those skills, and, in the case of Speechcraft, improves the prices you'll get from merchants.
Increasing lock picking increases the size of the 'Sweet Spot' when picking locks and increases the number of failed attempts before breaking a pick.