What's the use of skill points other than granting access to perks? Do I get more damage output when fighting with one handed weapon if my skill is high, even if I don't use perks in that tree?
Yes, every point of skill gives you a little more benefit even if you do not use the perks. For instance, every point of speech craft improves merchant prices by 1%. Every point of destruction lowers spell costs etc. Even if you don't put perks into the skills, its still highly beneficial for you to have high skills. Not to mention it also gives you more experience for leveling.