For an early, temporary solution, you can also go to Riverwood and solve Faendal's love letter crisis with Sven (talk to the lying jerk in the Inn) to get him to serve you as a follower for a while. Just tell him, "I need your help, follow me." Trade your stuff to him, and he will carry it for you for as long as he serves you (or go back to his life, still holding your stuff, until you ask him to serve you again--if you tell him to wait for you somewhere too long, or tell him to part ways). He works at the mill in the day, hangs out at the Riverwood trader in the morning, and is in his house at night.
Just be careful not to accidentally hit him with melee, magic, or arrows, as he dies like a chump, and only you can kill him permanently--and then you'll have to carry your /own/ stuff!
There are plenty of other, better followers around. Faendal is just the earliest and easiest for a new character to unlock.
Having a follower around is useful--they can hold onto a lot of your stuff, effectively doubling your carrying capacity. You can also outfit them with weapons and armor, but watch out--the dumb elf doesn't have the heavy armor or two handed weapons skills, but he /will/ equip those items if you let him carry them!