What I like to do is when you get the first house, the storage is (seeminly) infinite in the chests, so I have chests categorized by item type (useless, smithing/chanting, alchemy, etc) and when my carry is getting too high, I empty my stocks into their respective places, and every once in a while I sell, smith, enchant, etc. to get rid of all the stuff I don't use.
On any outings, I carry weapons, armor, arrows (they require no weight) along with proper potions (I don't need to have a potion to improve smithing while dungeoneering).
While looting I typically pickup anything with a decent value (at least 100+) and then I just drop generic weapons (I don't need to sell 50 Ancient Nordic swords). I tend to keep any profession material + weapons with enchants + high value items.
I'm more of a warrior spec so even with heavy armor + shield + a spare weapon for the occasional DW and a proper bow (though, I just moved on to Bound Bow) and I have usually around 200 extra carry weight to use for whatever I pickup.
It all comes back to play style in the end but generally assume most things have some value but keep in mind the things you can drop for other, more worthwhile items.
There is a perk once you get high enough speech craft that will allow you to sell anything to anyone no matter what kind of store it is and a separate perk to allow you to sell stolen items as well.