I just found out what these gems were and that I will need to collect 20-25 of them total. I have 7 right now and I do not like the fact that they're taking up space in my inventory. I'd like to store them in my barrel at Riverwood. Is it possible to ditch them until I need them all?
If you try to store them it'll just say "quest item, cannot be removed"