I played through the entire game, collecting gem stones (Ruby, Diamond, Sapphire etc.) as I went, and never found any particularly useful purpose for them beyond the *very occasional* item crafting and upgrading purposes.  I kept expecting them to be useful someday, but they really weren't.  Certainly no need for a huge pile of them anyway.

Sell off all but 10 (just in case) of every gem type you have and never look back.

>! There's a Gerudo woman in Goron City who buys gems at better than market price, if you're interested in maximizing the profits, but she only buys one type at a time.