I played through the entire game, collecting gemsgem stones (Ruby, Diamond, Sapphire etc.) as I went, and never found any particularly useful purpose for them beyond the very occasional item crafting and upgrading purposes. I kept expecting them to be useful someday, but they really weren't. Certainly no need for a huge pile of them anyway.
Sell off all but 10 (just in case) of every gem type you have and never look back.
There's a Gerudo woman in Goron City who buys gems at better than market price, if you're interested in maximizing the profits, but she only buys one type at a time.