I found my first "Tempering Blood Gemstone". I want to use it immediately but I don't know if I am going to find a lot more, if it is something I have to be careful with, or if they have unlimited uses and using it doesn't hurt me at all.
@AdamP makes a good point about being able to remove them from a weapon at your leisure; but to answer the question itself:
Yes, they're fairly easy to come by. I'm not that far into the game yet (3rd boss), but in the level preceding (Old Yarnham) I found about 6 on a couple of playthroughs of the level. They can also drop from enemies, albeit rarely.
Also, they have to be the right kind of Blood Gem Stone to be able to fortify it onto a weapon - so you mightn't even be able to use it yet.
Blood Gemstones are both easy to come by AND have unlimited uses, so do not be afraid to use them.
Blood Gemstones can be placed in and removed from equipment at the workshop with no penalty. So feel free to stick your best ones in whatever weapon is your current favorite. You will be able to change them out in the future.
Yes, blood gems are fairly easy to come by. Radial, Triangular, and Waning Blood stones can only be attached to right-hand weapons and Circle to left-hand weapons whereas, Droplet Blood Gemstones can attach to any slot regardless of shape. You can also consume some special items for unique Blood gems.