The other way to complete your bounty sets, including Gems, is to use the black market and near app to trade items, it seems to be very difficult to obtain the rare and scarce items simply by continuing to kill enemies in game. There are no specific enemies who have specific bounties, the rare and scarce items are just less likely to be dropped after a kill.
Unfortunately the near app and black market are not very easy to use, I found some useful information here on how to use the trade system.
The best way is to go to a forum and seek out players looking to actively trade bounties and friend them on PSN so you can trade with active players straight away as trades expire very quickly.