I read in a guide that they exist, but I'd like to know where to find them/what type of containers I'm looking for.
Thieves caches are located all over Skyrim. The Riften cache is located in the garden of the house you can buy there. In the back of the garden you should find there are some barrels, one of which is the Thief Cache appropriately named "Thief Cache" :)
The house is located at number 7 (top-left) on the map below if you don't know where it is.
Also keep a look out for this "shadow-mark", it's means there is a thieves cache somewhere nearby.
You can learn more about the Thieves Guild shadow-marks here.