If you have the default aim helper enabled, it's pretty easy to target the small animals, though the really tiny ones can be tricky.
For rabbits, just shoot them with a varmint Rifle. It only takes one shot and doesn't harm the quality. I simply shot at every rabbit I saw from my horse to fulfill the Horseman and the Master Hunter challenges about rabbits. If you have the Horseman challenge about killing 5 rabbits from your horse already, you can check the quality before shooting, simply targeting the rabbit with your Varmint Rifle will show if it's a three star or lower pelt.
I had issues with squirrels, the Varmint rifle seems to degrade pelt quality. Using small game arrows works, I'm not entirely sure if you have to hit them in the head or not.
I haven't used it myself for this purpose, but you can of course also use Dead Eye to help with killing animals.