There isn't a way to outright drop caps. You have to either use mods, console commands, or find alternate ways to store money.
As for mods, you can use this Nexus mod to turn bottle caps from an entity into an item. As the mod advertises, this allows bottle caps to be dropped or stored in containers.
Likewise, you can use console commands to drop your caps. Just be careful, entering negative numbers can crash your game, and dropping too many at once may have the same effect. You can read a little more from this Reddit post. The actual command is:
player.removeitem f <amount>
Lastly, while you cannot store bottle caps you can store legion money and NCR cash. Consider visiting a casino, investing all of your caps as casino chips and immediately ask the casino's cashier for payout in faction money. You should get a payout that is almost the same amount as caps (legion coins are worth a minimum of four caps, payouts of less than four chips are given in caps instead).
This faction money can be dropped, stored or even sold to vendors and can always be converted back into caps by reversing the process written above: convert faction money into casino chips, immediately turn in casino chips and request your payout as caps.
There are not many benefits to having your caps separated from your character - they weigh nothing and limit your ability to make purchases. But for the few occasions where all your caps are removed only losing three caps is preferable to losing thousands.