... aaand it's NOT gone, as it turns out!
While, yes, this is callback to the Margarita Ville episode, you can get your money back! After you gain the ability to shrink, you can go behind the counter through a mouse hole and hit the banker with your melee weapon to start combat. Once you beat him, his body will have an amount of coins on him equal to 150% of your total investment.
While I was not expecting to get my initial $20 investment I made near the start of the game back, when I first beat him he had $30 on him. So I quickly reloaded the auto-save, invested another $20, beat him, and his body had $60!
The upper limit on investments is $60; he will take more, but it will not increase the loot beyond $90. This seems to be roughly consistent with the number of variations in his investment spiel, so once he proclaims the you are an experienced and sophisticated investor, do not invest any more money. The banker will not respawn after you have beaten him, so this is a one time withdrawal.
As @LessPop_MoreFizz pointed out in the comments, while the base value of his loot is 150% your total investment (for a maximum of $90), you can further increase it by equipping armor and patches that increase money rewards. So by equipping the full pre-order assassin set, and a bunch of magnet type patches (Small Magnets, Buckyball Magnets), you can increase the return on your investments to $150!