There are several other factors in play on how to do this, but OP is asking "given x, should I do y or z?"
The answer here will almost always be: try to fight them, or at least try to discourage them from pushing on you (poke or get another lane autopushing).
Given the circumstances, it is likely that a fair teamfight should end in a slight favor to the team with the baron buff. This would, in a fair-case-scenario end with one player (lets hope its not Jinx) on the other team surviving. Most of the time, teamfights are also slightly spaced out in time, which means they will have to choose between overextending or not getting an objective. Either way, you win.
Lets consider the other scenario; stay and defend as 5. Considering equal compositions, you will almost always lose at least half your turret for free, and at the same time give up control of your own jungle and sidelanes. Yes, your risk is probably lower while the baron buff is live, but you will certainly always lose something.
There are pros and cons and you can discuss the weighing of arguments here, but in the end, I will always choose an option that gives me a chance to gain an advtange.