I've completed True Vault Hunter mode playthrough and I have tons of sidequests left and all the four of the DLCs. Problem is, I keep gaining XP and already a half way to level 51. The game has Ultimate Vault Hunter Pack 1 installed, does this mean that I'll end up in an unbalanced game where I either stomp over enemies who can't really damage my health pool or where I can't damage then because of the level 50 gear?
You can't stop leveling up.
As to your specific problem, monsters won't level above 50 (51-53 for bosses). You'll overlevel them somewhat, but I wouldn't worry much about that - AFAIK, sidequests won't take you higher than 53-54, and while this will make things easier (+13% HP per level), that won't make TVHM exactly a cakewalk.
You may have a problem when you enter UVHM, though, as you won't have lv.53 gear.
I would not worry about over leveling them, those monsters are monsters at somepoints. It still won't be easy, you'll just have an edge perhaps. When you enter the next play through of the game also the enemies adjust to your level every time you level up, that way you never have to be worried about being over leveled against them, you may even wish you were at some points...The equipment won't reach over level 50 until you go to the next play through though, not 100% possitive though since I was already max level by the time the add-ons came out. Also the enemies will be over level 50 at some points, pretty sure they will hit 52 before the next play through. P.S. get you a sandhawk