I'm afraid of beating him again because I don't want to change the difficulty yet. I'm worried that beating him more than once will further increase the difficulty.
No, there is only one difficulty transition you can make. Beating the Wall of Flesh again will not change anything further. In fact this is a general characteristic of all Bosses. The game only cares about whether or not you have beaten them at least once. You will likely want to beat them more than one time because they drop good items, and there are no downsides to doing so, except for whatever ammunition, potions, boss summoning items you use up in the conflict.
No, for instance, when you beat the wall of flesh for the fist time all monsters will increase in hp and damage (Like blue slime Pre-Hardmode has about 50 HP, and a Hardmode one has around 270 HP), but if you felt like you did not get the drops you wanted from the Wall Of Flesh, than you are able to fight him again indefinitely without changing the difficulty of your world. This applies to any boss ( though they never change the difficulty in the first place). Hope this helps.