I'd say go for it.
You might need a decent amount of health potions, but you have enough maximum health and a good armor to withstand his attacks. There's really not a lot you can do to dodge his attacks without Rocket Boots, especially the arms.
Obviously, ranged weapon are best. I recommend the Demon Bow and the best arrows you have.
As a melee backup weapon, something with a very long reach is preferable (e.g., the Phaseblade or the Blade of Grass), but you should really pack enough arrows to make that unnecessary.
If you decide to brave the Underworld before doing battle with him, replace the Demon Bow with the Molten Fury and add the Flamarang as a backup weapon. I did so too, but I found surviving in the Underworld long enough to get a decent amount of Hellstone a lot more difficult than the Skeletron encounter. After the Dungeon, it might be more manageable if you found a Blue Moon.
Although, risking your hide just long enough to gather the 150 Hellstone required for the Molten Fury might just be worth it, since it does a lot of damage — more than double of what the Demon Bow does. Plus, you won't have to waste any torches on Flaming Arrow crafting.