Short of downloading a Map Viewer to determine where the closest Crystal Hearts are, there's no easy way forward. Depending on how deep you explore and variance in luck, you can get streaks of crystal heart caverns and chests galore, or no interesting items sans torch pots.
That said, you definitely should keep exploring caves (and mining into new caves from existing caves wherever possible / you can find them). While exploring will obviously help your survival when you find new crystal hearts, it will also give you a chance to find silver and gold ore. Your priority should be a gold pickaxe (to mine Meteorite), but after that feel free to throw all your excess silver and gold into defensive equipment.
Better gear is like getting Crystal Hearts, because it increases the number of hits you take (though obviously your HP remains the same). As to finding Silver and Gold, Floating Islands tend to be full of them. Consider building a skybridge of some sort to find islands, but if you do, mind the Harpeys (typo intentional).