Now that this question is answered, I'd like to go back to that cave. I already have a ship (just set my foot on the third island, after you join the pirates. Can I go back somehow? If so, how, and if I can't do it yet, when?
You can fast travel via the start menu: