I've done everything right, I beat the monkey, octopus, and myself. I just can't access the castle and the hole like I'm supposed to :/
You need to complete The Forest to move on the the Castle entrance and the hole.
An easy way to complete The Forest is to use the Monkey Wizard Staff and the Octopus King's Crown. it is recommended to have 200 health with a Monkey Wizard Staff or a Octopus King's Crown otherwise with 350+ health
More info here: