And the very next evening I solved the problem.
Once you rescue the hermit, he will offer to train you. By beating him over and over (he isn't fighting with his real power, so you won't take damage and beat him and his fireballs with a single hit), you can gain the levels needed to beat the stage.
There are also two apprentices of the hermit hidden in the stage. Each will give you one of the hermits favourite foods and giving those to the hermit will make him stronger, thus allowing you to level up faster.
However, if and only if (this is what I failed to fully explore) you give him both of these food items, he will stop training with you at level 40. If, however you give him only one of the foods or neither, you can keep leveling beyond this, up to around level 60 where you'll start to run out of money to buy more time.
What I found was that giving him neither foods slowed the leveling up process so much, you didn't have time to get to level 60 before running out of money. But by giving him one of the foods, training untill I was level 60 and then (finding and) giving him the second food item, he would stop training me, challenge me and I was able to beat him.
And for completeness sake, my gear was Crimson Robe, Horned Helm, Spike Mace, Flash Shield and Bigfoot.