Make sure you have applied Relict Oil on your silver-sword. Also, equipping Ekimmara (Damage dealt to foes regenerates Vitality) or Ekhidna Decoctions (Performing actions that consume Stamina regenerates Vitality) can tremendously help you in this endeavor.
Next, use the Quen sign on Geralt as frequently as possible. This debilitates the Caretaker's life-steals when he hits you. If possible, learn the Fast Attack, 'Whirl' to deal more damage to the Caretaker in his moment of weakness.
Wait, weakness? Yes. This boss has one weakness, or opening so to speak, to deal extra damage.
When the Caretaker begins charging his petrification attack, (as seen when he charges his shovel into the ground and a circle of blue light emits around him) use your crossbow to stop him midway, thus causing him to momentarily be stunned. Use this opportunity to attack him with 'Whirl'. Normal attacks are fine too.
When he starts to summon his spirits, don't fret. Focus your offence on the spirits themselves. Attacking the Caretaker would only lead you being flung further away and gives him ample time to regenerate. Precise, the spirits disappears after one or two blows. One last tip of advice is to practice patience.
The more you rush, the more mistakes you would likely make. Go in steady. Keep your wits about you. Don't rush when attacking and don't get greedy too. Hope this helps.
P.S: It took me half an hour to learn and counter his techniques. If I can do it, so can you.