While back tracking old quests, I stumbled upon this place in Velen:
The guards won't let me in, and there's no real dialog with them. The whole place is surrounded with an unpassable fence.
This is the location in the World Map:
How can I get inside? I have completed all the secondary quests and contracts already (After taking the screenshots), and I'm very far in the main quest, so pretty sure it's not something in there. (Final Preparations quest)