After I went to the warrens in Markarth during the quest Forsworn Conspiracy, I have to brawl with Dryston when I get back out. In theory nothing bad should happen if I stick to a fist fight, but all the guards turn hostile almost right away. Is there a known bug concerning this or am I doing something wrong?
I already parked my follower inside the warrens to avoid aggression from him. I also have no active atronarch or similar support. There is currently no bounty on my head. What else could trigger such a reaction from the guards?

