It looks like you're suffering a bug.
According to uesp
If the Markarth Guards do not warn you off after having investigated the Inn for Eltrys and you get stuck (Guards won't arrest you) when returning to him after finding out all you can about the Conspiracy, if you wander the streets and do meet the Guard who warns you off and then returning to the Temple, you will still get arrested (or attempted to get arrested).
So it looks like you need to talk to all the guards in the city.
Another possibility is this (I hope this isn't your problem - no way out):
If you enter The Warrens before your first talk to Eltrys in Shrine of Talos, a Nord named Dryston will approach you. He will tell you that "it's time you learned a lesson" and brawl with you. Once you beat him, he will NOT tell you that "N" is Nepos the Nose as you did not get this quest yet, this will lead to a BUG where you can not complete this quest 'find out who "N" is' once you read Weylin's Note (if you read it right after you find it by lock picking the Weylin's door, it will not start the quest, as you did not speak to Eltrys yet).