In Batman Arkham Origins I beat the story but I'm having trouble with the challenges. Is there any way to do the challenges now that the game is completed?
You are able to do the challenges now that the game is complete, the same as you would during the normal campaign. HOWEVER, while the Shadow Vigilante and Gotham Protector tracks should be just as easy to complete as during the story, the Worst Nightmare Track is not. If you have not done the Worst Nightmare track of challenges during the story, there are a few that you will not be able to complete (specifically number 6 and number 14 I think).
You can replay a predator encounter on top of the GCPD building (where there's all the cops meander around), but other than there I don't believe there is anytwhere else to complete any of the Worst Nightmare challenges. To finish the track you will have to do it on New Game Plus or I Am The Night (or restart your entire game).