So I understand that Garrosh used a time portal to go back Draenor before the Orcs got corrupted by the demon blood, but what I don't understand is, we killed him in SoO, which means he had to activate the time portal before, so how could we fight him if he wasn't even there, or was that like a duplicate of him or something?
Garrosh Didn't Die in Siege of Orgrimmar
Kairozdormu tweaked time and space with his "device" which allowed them to travel to an alternate reality (universe) in the not too distant past. Kairozdormu referred to them as timeways in the shortstory Hellscream. These alternate universes, which are apparently infinite, are also experienced by Thrall in the novel Thrall: Twilight of the Aspects.
Quotes from Hellscream short story:
"Because most of the hourglass is still in our Azeroth, this piece resonates with our timeway. Call it a glimpse… a glint of time. With a little work on my part—"
"We can go back." Garrosh felt his heart race and his skin tingle. Plans began to unfold within his mind. "Not just back to our Azeroth. It could take us back to our time."
"And that is just the beginning," Kairoz said. He turned around, gesturing toward the sun dipping low on the Nagrand horizon. "First Azeroth. Then other worlds. All of them. As many as we need." The bronze dragon began to laugh. "We will be limited by nothing. Not even time. The possibilities are infinite. I will become infinite—"
You do confront Garrosh again, in Draenor, on a quest chain that takes place in Nagrand.