I've faced The Goblin Army and The Pirates a few times now. But the most recent time I faced them, I noticed that they only appear to be in the centre of world. I was in the Crimson farming for a key mold, and thought the invasion would help to get it.
When I spawned the Pirates, a few came but no more seemed to spawn. Later on when I went back home, there they where. I fought them and defeated them. The exact same thing happened with The Goblin Army.
So do they just spawn in the centre of world, or was I just doing something wrong?