Time in the "base world" actually has to pass for things to grow, corruption/hallow to spread, etc. For anything to happen basically.
So no, plants won't grow if you're off looting someworld else.
An addition/exception by Bob from the comments below: The player doesn't have to be physically in the world; the world just has to be running. One can leave the world running using the dedicated server launcher.