Time in the "base world" actually has to pass for things to grow, corruption/hallow to spread, etc. For anything to happen basically.

So **no**, plants won't grow if you're off looting someworld else.