No, they can grow at any time.
The only thing that influences how long this process takes is (pseudo) random chance. Depending on your luck, it could be anywhere from milliseconds to years.
After they've grown, they start blooming when certain conditions are met (which I've detailed in this answer); this too can happen with the player watching.