No, they can grow at any time.

Plants have three states: shoot, growth, and blooming. All plants start as a shoot.

They transition from shoot to growth randomly, and each plant grows at a different rate. The only thing that influences how long this process takes is (pseudo) random chance. Depending on your luck, it could be anywhere from milliseconds to years.

After they've grown, they start blooming when certain conditions are met (which I've detailed in [this answer][1]); this too can happen with the player watching.

If you're farming with clay pots, it is best to plant as many shoots of the desired type as possible to mitigate bad growth luck.

  [1]: http://gaming.stackexchange.com/questions/25288/what-makes-plants-bloom#25291 "terraria - What makes plants bloom? - Gaming - Stack Exchange"