Whenever you build a Settler or a Worker, all your extra food goes towards increasing the production of that unit. Due to this your city will always be stagnant when you're building either a a settler or a worker, and I'm willing to bet this is what is happening to you.