In Fallen Enchantress, your cities grow if you have food, but the rate is pretty slow. At level 2, you get to specialize your city (to research, military, or economy). Getting to level 2 unlocks a slew of new buildings particular to its specialty: military cities can construct buildings that improve troops and make them cheaper, research cities get access to large research-boosting buildings, and economic cities can get large boosts to gold and global boosts to food. After that, you still get bonuses if your city grows to level 3 and higher, but each new level generally just lets you build one new building, and an expensive one at that. These level-3-and-higher buildings give unique bonuses, but not particularly large bonuses.
Given that building up a city is an expensive/time-consuming process of constructing food-increasing and growth-increasing buildings, and then waiting quite a while, is it worth the effort to build up a city beyond level 2? Are there particular city types that are more useful to build up, while others aren't worth the effort?