When playing Civilization IV and trying to conform to the specialized cities strategy, the situation invariably arises where I have too many military units or run out of buildings that are geared towards the city's specialization (at least until new technologies are discovered). In those cases, I usually pick one of the buildings that don't necessarily belong to the specialized chain until a more appropriate building type comes along. The end result is that usually as we get towards the end of the game, most of the cities have all the buildings that can be built, and it seems like no cities are specialized at all.
My question: Is this a good strategy or does buying a barracks in an economic city or buying a factory in a military city etc... cause problems with the specialized city strategy (i.e. slower production/growth)? Is it better if I simply switch to gold, research or culture generation until an appropriate building/unit becomes available?
I'm relatively new to the game and this technique works at the lower difficulty levels, but I was wondering if it is an optimal strategy or not.