The problem with building lots of houses is that you have to support that population - you have to keep them fed and happy. This isn't too much of a problem at the outset, but as the game progresses each resident's needs are going to increase drastically.
Keeping them all happy means you've got to build a huge support network to meet all their needs. Buildings of various types will need to be constructed to provide them with their "intangible" needs, and production chains will have to be large enough to meet their need for goods and food.
For instance, Peasants require Food, Drink, Community, and Faith. Production chains that provide fish and cider meet their Food and Drink requirements, but you'll need enough of each that you can support the population or they will move out. Community is provided by the Marketplace, and Faith from the Chapel (or other upgraded religious buildings). These buildings have a limited radius, so you'll need enough of them to cover all your houses.
When you "level up" to Citizens, suddenly in addition to Peasant needs, you'll have to provide more goods - spices and linen garments. You'll also need taverns for amusement. Citizen homes hold more residents, so now you've got more of a demand for Peasant goods than you had before. Each successive "level up" means more residents, and another tier's worth of goods to keep them happy.
Most of the time, the limitation will be how much you can cram onto an island, and what other resources you have available. While you can spam houses early on in the game (and to an extent, you should focus on growing your population early), you can't just build forever. You will quickly exhaust your ability to provide for people, and they'll get angry, refuse to pay as much in taxes, and/or start to move away.
As you add houses or level up residents, keep an eye on the stock of your goods in the warehouse, and compensate for shortages before you run out completely. You will eventually find the proper balance between residents and production that fits the resources (land and/or fertility) you have available.