I've built all the settlers that I can from my residence, and as far as I can tell, the building doesn't seem to do anything else but consume my crop. Can I just demolish it now?
Residences have a couple other uses besides just training settlers. They also produce 77 Culture Points per day at max level, which is slightly above average. However, the main reason to keep it around is that it prevents your village from being conquered as long as it stands, requiring people to catapult it before they can start conquering.
If you have good defence and you think that you can protect your village, you can destroy this building. Quite often one more Granary will be far more useful then a Residence.
Here are the advantages and disadvantages of demolishing the Residence: