I remember Harbors were quite crucial buildings in Civ IV, but they don't seem to be anymore in Civ V. They only provide a 25% production increase for building sea units and establish a trade route, if one doesn't already exist. Usually one or two advisors are picking the Harbor as the next important building to build, and insist on their choice for centuries.
If I already have an established trade route through roads and am not particularly interested in building sea units though, am I missing out on anything important if I don't build one?