tldr: To be used as a canal, a fort needs to meed two conditions:
Be adjacent to a water tile (lakes and iced-over ocean tiles both qualify)
Be in your own cultural borders (or a vassal's cultural borders, or your master's cultural borders if you have open-borders access)
h/t PieceOfMind's explainer at the CivFanatics forum for the breakdown:
10. Forts can be used as canals
Any fort in your culture boders, or city, which touches a water tile (inland lake or coast) can have naval units pass through it. Forts in neutral territory can not be used as canals. For a civ that has open borders with you, their naval units cannot use your forts but they can use your cities.
Additional note: Masters can always use a vassal's fort canals, but vassals can only use the master's fort canals when they have open borders. Strangely, your privateers cannot be attacked by your vassals if they are in your forts, as it seems the vassal cannot see the privateers. (thanks DanF5771)
Another note, by way of example, quoted from UncleJJ (thanks): "I've built a line of forts 8 wide across the southern end of my continent when it was blocked by ice flows. Submarines could get through and my workers had nothing better to do so I built this "Sothern Passage". Each fort had to be either next to water or next to ice (which counts as water for this purpose) and also needs to be inside your own culture for the ships to pass. This was quite useful, in that game, for shifting ships from one side of my continent to the other, and massing my fleets, rather than a long trip around the other continent as mine was blocked at the northern and southern ends."