There's many ways in openTTD you can 'play dirty'. It will probably take quite some time until an AI is built that will be able to defeat any methods humans can come up with. And that AI will, of course, be playing in a 'very reprehensible' way if you judged it by online server standards.
And here's also where the solution lies:
Apply those standards to your single player game
Pick out some popular open or semi-open competitive multiplayer server. Its admins have had to deal with all kinds of "griefing" and create various rules in order to promote both fair, balanced, and interesting gameplay. Sometimes tradeoffs have been made.
Pick and match the rules you like (somewhat harder to make sure things are fair) or simply adopt one in its entirety. Try not to break any of their rules, and you should find that you will not be able to 'cheat' and bankrupt the AI in trivial ways.
As to your specific example
Most multiplayer environments would not allow you to do this behaviour under the term "Blocking". The AI is running a legitimate profitable enterprise and thus should be allowed the right to keep its waterway to its dock accessible for as long as it is being used.
In general, even if the AI dock is illigitimate, a server admin would likely ban you over the AI if you were to start blocking it. The correct response would be to notify the admin, who would instruct the other player to fix their dock to comply with the rules in some way.
When would multiplayer admins then ask the AI to close its dock? There's various rule(sets) in play, but here are some reasons:
Overlapping coverage areas
Used to block an obvious construction that another player is making right then in that location. (Think of placing canals and docks in front of a railroad another player is building)
Abusing the pathfinder to DDOS the server.
Judgement of whether something is in poor sportsmanship can quickly become subjective, which is why there's human adjudicators present and necessary in multiplayer competitive openttd*.
- This is not true in strict sense, but rather to maintain the three conditions of fairness, balance, and interesting gameplay. Most players would agree these rules, which are hard to program in, make their game better, so they're manually enforced.
Assume the AI is in the wrong here
So let's say your AI is playing in a way most human players would find breaks their common rules. Read its description, to see whether the AI you're using is meant to do these things. Most AIs are written to be fun to play with, and thus obey the 'gentleman's rules' that players have with eachother, but some are written for 'anything goes' competitions, and may engage in blocking, vehicle destruction, and the like. (Rondje om de kerk AI is a good example of an 'evil' AI).
If not, you could try to fix the code or file a feature request yourself. Most AI repositories can be found over at openttdcoop.
I cannot see why you would need to destroy a waterway to maintain a town's ability to expand. Only one tile width is needed to allow ships to pass, and you have the ability to build bridges, and any connected town will also build houses (and road tiles, if enabled) across a bridge. Why not leave a one-tile canal or waterway to the dock, and build your polder across it?
You could also use canals and locks to allow the waterway to cross higher land without having to dig ditches, if desired for efficiency reasons.