I had this problem several times. The first time I was in god mode because I was tired of dealing with the all the crap just getting down to the docks (but that's not important). I tried again out of god mode to see if that did anything, but it didn't; and my third attempt was the same. Now each time I tried this, I entered the dock area by going down the hill through the gap between the building and the rocky area.
But the fourth time, I entered the dock area along the shoreline. I also read the phrase "Kill the bandits at the docks" literally, meaning kill the bandits in the area of the docks (because I was fighting the bandits on the hill rather than in the docks), and it worked. I also didn't use any shouts or destruction spells the fourth time (I had been during my other three attempts). I kited them around the firepit (using my axe, shield, and conjured bow) and used fast healing.
Long story short: enter dock area along the shore, kill bandits in dock area, don't use shouts or destruction spells.
I hope this was helpful, sorry about the long explanation.