I had trouble with this too. Here's what I did:
If you head from Dragonbridge towards Rimerock/Lost Echo Cave, you know there's a small fork in the road not far from there. Left will take you to Rimerock and Lost Echo. Instead, go right. (Just past the fork on that side is a small alcove-type area that will unlock Wolfskull Cave.) Follow the path almost all the way down until you can access the shoreline. There are some wolves that will attack on the way. When you run into a farmer and his wife asking for help, you're getting close to the shoreline. (They tell you to watch out for dragons, but I didn't see any.) Walk off the path and down to the water.
Stay as close to the water's edge as possible and avoid or kill any enemies. You should come across a Vigilant of Stendorr fighting some crabs. Keep on for a while and you'll see a wrecked boat in the water. Just past that is a walrus (I didn't stick around long enough to see if it was friendly, so I'm not sure if they'll harm you or not.) and then Lower Steepfall Burrow. You should pass some more walruses shortly after that, and then you're nearly there. Just keep going until you see Avulstein (if you agreed to join him) or the marker.