Update
Delete a shared library
So that you need the other way around there is a good and a bad message.
The bad one is
Get them to remove you, its a one way set up.
Removing them from you share doesn't remove you from thier shares.
The good one however, there is a workaround available which I've not tested but as you just have to comment out a few lines you can't kill anything.
Managing and Revoking Access
Unlike the process of authorisation, managing and revoking access to your Steam library can be handled entirely within your own Steam client. To do this, open Steam > Preferences (Steam > Settings on Windows), select the Family section again and click Manage Other Computers.
To revoke authorisation for a specific device, and free up one of your ten slots, click the Revoke link next to the name of that device. For reference, the Steam community name of the person who uses that device to access your library will be shown under By Steam User.
Note, however, that if the person is authorised on any additional devices, they will still be able to access your library on those particular devices.
To revoke access to a person (and free up one of your five slots), click the Revoke link next to the name of that user. This listing will show the user's current Steam community name, so you should reference the time they were added if unclear.
Source