It's rumored that you can give 6 soul gem fragments to a necromancer named Balmer in Winterhold and you may be able to get a black soul gem. http://www.gamespot.com/the-elder-scrolls-v-skyrim/forum/messages/platform/xbox360?topic_id=m-1-61030216&pid=615803

Within the strategy guide for Skyrim the soul gem fragments are listed under "Clutter" so they are not important or useful items. They may be requested as part of a radiant quest though.