The above answer by ChrisHatez covered the varieties of relic pretty well but it's also worth noting that the Vault Hunter's Relic is far less useful than it appears. It doesn't actually increase the drop rate of Purples, E-Tech or Orange weapons, only of greens and blues and even then, the increase is tiny. I made the same mistake as you, never unequipping it but you really are better off using something else given how little a difference it makes.
So I just investigated the Vault Hunter Relic data.
It modifies two attributes:
Let's set them to +100.00 and -100.00!
Net result: I have not seen a single white weapon drop, greens drop,
seen a few blues. Nothing else.
So, what I can gauge from this, is that it shifts a 5% of your chance
for whites into non-whites. Doesn't touch anything else. Very low
impact, perhaps bordering on useless.
Source: http://forums.somethingawful.com/showthread.php?threadid=3504490&pagenumber=133&perpage=40#post408051645 (page is now not viewable by unregistered users; same post quoted in the Gearbox forums)