I've seen people quick drop their currently holding weapon in Borderlands 2 without accessing the menu, but I can't figure out what key combination I have to press! I'm playing on my PC with an xbox controller.
Actually, you can.
In the PC version there is a keyboard mapping for that in the controls menu. It's by the lower end of the list, so it's easy to miss. By default it is '-' or something. Not sure about xbox controllers though.
P.S: I didn't even know about that until I got a crappy loot spree once and the first time I came across a decent gun after that I thought "Hmm, I don't even want to keep this thing, there should be a more expressive way to get rid of stuff!" After I found that option, I had tons of fun tossing things over the ledge above the vending machines in Frostburn Canyon, as there were too many loot-dropping spiderants and my backpack was too small, so I needed several trips before everything was piled up and ready to be sold.
If you have a full backpack, and you hold E while looking at another weapon with a weapon in your hands, you will equip the one you are looking at and toss out the other one.
This only works if you have a full backpack. You cannot throw out items without going to the menu if you do not have a full backpack.