I'm not party to the agreements Steam has with publishers and other distributors, but I can get a pretty good picture of it from viewing it as a gamer/consumer.
If someone wants to sell a game, they can make a deal with Valve to sell keys to that game that are redeemable on Steam. In this case, Valve most likely gets a small cut when the key is purchased or used (they're providing the bandwidth after all...) However, Valve doesn't control the price the game creator sells their game at if they opt to sell outside of the Steam store.
For instance, Amazon might strike a deal with 2K Games to sell copies of Civilization 5 at a steep discount - the games include a Steam code as Civ5 is a Steamworks title. Valve is just providing the infrastructure, they're not setting the price that 2K can sell the game at across the wide expanses of the internet.
When it comes to retail, the store you're purchasing from is buying the games in bulk at a discount from the distributor or publisher. They might choose to pass that discount along to you, in the hopes that you'll think more highly of their store or purchase supplemental accessories or something similar. (This is called a loss leader and is a common retail practice.)
Thus, Blizzard might agree to sell thousands of copies of the latest Starcraft title at some percentage discount to Newegg. Newegg then turns around and either gives up that percentage discount as a "preorder bonus" to customers. You buy the game from Newegg, and you see that they've also got a sale on DDR3, which would be a perfect upgrade for your rig to play the game. Newegg sells a product they might not have sold otherwise, and you feel like you got a good deal. One axiom I like to quote is "if you can't figure out how they're making money, you're not the consumer - you're the product." The meaning here is that they wanted your eyeballs on their site, and they got them by offering you a discount.
Another factor at play here is arbitrage, where companies or people buy a product at a discount, and then resell at less of a discount. For instance, some people might take advantage of a Steam sale and get a game 50% off, and then turn around and sell it for 25% off after the sale ends. Alternatively, if the game keys work internationally, they might buy copies in a country where the price is low, and sell them in a country where the price is higher. Sometimes this is not allowed, though. There may be region locks or other protections to keep this type of arbitrage from occurring.
Be warned, there are deals out there that are scams. If it looks too good to be true, it probably is. I'd suggest only dealing with reputable distributors whenever possible. If you don't think you could contact their customer support and get your money back if the keys weren't legit, I'd think twice about buying.