The currently accepted answer does not really address explosive weapons at all, so I would like to add an answer to address this specifically:
Stick of Dynamite
The Powder Gangers are ever present in the area around Goodsprings. One could either kill them and take it off them, or ally with them and buy it off them. Eddie will give you some dynamite every few days, too. This should be quite a handy source of explosives early on.
While Dynamite, and the Long-Fuse variant, are not necessary the most precise or practical weapons, they certainly fit the bill for effectiveness. Human enemies, especially when surprised, can often be killed or seriously injured by just one hit.
Furthermore, since you mentioned you were playing a stealth-based character, you can assassinate people by pickpocketing them. If you put your Dynamite into their inventory, it becomes "Dynamite (Live)" and it will explode in their inventory. This is usually lethal to anyone who can be pickpocketed.
Tin Grenades
If you have the Gun Runner's Arsenal DLC, you have access to the Mad Bomber perk at Lv. 6 with 45 Repair and 45 Explosives skill. It allows you to craft a Tin Grenade from one piece of duct tape, one tin can (not the bent tin can though) and 50 units of pistol powder.
The Tin Grenade is a very easy source of explosives, given that the source materials are abundant throughout the game. You will likely have a lot of pistol ammo for calibers you don't use at all. As such, you can break them down and get access to the powder to turn into grenades. A Tin Grenade is about 25% more damaging than a stick of dynamite.
Mines
Mines may not be as "aggressive" as grenades, they can still be thrown towards enemies and be used to deter chasing enemies.
With the Mad Bomber perk, one Bottlecap Mine can be constructed by using a Lunchbox, 8 bottle caps, a sensor module and 3 cherry bombs. These are rather uncommon resources in comparison, but still manageable to obtain, if one knows where to look. You can also craft these without the perk, but you will need 5 Cherry Bombs instead of 3, and 2 more bottle caps.
Regular Frag mines can be found around the wasteland too. These can be deactivated and picked up, which means you will often have a few of these available, even if you never go out of your way to find them.
25mm Grenade APW
This is by far my favorite grenade launcher in the game. 25mm ammo can be obtained by visiting the Gun Runners later in the game, or bought from the Boomers or Great Khans. All of these sources are close to the New Vegas area, meaning that you will not have access to them early on. This is unsurprising, given how incredibly powerful the weapon is.
The upside to this rarity is the fact that you have a very accurate explosive weapon, able to take down a human enemy in only a few shots, while offering surprising potential for splash damage. With all mods installed, the APW can load 8 shots into a magazine, shoot more than three times per second, and deliver this 25mm ammo very far.
As you mentioned, you are currently equipped with the "Merc's Grenade Rifle". This weapon is very powerful in the beginning of the game. Too powerful, for the worse. 40mm ammo is very rare in comparison to 25mm ammo or other explosives, meaning that every shot has to count, and has to go to a "deserving" target. This makes it difficult to decide when to fire it or not.
Later in the game, you will be able to get "Thump-Thump", which adds some of the benefits of a fully modded 25mm APW to the 40mm launcher, but the ammo is still the most difficult part to obtain.
More explosive weapons
The game has more explosive weapons to offer, such as the Fat Man, Rocket Launchers or the Grenade Machine gun. These are, for my personal taste, rather impractical and serve more as a Shock-and-Awe weapon than anything practical. Of course, one might argue that this is the whole point of a character focussed on explosives, and I am in no way implying you should not get a Fat Man if you want to get a Fat Man. My point is rather that you will use the 25mm APW more than the Fat Man.
Tips on how to play an explosive character
Explosives are a rather unique kind of weaponry. They are all rather inaccurate (compared to other weaponry), heavy and spread few and far between. As such, you need to adapt your playstyle accordingly.
Get all the Explosive Perks
This should be a no-brainier, but I still want to mention it. Perks such as Demolition Expert (+20% explosive damage), Hit the Deck (+25 DT against Explosives / Useful to prevent self-damage) and Splash Damage (+25% larger AoE) are very useful.
Use Guns
Playing an explosive-focused character does not mean playing an explosive-only character. As the above-mentioned answer already says, Cowboy Repeaters or Service Rifles are easy to obtain early on and useful to dispatch enemies such as Mole Rats or Bloat Flies, where explosives would just be too valuable to use. Unless you're the kind of guy who just likes chucking sticks of dynamite at local wildlife.
Never throw more than one explosive in V.A.T.S.
Explosives and V.A.T.S. don't go well together. While the initial shot or throw usually is accurate, by the time the second one comes around the target has already moved and you just throw your stick of dynamite away.
As such, learn how to get good at manually throwing Dynamite.
