Is the presence of dirt and gravel underground supposed to be a an indication of something being in the area (eg. minerals or ores) or is it just placed there when the world is generated just for the hell of it?
Dirt and Gravel are found underground, along with stone, water, lava, and occasionally obsidian. All of these are 'natural', with no significance. (You can also sometimes find sand)
Moss Stone indicates Dungeons, and if you find Wood/Rails, that's usually an indication you've wandered into an abandoned mine-- but gravel and dirt alone are inconsequential. They're just there for variety.