You need to replace the files with empty ones to disable the ambient sounds, but Minecraft will put them back how they were at launch unless you prevent it.
I'm using the ambient cave sounds as an example, but you can browse around the newsound directory and repeat this with other sounds you don't want.
Back up the existing files (in case of ...something. OK, they're self-healing, don't worry too much).
Assuming you're on Windows, they are located in: %appdata%\.minecraft\resources\newsound\ambient\cave
Copy them away somewhere if you want to keep them. There should be 13 files with a .ogg extension.
Replace the files with empty files (or something completely different if you fancy - in OGG format).
First delete all the existing files. Then create an empty .txt file and rename it to cave1.ogg, copy/paste that and rename as required until the directory looks like it did before.
As suggested by Koviko, you can automate this if you want. Create a batch file in the empty directory with the following content, and execute it as an administrator:
@echo off
echo. 2>empty.ogg
attrib +r empty.ogg
for /L %%i in (1,1,13) do (
mklink cave%%i.ogg empty.ogg
attrib /L +r cave%%i.ogg
)
This creates an empty.ogg file and then creates shortcuts/symlinks to it for the rest. It's faster than doing it manually.
Protect your files from being replaced
Select all of your new files, right click and go to Properties, then select the Read-only tickbox and click OK.
The next time you start Minecraft, it detects that the files have been changed, and tries to replace them with fresh ones but finds that it can't overwrite the files in that directory. In testing this I've not noticed any errors or other weirdness as a side-effect, so it seems to work ok.
An alternative to this is to use something like Sound Mod Enabler, which will probably allow you to make any modifications you like, but personally, I like to avoid using mods to do things I can achieve in the OS anyway.