The design of the GBA and older Nintendo handhelds was a bit more simplistic than later DS and 3DS designs. On the GBA, the cartridge is basically an extension of memory, as opposed to the later devices where the cartidge is more like an SD card and hot-swappable. So, when you pull the cart from the GBA you are cutting out that memory and leaving a big hole. This includes not only the sound data that was being played, but also the code that was itself running. This triggers what you could bascially call a crash.
In theory, this situation could be accounted for, but unlike on your desktop computer or later consoles, there isn't much memory or space to store the backup routines and keep them loaded for the rare situation where the user pulls the cartridge.
As for the glitchy noise, I'm not 100% sure on what precisely it is, but if I was to take a guess, I would say that it has something to do with "DMA" (Direct Memory Access), which is a hardware acceleration offered by the GBA that essentially lets software say "Please copy some data in the background from place X to place Y". While that transfer is happening, the CPU can move on and do other stuff. This is important, because memory access is comparatively slow on the cartridge (much slower than RAM, though faster than even SSDs). This is often used during sound processing since sound data tends to need loading more often than graphics.
My guess is that when you crash the CPU, the DMA gilitches out and either copies too much, or writes garbage or something, and the result is... not too pleasant to hear.