To fully understand why this works (because it's not simple), I have to talk about programming briefly.
Most of the time nowadays, programmers don't worry about hardware when they code business software; they just write the logic in the best way possible and the code takes care of most of the memory management issues for them. Modern programming languages make this sort of optimization automatic.
In game programming, particularly old games and console games and very particularly old console games, the code is much more "low-level." This means that you're using the code to directly interact with hardware, swap memory addresses in RAM, etc. Reason being is that in the beginning, computer hardware simply was not made for games; there was not enough money in the industry for it. Using low-level programming, coders can fine-tune just exactly how the machine utilizes memory and other resources. The downside is that it requires a lot more knowledge (namely, how your hardware works) and is much more prone to mistakes, because many more instructions need to be written. That's why these languages have been fading out of primary use in the software industry.
To emphasize just how important low-level coding was for games, conventional machines were not made with GPUs designed to handle the ridiculous amount of mathematical operations involved with 3D graphics, or sound cards containing their own synthesizers. Even when those things came about, like with the N64, they were fairly basic pieces of hardware and programmers had to be very frugal when they wrote the game to save memory and work with other hardware limitations. Back in the SNES days, a SNES was actually much more powerful than your typical PC when it came to video games, and PC game developers had to virtually perform magic to get some games to function. PCs could not even handle real-time scrolling like the NES could -- John Carmack actually approached Nintendo with this tech working on PCs back in the day, showing that Super Mario Bros. could be ported to PC, but he was turned down, and thus Commander Keen was born as a franchise.
The primary issue
In code, when you want to indicate something -- say, the warp portal at the end of a boss fight in Ocarina of Time -- you use something called a "pointer." A pointer is basically just an address that tells you where something is in memory, so you don't have to copy that data in memory every time you want to refer to it; you can just refer to the same address every time. Now, in modern programming this would be considered very "unsafe" (prone to errors), but the coders on the Zelda team decided to re-use pointer variables for completely different things (numbers, graphics, words, etc), because with their RAM essentially completely full all of the time they just have to swap out what they're referring to; there isn't enough space to refer to everything all the time and there isn't a hard disk to store anything long-term.
This is going to be difficult without a flowchart, so I'll try to lay it out in a nice numbered list. Also, I know more is involved in the glitch than just pouring bugs on the ground, but I omitted those steps because they serve the same purpose and in the interest of brevity.
Imagine this: In the code for Zelda, there's some address assigned to a warp point that tells the game what area you're going to. Now let's say you have some bugs in a bottle in your inventory. When you let those bugs out of the bottle, the code tells that address to point somewhere else -- it needs to use it right now in order to tell the game how to render the bugs, or perhaps what pattern they move in when they hit the ground. Now, there's some data at that original address. What does that data mean to the warp point? The answer is that it is usually impossible, or at best very difficult, to determine -- only the game truly knows that. Fortunately for us, that data happens to translate to "Ganon's castle" for the game's purposes. So let's review what just happened:
- You kill the boss. A warp point appears.
- At this point, the game uses a pointer (we'll call it Pointer A) to store the data that tells the game what area to warp to.
- You take out a bottle of bugs and empty it.
- Pointer A is re-assigned in the code and repurposed so that the game can properly handle this event -- I could not tell you what its exact purpose is because I do not have access to the OoT source code, but I can tell you that it was used for something else (that's the only way seemingly-innocuous glitches like this can have such profound effects).
- You walk into the warp point. Cutscene of you leaving happens.
- The game completes the warp sequence and decides which transition in the game to take you to based on what's in Pointer A. By inputting what is normally garbage data into Pointer A with our previous actions, the game interprets the data anyway and discovers that it is actually a transition in Ganon's castle. Not one to ask questions, the game complies.
The reason that the glitch is so much more involved and has many more steps is that in reality these pointers do not perfectly point to the same address in memory. That space takes up a certain "width," to use an abstraction, and other addresses overlap with that one partially, but not fully. So, if we have pointer A pointing to an address, pointers B and C might point to a space that half-contains the space that pointer A also points to, and half-contains space before and after that space (if you imagine three rectangles, with the halves of two of them overlapping the full rectangle in the middle -- this is basically how it works). So, in order to actually get at pointer A, we need to perform in-game actions that modify both B and C. That's just a basic example -- there's many more than just 2 pointers that the speedrunners need to modify in the actual game.
It's relatively easy to explain this glitch in terms of what you do in the game, but I wanted to take this opportunity to explain how the glitch actually works. Feel free to ask more in the comments and I'll provide what details I can based on my knowledge as a programmer. I tried to explain this without using any jargon and in layman terms, but I am an introvert and my communication skills aren't perfect -- don't be afraid to ask for clarification.