In summary it uses trial and error. There is a sheet of game data which has base stats and then a bunch of functions that just try different values of IV that would match the numbers you have given it. Since a given CP/HP combination can have multiple possible sets of stats it suggests levelling it up and trying again and this way it can eliminate some of the incorrect possibilities.
In more detail this is the algorithm that is uses...
It works out level of Pokémon (and levels can be halves as well as whole numbers) by the stardust cost of the next upgrade (table is in the game data sheet). The "powered up?" column determines whether these half levels are actually allowed. If "powered up" is TRUE then it will allow half levels, otherwise the level must be a whole number. I assume this is because pokemon when caught will always have an integer level.
It then loops through all possible levels getting all the combinations of stats that work for that.
2.1. First for a given level it works out possible stamina values. Each level has a CP multiplier (in the game data sheet). IV stamina can be anything from 0 to 15. It adds this value to the base stamina of the pokemon and multiplies it by the CP value. It then takes the floor of this value (so taking just the whole number part so 59.9 would be 59). If this matches the HP you were given then the stamina is a valid possibility.
2.2 Attack and Defence are also between 0 and 15 and it tries all 256 combinations of those with the possible staminas. It works out the CP for each possible combination and any that match the CP you have given it are added as possibilities.
2.2.1 CP is worked out by taking the attack ((base+IV)*CPMultiplier), the square root of the defence ((base+IV)*CPMultiplier), and the square root of the stamina ((base+IV)*CPMultiplier), multiplying them together and dividing by 10.
This gives you your list of possibilities.
Once you try again after a level up because of the way the rounding works some of the previous calculated possibilities will likely differ by 1 (or maybe more) and therefore can be discounted. Do this enough times and it will work out a unique set of values for the stats.
The Dratini example should show you how this works. Follow through the maths on each of them to see how it works.
If you have any questions on specific bits of this operation then let me know and I'll see if I can clarify anything.
Source: This information is all derived from reading the scripts in the script of the sheet. You can see the code once you have copied the sheet for yourself by clicking on "Tools...Script Editor..."
htmlviewin that link with
edit. I strongly suggest that you File => Make a copy before poking around, to reduce the lagginess of the spreadsheet.