My observation is that it works in a following way.
Nearby list shows the Pokémon at the Pokéstops which might be up to ~1 km far away from you.
If there are four or more Pokémon in the Nearby list, you will see only the Nearby list.
If there are less than four Pokémon in the Nearby list, then the Sightings list (aka the "old tracker") is shown too displaying Pokémon that are 200m from you or closer (not necessarily at the Pokéstop).
It looks like a first step towards the rural players who don't have tons of Pokéstops around them, so this way they at least see three Pokémon that are close.
Thus, what distance determines whether Pokémon are shown as Nearby versus Sighted? Is it distance to the player or to the Pokéstop?
It is the distance of the Pokémon from the Pokéstop plus some internal hidden flag (not all Pokémon at a Pokéstop are counted as Pokéstop ones and would be shown in the Nearby list). It is however safe to say, that all Pokémon shown in the Nearby list, will be up to 10-15 meters from the Pokéstop (up to 1 km from you). And Pokémon in the Sightings will be up to 200 meters from you (no matter whether at a Pokéstop or not).
Imagine being in a situation shown below. Note, that the "Pokéstop Pokémon" is displayed as linked with the Pokéstop, while the Pokémon that is not related to the Pokéstop, it just happens to be at the same location, is displayed as not linked with the Pokéstop (e.g. Charmander).
At the beginning you will see only the Nearby list with the Caterpie, Rattata, Ekans and Spearow. You won't see Bulbusaur, as it's too far (let me say that I don't know what exactly the limit is, but it looks something around 1km +/- 200m). And you won't see Pikachu or Charmander, as they are not the Pokéstop Pokémons. Then you'll catch Caterpie. Since there are only three Pokémon in your Nearby list now, the Sightings list will appear showing Pikachu. You still won't see Blastoise or Charmander as they are too far for the Sightings.