Note: This answer contains spoilers.
I'm going to start this answer by saying that, unless we get word from the people who wrote the story for the game, this question will result in inherently speculative answers. Regardless of what makes sense based on our frames of reference regarding time travel and multiple universes, any of us could be correct. The actual correct answer lies within whatever "laws of the universe" the writers of Bioshock Infinite created.
Multiverse theory posits that, for every decision that is made, it results in a new universe. There's the existing universe, and an alternate universe where another option was taken. You can visualize this as a series of paths which continue to fork at each decision point.
When you hop between "alternate realities", you're actually hopping between universes where different decisions were made. If you change something in one of these alternate universes, it doesn't actually change anything in any of its variants. Traveling between multiverses at the same point in time would be like traveling horizontally across the image above: you see what would have happened if a bunch of different decisions had been made, but you don't affect any of the other branches.
An example: If I killed Comstock in one alternate universe that was formed based on a decision after the baptism, he'd be alive in another alternate that was formed based on a decision after the baptism.
Elizabeth's tears were essentially travelling horizontally across different multiverses... at least, the ones they entered. You do see a few tears that seem to cross time as well (like the Paris tear in her room), but these are colored red and they never enter the red tears.
During the ending, Booker and Elizabeth enter the "lighthouse ocean", which essentially takes them in the space outside of any of these branches. They start going through a series of doors. These doors are doing more than the blue tears did (which allowed them to travel between universes at the same point in time)... they're allowing them to go backwards and forwards in time. In other words, they can travel vertically in the image as well as horizontally.
After going through one of these doors, they travel back to the original baptism, thus becoming PastPlayerBooker. Here, PastPlayerBooker either decides to back out of it or to continue getting baptized. Think of this decision as the bottom point in the image.
When he backs out of it, he follows path '0' and becomes PastPlayerBooker0. Every universe where he never got baptized would be the ones ending in '0' in the image (the entire left side). This represents the Booker Universes. The first time you go back to the baptism, this is the path you see play out.
This decision spawned another universe where he didn't back out of it. This is path '1', where he becomes PastPlayerBooker1. Every universe where he gets baptized would be the ones ending in '1' in the image (the entire right side). This represents the Comstock Universes.
Decisions continue to be made in both universes, thus causing more branches. Each branch, however, shares a single parent point in time: the time when Booker decides whether or not to go through the baptism.
Later, when Booker decides to go back and kill Comstock before he's 'born', they travel through another door and go back to the baptism again (the bottom point).
This time, you begin to see him going down path '1' into the Comstock Universes: he accepts the baptism. However, before any further decisions can be made, the Elizabeths drown him. This action snuffs out the entire set of Comstock Universes (the right side of the image) because it changes the result of the decision: Instead of the baptism decision turning him into Comstock (or PastPlayerBooker1), it turns him into a corpse.
Basically, every Booker that decides to get baptized dies because, at the point of the drowning, Booker has committed to path '1'. However, this doesn't affect any of the Bookers that committed to path '0' (no baptism).
In your question, you posit that this explanation can't be the answer because, by traveling back in time, "PlayerBooker" would see "PastPlayerBooker" and have to kill "PastPlayerBooker" rather than "PlayerBooker". Since they kill "PlayerBooker", they didn't kill the one that makes the decision.
Your assertion that this can't be the case seems hinged on the fact that, by traveling through time (rather than across universes at the same time), you'd be a different physical entity than the PastPlayerBooker.
This would be the case if traveling backward along the forks worked like traveling across them. You're right in that there are tons of Bookers running around at that point in time that are temporal decedents of PastPlayerBooker.
However, when you travel back to the baptism, you're traveling back to a point where those temporal decedents are all the same person: PastPlayerBooker. Until the baptism happens, the multiverses with all the other Bookers that result from the decision don't exist. When you travel back, PastPlayerBooker is PlayerBooker and also Booker100 and also Booker10101, etc (this is confirmed when Booker says "I'm both" in the end). He then decides to undergo the baptism (becoming PastPlayerBooker1), then drowns immediately before he can become any other PastPlayerBooker on the right side of the image.
This is also how, after the credits, you see Booker again, and he heads to the crib to see Anna. What you're seeing is one of the Bookers who spawned from PastPlayerBooker0 (the one who decided not to get baptized) and later decided to do something which resulted in Anna being born.