Note: This answer contains spoilers.
This question will result in inherently speculative answers. Unless we get word from the people who wrote the story, any of us could be correct.
Multiverse theory posits that every decision you make results in a new universe. There's the universe you're in, where you made the choice you made. There's also an alternate universe, where you chose another option. You can visualize this as a series of paths which continue to fork at each decision point.
When you hop between "alternate universes", you're actually hopping between universes where different choices were made. If you travel to one of these alternatives and change something, you're not changing the outcome of that alternate universe. You're actually creating a new alternate universe where a new choice was made.
Traveling between universes at the same point in time would be like traveling horizontally across the image above: you see what would have happened if a bunch of different choices were made, but you don't affect any of the other branches.
For example, after the baptism, a lot of choices were made by a lot of people. If I traveled to an alternate universe where Booker became Comstock, then killed Comstock, all I actually do is create a new alternate universe where Comstock dies at that point in time. There'd still be an alternate universe where a copy of me traveled to it and chose not to kill Comstock. Comstock would also be alive in many other alternate universes.
Elizabeth created rips in space. The blue rips she created were links between alternate universes at the same point in time (or horizontally in the above image). The red rips appeared to cross time as well (like the Paris tear in her room), but they never enter the red tears.
During the ending, Booker and Elizabeth enter the "lighthouse ocean". This takes them to a space outside of any of these branches. They start going through a series of doors. These doors allow them to go backwards and forwards in time. In other words, they can travel vertically in the image as well as horizontally.
After going through one of these doors, you travel back to the original baptism. However, when you travel back to the baptism, you're traveling back to a point where every Booker or Comstock that was created after the baptism are all the the same person. Until this pre-baptism version of Booker makes a choice about whether to go through the baptism or not, no alternate universes based on that decision exist.
Let's treat the bottom node of the above image as the choice of whether to get baptized or not. Before you make that choice, you're "PreBaptismBooker"
When you back out of the baptism, you follow path '0' and becomes NoBaptismBooker0. Every universe where you never got baptized would be the ones ending in '0' in the image (the entire left side). This represents the Booker Universes. The first time you go back to the baptism, this is the path you see play out.
This choice spawned another universe where he didn't back out of it. This is path '1', where he becomes BaptismBooker1. Every universe where he gets baptized would be the ones ending in '1' in the image (the entire right side). This represents the Comstock Universes.
Choices continue to be made in both universes, thus causing more branches. Each branch, however, shares a single parent point in time: the time when Booker decides whether or not to go through the baptism.
Later, when Booker decides to go back and kill Comstock before he's 'born', they travel through another door and go back to the baptism again (the bottom point).
This time, you begin to see him going down path '1' into the Comstock Universes: he accepts the baptism. However, before any further decisions can be made, the Elizabeths drown him. This action snuffs out the entire set of Comstock Universes (the right side of the image) because it changes the result of the decision: Instead of the baptism decision turning him into Comstock (or BaptismBooker), it turns him into a corpse.
Basically, every Booker that decides to get baptized dies because, at the point of the drowning, Booker has committed to path '1'. However, this doesn't affect any of the Bookers that committed to path '0' (no baptism).
In your question, you posit that this explanation can't be the answer because, by traveling back in time, "PlayerBooker" would see "PreBaptismBooker" and have to kill "PreBaptismBooker" rather than "PlayerBooker". Since they kill "PlayerBooker", they didn't kill the one that makes the decision.
Your assertion that this can't be the case seems hinged on the fact that, by traveling through time (rather than across universes at the same time), you'd be a different physical entity than the PreBaptismBooker.
This would be the case if traveling backward along the forks worked like traveling across them. You're right in that there are tons of Bookers running around at that point in time that are temporal decedents of PreBaptismBooker.
However, when you travel back to the baptism, you're traveling back to a point where those temporal decedents are all the same person: PreBaptismBooker. Until the baptism happens, the multiverses with all the other Bookers that result from the decision don't exist. When you travel back, PreBaptismBooker is PlayerBooker, and NoBaptismBooker0, and NoBaptismBooker0100, and BaptismBooker1, etc. This is what Booker means when he says "I'm both" in the end.
He then decides to undergo the baptism (becoming BaptismBooker1), then drowns immediately before he can become any other BaptismBooker1 on the right side of the image.
This is how, after the credits, you see Booker again, and he heads to the crib to see Anna. What you're seeing is one of the Bookers who spawned from NoBaptismBooker0 (the one who decided not to get baptized) and later decided to do something which resulted in Anna being born.