There are no upgrades, weapons, vigors, cash, gear, or anything else character-related that carries over from one new game to the next. Every time you start a new game, you play from the start of the game with the mint condition start-of-game Booker.
The chapter saves are just that - saves from the beginning of each chapter. You can load a chapter save and start playing from whatever the state your game was when that save was created. If you get to a chapter boundary, your current game will overwrite any existing chapter save for that chapter.
There is only one set of chapter saves, so if you play on Medium and beat the game, and then go back on 1999 Mode, the 1999 Mode saves will overwrite your Medium saves. (... and you can't change the difficulty in 1999 mode!)
When it comes to achievements, collectibles and other counters are tracked across games. If you need to get 20 kills with a particular weapon for an achievement, the counter is global and counts even if you reload from a checkpoint or start a new game. For the "collect all the Xes" type achievement, you must find each unique instance at least once.
For example, if you miss a voxophone in your first playthrough and you start a new game, all the voxophones will be present, but only the one you missed will count towards the achievement. Kinetescopes and telescopes work the same way.
The infusions are kind of a special case - you have to have a total of 24 (the number there are in the base game) to unlock the achievement. This means if you get one, reload a checkpoint, and miss it on the second try, you will not get credit. Conversely, if you have the preorder bonus which gives you 5 early on in the game for free, you only need to find 19 of the rest of the infusions in order to get this achievement.