The store ("lost and found") holds one upgrade of each type. So far all of my upgrades were found in the game world. Do I have to buy an upgrade from the store in order to get the 5th upgrade, or can I skip those purchases and just find them later in the game world?
To fully upgrade them on your first play through, yes. However, after you beat the game a New Game+ unlocks which carries over all your upgrades and weapons from the previous game. This makes buying from the shop unnecessary if you plan on doing a second play through.
However, unless you plan to fully upgrade every weapon in the game on your first play through, you won't need to concern yourself with money. You'll have enough money to clear out the lost & found, in addition to fully upgrading at least two weapons, as well as partially upgrade a few others.