Reds are "hostiles".
If you're friendly with a faction that a ship is hostile to, they can show up red. If a ship is shooting at you, they'll show up red. If you shoot at a ship, they'll change to red (not sure if it's after they shoot back or immediate).
Assuming you're neutral (or better) and "clean" in a system, you can shoot at (and kill) a ship that is "wanted" and you can shoot back at (almost) anybody that fires at you first.
In other words, just because a ship is "hostile", it's not necessarily legal to fire at it. It's just likely to fire at you.
If you shoot at a clean ship, you will immediately get a (small) fine and become wanted (there will be a message about the fine in the upper right info area and a Wanted notification over your Fuel status area). Once you're wanted, it's legal for any ship to fire at you, and illegal for you to shoot back.
Probably what happened is that when you were firing at a wanted ship, one of your shots hit a clean ship, you became wanted, almost everybody became hostile, and you ended up with a large fine for killing a clean ship (or two or three of them?).
When you're in a crowded space, you really need to watch your shots and be careful not to hit any other ships. If you do unintentionally hit a clean ship, don't kill it (that will get a much bigger fine), don't shoot at any other clean ships, and run off to a station to pay off your fine/bounty quickly.
Note that ramming (or accidentally bumping) another ship counts as an attack.
Note that warning shots (firing but not hitting a ship) do not count as an attack.
Personally, I don't care for bounty hunting at nav beacons because they're so crowded and it's easy to accidentally hit a clean ship. I prefer to pick up a few tons of something reasonably expensive (look interesting to pirates with cargo scanners), and go for unidentified signal sources, being interdicted by pirates, or sitting in frame shift with the throttle low scanning ships until I see a wanted one that I can use my frame shift interdictor on. At the signal sources, it's common to find both pirates and a couple system authority ships, and in the other two cases it's likely system authority ships will eventually show up. However, when it's just a couple clean ships instead of a dozen, it's a lot easier to avoid accidentally hitting one.
In an anarchy system, there is no system authority and you can shoot at anybody. If you have a warrant scanner and warrant scan them before killing them, you can often get bounties redeemable in other systems.
If you're in a conflict zone, the rules are different. I think in that case, if you choose a faction (in the functions tab of the system/right panel) the opposing faction shows up red and there won't be any fines. I'm not sure, since the only conflict zone I've visited was one with only a single faction in it.
