When you're rushing a zerg player it's best to take out their Spawning Pool. It takes a long time to rebuild (plus its pretty costly), there's not normally multiples of them, and it's vital to much of the zerg tech tree (coupling this with destroying the Lair or Hive is particularly effective).
When you're rushing a protoss player, it's best to take out pylons powering the production buildings. They're relatively weak and can knock out unit production for a minute, plus it throws off the protoss player's supply. After that the Cybernetic Core is good to knock out the tech-tree.
But what about terran players? Nothing seems particularly vital to a terran player. All of the buildings are redundant, cheap, and easy to replace. Their main tech tree (Barracks -> Factory -> Starport) is all buildings that they tend to make many of - there's no equivalent to the Spawning Pool or the Cybernetic Core where taking out a single building throws off the entire tech tree. You could destroy the Supply Depots to have a bit of an impact on unit production, but they don't give as much supply as Pylons, and there's no other benefit like knocking out a power grid as when hitting a Pylon. Add-ons are quickly and cheaply replaced.
So, what's the best building to hit when you're attacking a terran player (I'm specifically wondering early game, but later game information is good to know, too. Also, if anyone has better buildings to hit when you're hitting a zerg or protoss base than what I suggest in this question, do share.)