The composition of your team is the first line of defense against Anti-Mage. Having a good core of gank-tasked heroes (3+) and ideally, no more than 1 hard carry. With Anti-Mage, like many of the late-game carries (Spectre, Void, Antimage), your game should not fixate on countering him late game, but instead ask what counters Anti-Mage?
Wards, Ganks, and Pressure
You need to shut down these hard late-game carries early. Even allowing the completion of a single core item uninterrupted is a bad sign. Where you listed 3 high tiered items, the momentum for Anti-Mage was likely far out of your control to rein in late game.
So you're looking to gank him frequently. Note that if the enemy team has more than one late-game carry this becomes more difficult for both teams, where your team will need to keep both late-game heroes in check and from the enemy team's side they need to  not die, and  wait you out. So logically, you need to do the opposite, gank them far more often than you would consider necessary and push down towers.
While "kill them and push towers" is a bit too simple-minded of an approach, the crux of the issue is defined.
The problem is that in execution, as is commonly the case especially on randomly assembled teams, not many players risk leaving their lane, leaving behind their potential farm-able gold, and losing experience roaming from one lane to another.
But what is important here is to reframe the issue, again, and that is a successful gank does not always need to result in a kill. Pushing Anti-Mage out of the lane to hug the tower, deny his gold/experience gain, return to base to heal, or even just added pressure that you guys are coming for him stacks up for an effect in your favor. Understanding that 'yes he has blink' and 'no you probably won't kill him' you are primarily focused on disrupting his acquisition of gold and experience.
Laning against Anti-Mage should involve 2 heroes that can stun or disable, understand when picking a lineup against Void, Anti-Mage, or even Spectre, their escape mechanisms (blink, timewalk, dagger) are going to allow them to get away from most gank attempts. Understanding this, a gank would ideally involve a third hero circling into the lane to stun/disable first (unexpected) and then following up with the 2nd and 3rd from the heroes already in the lane stacking the disable duration for a kill and to prevent the blink out.
Tri-Laning (3 heroes in your lane against Anti-Mage from the start) - is a decent option as well, all 3 heroes need to have a stun, disable, or a decent slow skill for this to be appropriate. You're looking to press hard on the enemy heroes and push through their towers when they linger back. The general idea being heavy pressure and pushing down towers early and getting in the way of the usual progression a carry needs to farm/level.
Lion - an excellent ganking hero early game (even before level 6), leveling hex and stun, coupling Lion with any other hero (or two heroes!) is an easy combination for early harassment. Lion should also be looking to incorporate a
blink dagger into his build sooner than later, to quickly get the drop for ganks (few people stand around as Lion charges at you) and also in order to keep up with Anti-Mage when possible.
Blink in, hex, wait, wait, stun, wait wait. This is a long time to be disabled. You can blink to chase as well, but the key here is to get in his face before he realizes stuns are incoming so ideally you are blinking in and disabling before he can blink out.
(Rhasta could potentially be an option, though less recommended as his second disable, Shackles, requires him to be in the line of fire and also immobilized for the length of the disable, making it less reliable to get the full duration of shackle time)
Spectre - if you're looking for a late game carry that will stand up to Anti-Mage when a game has gone 50-60+ minutes. The problem here is that you are relying on a number of other factors:  that you can out-farm anti-mage,  the enemy team allows you to get to late game without too much gank interference,  the enemy team does not push through to your barracks before your core items are farmed.
Scythe of Vyse - nothing too surprising on this recommendation, hopefully. Later in the game you're looking for a sheep-stick on one or two heroes to help with a few more Hex casts as Anti-Mage becomes more difficult to kill outright.
Other items - most other items are going to focus on dealing damage, because while mass disables are good, you still need your carry/semi-carries to start taking down his HP (not necessarily with nukes).