Pro players seem to have some sixth sense when counter-warding, but I always seem to fail. For observer wards, you can try the usual ward spots (runes/rosh/hills).... however, if your five-man push fails because the enemy saw you coming, there are a lot of possible ward spots you may have walked by. And how about countering sentry wards to protect a riki/gondar/brood?
It is pretty complex answering here without posting a tons of images. So I copy only the summary picture and suggest you to read this well written guide: Wardposition Guide on PlayDota. It contains a lot of in-game screenshots that explain in detail the positioning.
- Red Pins: Wards with runesight
- Orange Pins: Wards to block creep camps
- Pink Pins: Wards to monitor important places
- Blue Pins: Wards with basesight
The true answer is not to guess, but to know. Watch your minimap and your opponents' inventories constantly. In most games, it will be the enemy support heroes placing the majority if not all of the wards so you'll know where to focus your attention. If they are carrying wards you need to monitor their hero as much as possible and notice when that ward count drops by 1. When it does, you should know where they've been and where the common ward spots in that area are. If you've kept Observer Wards up yourself, you can determine exactly where they've placed it because doing so causes an animation. If you don't see this, you can usually narrow it down to one of a few spots and counterward appropriately. Most good counterwarding locations will reveal multiple frequently warded spots in an area- the reveal range of 950 is much larger than the vision range.
It's also worth noting that many of the common support heroes have spells that reveal high ground (allowing you to counter wards on hills or high ground that you couldn't otherwise see). AA's slow, Dazzle's ultimate, VS's howl, and others.
The other way to counterward requires a lot of experience and an expectation of high level opposition. Judging from how your opponents react to movement and gank attempts by your own team, you can tell when they've spotted someone moving between lanes and thus where they've warded. This is much more difficult and really only applicable if you're playing against competent opponents, weaker players are usally too sporadic for their reactions to be of much use. For example, I know that after a successful teamfight win in midgame by a good opposing team their supports will probably be spending the downtime placing offensive wards. With a combination of everything you've read above, I can make the chances of succesfully counterwarding worth the 100 gold chance to possibly catch a ward even if I don't know exactly where one has been placed.
As for the second part of your question, you will never really deliberately counter Sentry Wards unless it's to protect your own Observer Wards since doing so requires that the area be clear of opponents and thus their Sentry wards are unlikely to be used again anyway, they only last 3 minutes. If you really want to do so because your team is relying heavily on invisibility, a Gem is preferable because it will allow you to both reveal Sentries as you move around the map and also allow you to counter Observer Wards. This is only viable if carried by a hard to kill member of your team or the invisible hero they're countering, however. You don't want to die and give the other team the gem.