In addition to the excellent answers already here, I'd like to mention the relatively new SMAA, or "Subpixel Morphological Antialiasing", which is, like FXAA, a post-process antialiasing method. In many cases it outperforms other AA methods in terms of the image quality/cost ratio.
WHAT IS SMAA?
In the words of the developers Jorge Jimenez, Jose I. Echevarria, and Diego Gutierrez from the Universidad de Zaragoza, and Tiago Sousa from Crytek,
SMAA is a very efficient GPU-based MLAA implementation, capable of
handling subpixel features seamlessly, and featuring an advanced
pattern detection & handling mechanism. (source)
Or, more extensively (with my emphases):
We present a new image-based, post-processing antialiasing technique,
which offers practical solutions to the common, open problems of
existing filter-based real-time antialiasing algorithms. Some of the
new features include local contrast analysis for more reliable edge
detection, and a simple and effective way to handle sharp geometric
features and diagonal lines. This, along with our accelerated and
accurate pattern classification allows for a better reconstruction of
silhouettes. Our method shows for the first time how to combine
morphological antialiasing (MLAA) with additional multi/supersampling
strategies (MSAA, SSAA) for accurate subpixel features, and how to
couple it with temporal reprojection; always preserving the sharpness
of the image. All these solutions combine synergies making for a very
robust technique, yielding results of better overall quality than
previous approaches while more closely converging to MSAA/SSAA
references but maintaining extremely fast execution times.
Additionally, we propose different presets to better fit the available
resources or particular needs of each scenario. (source)
HOW DOES IT WORK?
SMAA is based on MLAA and FXAA:
MLAA (Morphological AA) and FXAA (Fast Aproximate AA) are post AA modes
that use blur filters. First, they detect contrasts ("edges") in the
frame and then blur it along the gradient.
This results in highly reduced visible "jaggies" that also covers
alpha-textures, but it also blurs everything, including textures. It is
also the cheapest form of AA and often used in console version of
The detection of edges of SMAA, however, is comparable to that of SSAA:
The alisasing "detection" is upgraded and is closer to the detection
used in MSAA then the detection used in MLAA and FXAA. The result is
that SMAA still remains very cheap, still smoothes alpha-tectures and
still greatly reduces the visible "jaggies", but doesnt blur the image
as much. (source)
This post by SomeoneSimple on the Anandtech forums explains briefly what SMAA does, and how it can work in conjunction with MSAA:
SMAA Injector on its default 'SMAA_PRESET_HIGH' is just a bit more
GPU-intensive then plain FXAA, so performance loss is pretty
negligible in most cases.
SMAA doesn't compare to MSAA though. MSAA actually adds
(subpixel-)detail to smooth polygon edges, SMAA only tries to hide
jaggies (but does a good job doing so). Most noticeably, you'll still
have pixel-crawling/flickering with SMAA.
However, SMAA does support being used in combination with MSAA. That's
the 'subpixel' part of its name, unlike any other post-process AA
filter (i know of), it can correctly detect gradients, thus it can be
used in conjunction with other forms of non-PP AA, like MSAA and SSAA.
That said, I'm currently using 1.2x OGSSAA (NV custom resolution
override) with SMAA injected, in games that don't play nice with MSAA.
SMAA takes care of anti-aliasing, and SSAA greatly lessens the amount
of pixel-crawling. It looks gorgeous, far better than only 1.2x OGSSAA
or SMAA on its own.
I will not go into further detail here, as the other answers have already provided background information on the mechanism of different AA techniques.
You can read more about this specific method in their paper (PDF); or by leafing through their slideshow presentation (direct link); or look into the code on GitHub.
COMPARISON WITH SEVERAL OTHER AA TECHNIQUES
A few screenshots that compare SMAA with other antialiasing techniques:
A video comparison using Crysis 2/CryEngine 3, that implemented it for the first time (Youtube):
Here is a direct link to the original video, unspoiled by compression.
And the following is a comparison with some MLAA, SRAA and FXAA methods (sourced from the paper linked above):
A version of SMAA, made injectable by Andrej Dudenhefner, can be downloaded here, and injected into many DirectX 9/10/11 (x86) applications.
This article describes how to do so in Fallout: New Vegas, but this method is very similar in most Bethesda games. It also provides some more information comparing and illustrating the different AA techniques.