How To Choose the Right Anti-Aliasing Mode for Your GPU

By Matthew Braga

As any serious gamer will tell you, you don’t want jagged edges. When it comes to anti-aliasing, we’re here to demystify the jargon, and show you how each process works.

Old games had it good. There were no shaders, or lightmaps or anti-aliasing woes – just two dimensions of graphical wonder, fueled by pixelated style. It was blocky, sure, but that was all there was. Who knew we’d come to loathe those jagged edges we once viewed with awe?

NVIDIA has a good comparison of what anti-aliasing does to otherwise jagged pixels. 

This is where anti-aliasing comes in. It’s a technique used to smooth otherwise jagged lines or textures by blending the color of an edge with the color of pixels around it. The result should be a more pleasing and realistic appearance, depending on the intensity of the effect. In fact, modern 3D gaming has relied on some form of anti-aliasing for over a decade now, first appearing in the now-defunct line of 3dfx Voodoo cards. We’ve come a long way since those early days, however – but that doesn’t mean things are any easier to understand.

Supersampling or Full-Scene Anti-aliasing (FSAA)

This is the oldest form of anti-aliasing, and fairly rudimentary compared to more modern techniques. With this approach, individual pixels are divided into multiple coverage samples. By analyzing the color of the pixels surrounding each of these samples, an average is produced, which determines the original pixel’s over-all color.

However, analyzing a single pixel is no good for making precise corrections to an edge or line; for supersampling to work, a game is rendered at a much higher resolution than what is being displayed on screen. For example, rendering an image at 4xFSAA would render an image four times as large. This allows more precise color data to be deduced from each sample, and results in smoother lines and edges than usual.

in every frame – before it is downsampled back to the intended resolution. This means a card must effectively render every scene four times over, which isn’t terribly effective in terms of performance. 

Multi-sampling Anti-aliasing (MSAA)

While FSAA filtering may have been inefficient, the concept of analyzing individual pixels for coverage and color samples was a novel idea. It just needed to be implemented differently. With MSAA, instead of sampling one pixel on a much larger scale, two or more adjacent pixels are sampled together while rendering an image at its intended size. Because multiple pixels are being sample together, coverage points can be shared between them. For example, at 4xFSAA, a single pixel would require four different sample areas. But at 4xMSAA the same amount of sample areas can be split amongst two or more pixels. The advantage is that a coverage samples are reused for adjacent pixels – so, in other words, if you have a large group of similarly colored pixels, not every pixel will need to be analyzed, freeing up computational power for other tasks. 

Of course, because not every pixel is analyzed equally, MSAA isn’t quite as accurate as conventional FSAA – but it’s pretty damn close. For example, while MSAA is good at smoothing the edges and lines, it isn't as effective at smoothing textures or color detail. Nevertheless, this is a considered an acceptable tradeoff considering the performance gained, and as a result, MSAA is basically the norm within games these days; numerous tests have shown that AMD and NVIDIA cards both perform similarly where this style of smoothing is used.

Coverage-sampled anti-aliasing (CSAA) and Custom-filter anti-aliasing (CFAA)

But for all its benefits over FSAA, MSAA is still an old style of pixel smoothing, and doesn’t take advantage of modern hardware as efficiently as it could. Thus, both NVIDIA and AMD have developed their own successors – CSAA and CFAA respectively.

Both techniques work in a similar manner, in that they can both store more coverage information about a pixel without increasing strain on the GPU. For example, with MSAA, if a pixel is sampled in four different places, four different pieces of color information are stored. However, when you move to 8xMSAA, color information is now being saved for all eight of those coverage samples. Inconsequential at first glance, perhaps, but not when applied to thousands of pixels per frame. That’s a lot of processing bandwidth. 

Initially, AMD's CFAA filtered edges a little too well. In this screenshot from Xbitlabs, the fence should actually continue into the horizon, but instead appears invisible.


The ‘Q’ stands for Quality

It’s a common misconception that, the higher the anti-aliasing multiplier, the smoother your game will appear. But as you’re about to see with NVIDIA’s CSAA, that isn’t always the case. If you recall, CSAA is optimized to look very similar to MSAA, but place less load on your GPU by sampling fewer colors per coverage area. This is great in terms of performance, but can result in decreased color accuracy and video quality. Thus, for those wanting the best of both worlds, there is a special ‘Q’ option under certain CSAA modes. 8xQ CSAA, for example, increases the number of color samples from four back to eight – just as you’d get with 8xMSAA – but with less of a performance hit.  

 An example of AMD edge-filtering technique — better at smoothing edges while maintaining the same number of coverage samples.

Similarly, AMD’s CFAA filter can also be misleading. Though Catalyst lists the companies edge-detect technology as available via 12x and 24x options, neither is capturing any more data than other anti-aliasing techniques. Both are actually similar to 4x and 8x MSAA, but with AMD’s edge detection algorithm providing improved line filtering – though the same amount of color and coverage samples.

What does it all mean?

Next time you load up your favorite game, don’t be afraid – anti-aliasing is there to help, not hinder. Everything may sound like a great deal of jargon, but at its most basic, all anti-aliasing works the same, smoothing jagged edges for a more realistic look. The challenge is finding what works best, not just for your game, but your graphics card too.  

What type of anti-aliasing do you use, and do you find it makes a difference? Do some games perform better with different settings than others? We want to know what you think. Be sure to share your thoughts in the comments below.
Diagrams via NVIDIA, AMD.