{"id":1535,"date":"2020-03-30T20:33:35","date_gmt":"2020-03-30T12:33:35","guid":{"rendered":"http:\/\/blog.coolcoding.cn\/?p=1535"},"modified":"2020-09-22T15:49:01","modified_gmt":"2020-09-22T07:49:01","slug":"anti-aliased-alpha-test-the-esoteric-alpha-to-coverage","status":"publish","type":"post","link":"https:\/\/blog.coolcoding.cn\/?p=1535","title":{"rendered":"[Unity]Anti-aliased Alpha Test: The Esoteric Alpha To Coverage[1]"},"content":{"rendered":"\n

ref: https:\/\/medium.com\/@bgolus\/anti-aliased-alpha-test-the-esoteric-alpha-to-coverage-8b177335ae4f<\/a> <\/p>\n\n\n\n

AlphaToMask on<\/p>\n\n\n\n

Aliasing is the bane of VR. We have several tools we use to try to remove, or at least reduce aliasing in real time graphics today. Tools like super sampling<\/a>, temporal anti-aliasing (TAA)<\/a>, and multi sample anti-aliasing (MSAA)<\/a>. They all have their pros and cons, but there\u2019s one tool I don\u2019t feel like gets the use and respect it deserves. That tool is a feature of MSAA, Alpha to Coverage.<\/p>\n\n\n\n

This article will be assuming you\u2019re using forward rendering with at least 4x MSAA. If you\u2019ve chosen to use deferred rendering this article will be less useful for you. Some techniques are still applicable though. If you\u2019ve chosen to use TAA and forward rendering this may solve one of the reasons you chose that option!<\/p>\n\n\n\n

The Problem<\/h1>\n\n\n\n

Self Inflicted Aliasing<\/h2>\n\n\n\n

A very common case of aliasing is the use of alpha test, used for things like trees or bushes. Lets look at why it\u2019s used so much even though it causes so much aliasing.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

This is a basic plant mesh in Unity with a very basic lit alpha test shader. In Unity these are often called Cutout shaders. I\u2019m using the bush03 mesh and texture from Nature Starter Kit 2<\/a> on the asset store if you want to try this yourself. It doesn\u2019t look to bad, but there\u2019s a lot of aliasing there from the alpha testing. Note that the above image is <\/em>using 4x MSAA. That doesn\u2019t help here since this is shader aliasing which generally MSAA can\u2019t help with. This is something people usually fix with some extra super sampling or they give up on MSAA and use TAA instead. Or they just live with it.<\/p>\n\n\n\n

What about traditional alpha blending? Seems like that should be an obvious answer, so why don\u2019t people use that?<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

That looks way better, at least in this static shot, maybe a little blurry, but the aliasing is gone! Success, right? Of course it isn\u2019t that easy. In motion using alpha blending for this case looks really <\/em>wrong. <\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

It almost looks like the bush is inside out! It\u2019s even more obviously wrong in VR, so this isn\u2019t a usable option. So what\u2019s happening here? Alpha blended shaders don\u2019t get to use per pixel depth sorting like opaque and alpha test shaders do. The bush is made from a bunch of intersecting polygons. You could sort the polygons, but they\u2019re still intersecting so that only helps so much. So, that\u2019s why we\u2019re stuck with using alpha test. <\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

But there are <\/em>ways to have the benefit of both alpha test and alpha blend; a way to get per pixel depth sorting and the smooth edge. There are some two pass techniques\u00b9<\/a> that can work well. There are also some order independent transparency approximations, though they are potentially expensive. There\u2019s an even better option if you\u2019re using MSAA. Enter Alpha to Coverage.<\/p>\n\n\n\n

\n\n\n\n

The Solution<\/h1>\n\n\n\n

This One Simple Trick \u2026<\/h2>\n\n\n\n

Alpha to Coverage (sometimes referred to as A2C or ATOC) is a less commonly mentioned graphics technique. While used by several big games, it is seemingly unknown to the vast majority of smaller devs. It\u2019s a powerful tool that is often overlooked due to the requirement of MSAA. Games often treat MSAA as an expensive option for high end PCs. Or they\u2019ve used rendering techniques that don\u2019t allow for easy use of MSAA to begin with, like deferred rendering. The result is games have overwhelmingly preferred post process anti-aliasing like FXAA<\/a>. This made Alpha to Coverage even more uncommon for a while. But the rise of VR has lead to the resurgence of forward rendering and of MSAA. This means A2C has become a useful tool again, especially for VR.<\/p>\n\n\n\n

Anti-aliased alpha test is one of the most basic use cases for Alpha to Coverage. The basic idea for this has been around for as long as A2C has existed. It may even have been the original intended use for it. The first time I saw this idea was on Emil Persson\u2019s (better known to many as Humus) site here<\/a>. I believe the method in this article is superior to the one used in his original demo.<\/p>\n\n\n\n

Multi Sample Anti-Aliasing<\/h2>\n\n\n\n

Before we get into that though, I\u2019m going to quickly run down what Alpha to Coverage is<\/em> and why MSAA is a requirement.<\/p>\n\n\n\n

Alpha to Coverage maps the alpha output from a pixel shader to the coverage mask of MSAA. Obvious, right?!<\/p>\n\n\n\n

Okay, that\u2019s going to take a little more explanation. Be warned, I\u2019m going to get into some of the gritty details here.<\/p>\n\n\n\n

The big trick with MSAA and how it differs from super sampling is what Multi Sample Anti-Aliasing is doing multiple samples of<\/em>. Super sampling at its most basic is rendering everything <\/em>at a higher resolution. The final on screen image is a scaled down average of this higher resolution render. MSAA renders the color at the same resolution as the target resolution, but multiple coverage <\/em>samples per pixel. In simpler terms it\u2019s rendering the depth of the scene at a higher resolution than the target resolution. Each rendered triangle\u2019s depth samples are its coverage. If you\u2019re using 4x MSAA it\u2019s rendering 4 depth \/ coverage samples within the bounds of each pixel. Each coverage sample can store a unique color. But it\u2019s only getting a color by running the pixel shader once per triangle<\/em>, per pixel. Most of time all 4 depth samples will be the same triangle. In this case all 4 coverage samples use the same single pixel shader output color. This behaves no different than when having no MSAA enabled at all.<\/p>\n\n\n\n

Where MSAA\u2019s strength comes into play is when a triangle doesn\u2019t cover all the depth samples. For example, if a triangle only covers two depth samples the two matching coverage samples are set to the pixel shader ouput. The other coverage samples can store the color of other triangles\u2019s pixel shaders that can be seen.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

\n\nfrom https:\/\/mynameismjp.wordpress.com\/2012\/10\/24\/msaa-overview\/<\/a><\/p>\n\n\n\n

More in depth write ups can be found elsewhere, like on Matt Pettineo\u2019s (MJP) site.<\/p>\n\n\n\n

\nA Quick Overview of MSAA<\/a>\n<\/div><\/figure>\n\n\n\n

I\u2019m glossing over a bunch of details, but the main point is MSAA can store a list of multiple colors per pixel, one color per coverage sample. These get averaged together to make the final on screen pixel color. But it\u2019s not always having to render a pixel shader more than once per pixel resulting in significant savings over super sampling of similar quality.<\/p>\n\n\n\n

Alpha to Coverage<\/h2>\n\n\n\n

This is where Alpha to Coverage comes in. Alpha to Coverage let\u2019s the pixel shader\u2019s output alpha change the fraction of coverage samples rendered to. Hence the name. In the simple case of a triangle covering all 4 depth samples, all 4 coverage samples store the color output. With Alpha to Coverage enabled and an output color alpha of 0.5, only half of the possible coverage samples will store the color. The result will be as if it\u2019s half\u00b2<\/a> as opaque as it\u2019s only contributing color to half the samples!<\/p>\n\n\n\n

Okay, so with 4x MSAA and Alpha to Coverage you get 5 levels of opacity you can play with, 4\/4 through 0\/4 samples. What good is that? With traditional alpha blending you get at least 256 levels, so why would you ever want to use this? Because with Alpha to Coverage those coverage samples are opaque. That means they can be depth sorted! You can have a mesh of polygon soup and have intersecting triangles accurately sort at better than per pixel accuracy. This is no different than opaque objects or even alpha test with MSAA, but you also get soft edges.<\/p>\n\n\n\n

Lets just turn Alpha to Coverage on and see how that looks. In Unity\u2019s ShaderLab it\u2019s enabled by just adding AlphaToMask On<\/code> to an otherwise opaque shader that outputs alpha. So here we go \u2026<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

And we\u2019re done, doesn\u2019t that look so much better than before?! Lets compare it to the original alpha test version!<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Wait, no, Alpha to Coverage looks terrible! It\u2019s somehow simultaneously fuzzy and aliased, not to mention sort of noisy, and seems like the bush got more transparent through the middle? The story only gets worse when we get close up to those leaves <\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Oy vey, that\u2019s no good at all. The limited number of opacity steps makes for some ugly banding. Clearly there\u2019s a reason why this isn\u2019t used more often, so why would I even suggest doing this if it looks so bad?!<\/p>\n\n\n\n

Fixing the Solution<\/h2>\n\n\n\n

Well, because that\u2019s what happens if you use Alpha to Coverage with the alpha of the texture straight. This is what replacing basic alpha blending using alpha to coverage looks like. Many implementations of Alpha to Coverage I\u2019ve seen in games really do leave it here.<\/p>\n\n\n\n

But what\u2019s happening in that first alpha to coverage image to make it so bad? It\u2019s not just the banding, it\u2019s also the interaction between the coverage samples. In short it\u2019s a limitation of using Alpha to Coverage. Essentially when using Alpha to Coverage only \u201cone layer\u201d of a specific coverage can be seen at one time. Two or more overlapping triangles with 50% Alpha to Coverage results in no change in opacity. Only the closest coverage points get rendered. This is unlike alpha blending where they will blend over each other increasing the perceived opacity. If you remember earlier I said the coverage samples are individually opaque. If the output alpha is 50% with 4x MSAA only two coverage samples are written to. But it\u2019s going to be the same<\/em> two coverage samples for everything that renders to that pixel with 50% A2C! The answer might seem like \u201cwell, render to different samples!\u201d. But that\u2019s harder than it sounds to do and be effective. It is an option though. Or you might be thinking \u201cnope, still don\u2019t get it at all\u201d, and that\u2019s okay. Don\u2019t worry about it. It\u2019s unnecessary to understand for this particular case as it\u2019s relatively easy to solve.<\/p>\n\n\n\n

So what is the solution here? Alpha to Coverage is great for anti-aliasing alpha test<\/em>, not so much for replacing an alpha blend! We need to sharpen the alpha so there are fewer overlapping areas of partial opacity. This will simultaneously solve the banding, and interior transparency. Anyone who\u2019s done signed distance field text, or shader based line rendering probably knows whats coming next. For everyone else I\u2019m going to introduce you to one of my favorite functions in shaders:<\/p>\n\n\n\n

fwidth()<\/code><\/p>\n\n\n\n

With out further explanation we get this (I\u2019ll talk about it later)<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Now that\u2019s better. Looks very similar to the original alpha tested version, but is nicely anti-aliased. And it\u2019s not strangely transparent, noisy, or blurry anymore! Here\u2019s the original alpha tested version one more time for reference <\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

So, what did I do? What was that fwidth()<\/code> I mentioned earlier? All the magic is in this one line of shader code.<\/p>\n\n\n\n

col.a = (col.a - _Cutoff) \/ max(fwidth(col.a), 0.0001) + 0.5;<\/pre>\n\n\n\n
What that line does is sharpens the alpha of the texture to the width of a single pixel. The fwidth()<\/code> function returns the sum of the pixel\u2019s partial derivatives for a value, usually described as the equivalent of abs(ddx(value)) + abs(ddy(value))<\/code>. Those two functions, ddx()<\/code> and ddy()<\/code>, are partial derivative functions. They measure the change of a value between the two pixels either to the side or above\u00b3<\/a> the pixel that\u2019s currently being processed. Derivatives are wonderful, magical things in shaders. For the moment it\u2019s enough to know fwidth()<\/code> gives a good approximation of how much a value changes per pixel. If you divide a value by how much it\u2019s changing it\u2019ll rescale it to give you a nice crisp edge. The max()<\/code> is there because if the texture\u2019s alpha is a flat color at some point, like black or white, fwidth()<\/code> will be zero. Dividing by zero is bad! The 0.0001 is just an arbitrarily small number, small enough that you have to really get close to a texture before the banding will show up again. The _Cutoff <\/code>is the same value you would be using for an alpha test shader and helps center the alpha around the edge you want.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\n\nWe could end here, but another problem that will show up that most people will probably miss initially. It is an issue that plagues alpha test already, so people might just be living with it. As you get far away from an object using alpha test it\u2019ll appear to shrink and eventually vanish. Alpha blending does not suffer from this\n\n<\/p>\n\n\n\n
 We could end here, but another problem that will show up that most people will probably miss initially. It is an issue that plagues alpha test already, so people might just be living with it. As you get far away from an object using alpha test it\u2019ll appear to shrink and eventually vanish. Alpha blending does not suffer from this <\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
 The cause is the texture\u2019s mip mapping making the alpha fade out from averaging. Since the alpha is fading out that hard \u201ccutoff\u201d edge is vanishing too <\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
 There are several workarounds to this problem. Lowering the cutoff value (a common solution for alpha testing in general). Super sampling in the shader (I do this for font rendering). Or using some kind of noise like stochastic sampling or hashed alpha testing<\/a>. Probably the best solution is the one proposed by Ignacio Casta\u00f1o on The Witness blog back in 2010. It requires recalculating each mip maps\u2019 alpha values in a way that\u2019s aware of it\u2019s intended use for alpha testing or Alpha to Coverage. The Witness uses A2C for foliage as well as several other effects too. <\/p>\n\n\n\n
Computing Alpha MipmapsA little problem that we had when we started to create trees and vegetation is that the standard mipmap generation\u2026the-witness.net<\/a><\/p>\n\n\n\n
Computing Alpha Mipmaps<\/a><\/blockquote>