Fancy Shaders - Part 3: Deformable Surface with Tessellation

Snow deformation in "Assassin's Creed III" by Ubisoft.
Image is taken from lanoc.org

Hey! This is another post in Fancy Shaders series and I want to touch another cool topic - surface deformation. It's not something that is being used in every game these days but is getting more and more popular. So what's special about it?

Well, there may be certain situations when we want to change the surface of the ground and make it as realistic as possible. Just think about footsteps on snow or sand! Many games had been using different tricks for achieving that effect, but it looked fake most of the time. However, since DirectX 11 has introduced tessellation, developers started using other methods for solving that problem. Today I'm going to shortly talk about a common and quite simple way to deform surfaces in real-time using DX11 and Unity. Of course, I'm not going to talk particularly about tessellation, because it's a big topic, but do not worry! Unity can do a lot of things for you! 

In general, with tessellation enabled in your pipeline, we can increase the number of vertices and polygons of a mesh by subdividing its parts. We need to do that if we want to move the triangles and reshape the mesh in order to fit the "collider" (foot, for instance). Fortunately, it's easy to enable tessellation with Unity's Surface Shader. For example, below is the implementation of distance tessellation from Unity's documentation:

Ok, we prepared our surface for deformation, but what's next? Have you ever tried to work with heightmaps? Actually, we can use the same method for making hollows! We get information from a heightmap texture and then subtract our normal*displacement from the vertex instead of adding. Alright, that was not difficult, but how to generate that texture in real-time? We can do that with one extra shader and a script. In a shader we are going to simulate a brush, like in terrain editors:

And then in a script, we do a simple raycast to the ground from our "object-collider":

It's not really optimized, as performance will drop if we do raycasts from a huge number of objects, but even like that it looks pretty neat in Unity!

So you can actually use that idea for simulating sand, mud or other similar surfaces... What I showed you is a basic setup and there may be a lot of improvements. Personally, I can  think of these games with something similar: "Batman: Arkham Origins" by WB Games Montréal, "Rise of the Tomb Raider" by Crystal Dynamics and Eidos Montréal, "Assassin's Creed III" by Ubisoft and one recent title - "God of War" (2018) by Santa Monica Studio. I highly recommend reading their papers if you want to improve your surface deformation, because they described their own ways and optimization guidelines for this feature.

Links:

1) https://www.gdcvault.com/play/1017710/Rendering-Assassin-s-Creed

2) https://www.gdcvault.com/play/1020177/Deformable-Snow-Rendering-in-Batman

Other useful resources:

1) "Deferred Snow Deformation in Rise of the Tomb Raider" by A.K. Michels and P.Sikachev, from GPU Pro 7

Fancy Shaders - Part 2: Shell Rendering

Cg Fur Shader with different parameters in Unity.

It's been a while since my last post, but I have something interesting for you:) You will see another cool trick that is easy to do with shaders anywhere you want. We will talk about fur in real-time computer graphics!

Obviously, rendering realistic fur in real-time is not a trivial task. Just think about it: somehow we need to create a huge number of fibers on the surface and also control them dynamically (for wind, gravity, external forces, etc.) Several solutions exist for that problem, so let's analyze them!

First of all, we can use a "brute-force" approach, which is creating primitives for every single fiber, such as a line or triangle strips and then manipulating them. Our fur then will be super realistic, but very inefficient to compute. Although we can use geometry or tessellation shaders, our performance is going to be really poor even on modern machines with high specs. Actually, something similar is being used in CGI, as real-time simulations are not required for that.

The second approach is billboarding. Even if you are not familiar with this technique, you will get the idea pretty quickly. Game developers have been using this method for ages when grass or objects in distance are to be efficiently rendered. Even these days billboarding is the main approach for rendering grass (that is similar to fur) in most of the games. We just create 2D sprites of textures in 3D world space and rotate them according to our camera position. Developers can achieve really decent results while rendering grass that way and save a lot of resources, of course. But what about fur? Unfortunately, it will be problematic to render fur with billboards. Fur can be spread on very rough surfaces, like the skin of an animal, and not on a "plane" terrain as grass. That is why billboards may have incorrect angles and players can notice those artifacts. However, I'm sure that some teams can get pretty cool results with that technique if it is set properly.

The last approach that I can think about is the "classical" way of rendering fur in real-time applications. A technique is quite old, but still can be noticed in modern games. It's called shell rendering that is a form of volumetric rendering.

"Conker: Live & Reloaded" (2005)

The idea behind shell rendering is also easy to understand and to implement.  We basically make bigger copies of our mesh which is similar to normal extrusion. It is also not that difficult to control the length of our fur! For example, we can use a mask texture and store information in it. What I did in the scene (top image) was just use a grayscale noise texture that functioned like a heightmap. I'm pretty satisfied with the result, however, you can also do that with a control mask texture and get the values from an alpha channel. Because the mask is created from your pattern texture, your fur won't be random anymore. Instead, you will have distinct regions of fur or its absence. For instance, some animals' skin has certain patterns so it will add realism to your application. Down below is the logic of our Cg shader:

Taken from XBDev.net

Without a doubt, this method is not perfect. As you may have thought, we are dependent on the number of copies of our model. That means the more we instantiate, the more resources we spend and higher quality we achieve. For example, fur from the top image was created by making 20 passes and still has the evidence of those shells.  On the other hand, it requires a lot fewer computations compared to a "brute-force" approach. I'd repeat myself again: there are other industry solutions, such as NVIDIA HairWorks or AMD TressFX, but they do require resources and not every machine can run them these days. Although these approaches are way more realistic and flexible, on mobile platforms we are still limited to shell rendering.

Finally, I want to show you how fur quality has changed over the last 10-15 years in video games. I have specifically chosen an example of a game that uses a lot of fur in it. It is called "Shadow of the Colossus" and was originally developed by Team Ico for PlayStation 2 in 2005. It definitely used shell rendering at that time. However, in 2018 Bluepoint Games released a remastered version of that beautiful game for PlayStation 4. They combined modern technologies with powers of the system and, of course, changed the look of fur in the game. I'm not even sure what was the approach that they'd used, but probably it was not shell rendering. My guess: they did some tricks with something that is similar to billboarding. What do you think?

"Shadow of the Colossus" (2018 - top, 2005 - bottom)

Links:
1) http://www.xbdev.net/directx3dx/specialX/Fur/index.php
2) http://developer.download.nvidia.com/SDK/10.5/direct3d/Source/Fur/doc/FurShellsAndFins.pdf
3) https://dl.acm.org/citation.cfm?id=617876
4) https://forum.unity.com/threads/fur-shader.4581/
5) http://hhoppe.com/fur.pdf

Other useful resources:
1) "Unity 5.x Shaders and Effects Cookbook" by A.Zucconi and K.Lammers
2) "Implementing Fur Using Deferred Shading" by D.Revie, from "GPU Pro 2"
3) "Animated Characters with Shell Fur for Mobile Devices" by A.Girdler and J.Jones, from "GPU Pro 6"

Fancy Shaders - Part 1: Oil Interference

Final Cg Shader in Unity

I have decided to start a series of posts called "Fancy Shaders". From time to time I'll be posting some interesting tricks that you can do with shaders. Each post will have a bit of theory and explanation and, of course, source files. My main tool is Unity, as it will be easier for you to implement shaders there:)

The first post in "Fancy Shaders" is about interference. That's a pretty cool topic from wave physics which can be used in computer graphics as well! The effect is showed in the GIF above. Basically, it can also be used for puddles, some painting materials, etc. By the way, if you want to read more about diffraction & interference with Unity examples, I highly recommend you to check out the posts by Alan Zucconi. Also, another great source was this awesome book:

"Graphics Shaders: Theory and Practice, Second Edition"
by Mike Bailey and Steve Cunningham

  

Firstly, let's start with some light theory. I will just remind you some things from your high-school physics course:) Iridescence is a phenomenon that appears to change object's color when the view or illumination angles change (yeah, like in soap bubbles). We also know that light is a wave. More specifically, light is a disturbance in the electromagnetic field. A photon is a particle, the quantum of that field. It carries the energy which describes the color of light. As light is a wave, it has some wave's properties, such as wavelengths. We can also represent the energy as a wavelength, so different energies have different wavelengths and thus there're different colors (but the speed of light is constant). Take a look at this table:

Alright, now the question is how to convert this specter to a common RGB palette? There is no straightforward solution for that, but there are many functions which can approximate RGB values for wavelengths. I won't go into the details, as Alan Zucconi explains (here) that topic very well, so I will just use his method. I find it really decent and optimized for our purposes. Because you should understand that we must keep our shaders as simple as possible: i.e. putting a lot of branch-statements is not a good practice!

Ok, we converted the spectrum "somehow". Now to keep things a bit more clear we need to apply this approximation to our color in the shader. But we do not have anything for our function's input parameter - wavelength. Let's then move to interference. Again, to make it simple, I won't dive deep into physics. You can understand interference as a process of getting a new wave from two light waves that are reflecting from the surface (oil/water, let's say). To get a clear understanding take a look at this picture:

Also, for a good approximation, one incident ray is definitely not enough. We need to set a number of multipliers for our function and then loop through them. Another thing to mention is the refractive index which determines the deviation of the light angle. If you look at the picture, you will see that light changes its angle when hitting the oil surface. These refractive indexes are different for various surfaces. (air/oil is about 1.4) Down below is the formula that can be used for calculating the wavelength using the things above.

The final thing is d - the height of our refractive surface (in our case its oil). How can we calculate it? Well, we can use a noise function for that! "Graphics Shaders: Theory and Practice"  has an interesting approach: we multiply the height value (from the input, for instance) by an exponential height of the "hump" that we get from the noise texture. In fact, we assume that our oil surface has the shape of this "hump". That is why we can see all colors from wavelengths of different values.

Now we can calculate d:

And that's basically it! Now you know how we can achieve this cool interference effect with a couple of shader instructions. In addition, there is a lot of space for playing with variables and their values. Many things are editable so you can set everything for your special needs. You can use different noise textures for the pattern, too. And one more time I want to mention Alan Zucconi and his awesome blog. I recommend to go there and clarify all the things you did not understand or I did not mention here! Because the purpose of this post was to give a basic overview of this light phenomenon and its usage with shaders.

Once you have an understanding of the process, you can download my Cg shader here.

Links:
1. https://www.alanzucconi.com/2017/07/15/the-nature-of-light/
2. https://www.alanzucconi.com/2017/07/15/improving-the-rainbow/
3. https://www.alanzucconi.com/2017/07/15/improving-the-rainbow-2/
4. https://www.alanzucconi.com/2017/07/15/understanding-diffraction-grating/
5. https://www.alanzucconi.com/2017/07/15/the-mathematics-of-diffraction-grating/
6. https://bit.ly/2ougyTW

Simple image processing with Xenko Engine

This will be a very short post, as I only want to introduce you Xenko Engine. It's a modern, open-source, 2D/3D cross-platform engine by Silicon Studio that has a plenty of cool features: scene editor, VR support, physics, animation, PBR and many more. Xenko is very similar to Unity because it also uses C#, so working with it wouldn't be painful. In addition, it has its own Xenko Shader Language that is based on HLSL. That's why I'd like to show you a couple of small shaders which I implemented while playing with the engine last night. They're just simple color tranformations: inversion, thermal vision and posterization. Nothing special, however, I find it kind of useful in order to understand how things work :)

Ok, so let's start with something difficult - color inversion... Of course, it's ridiculously simple! We just subtract our input color values and that's it :)

Inversion effect

Then we move to the thermal vision effect. You should understand that it's still a "fake" simulation of heat vision. However, it might be useful in some cases. Again - super simple: make gradient, calculate pixel's luminance and change it's color using linear interpolation. Here's the code:

Thermal Vision

Lastly, posterization effect. It is a process of a continuous gradation of tone to regions of fewer tones, with abrupt changes. Shader itself is quite simple, too, although, you may play with some settings: gamma value and number of colors.

Posterization Effect

If you want to add your own custom transform shaders, some .cs scripts should also be included. You can read more in Xenko's documentation. Don't worry, I will put everything into an archive down below. Oh, and by the way, some people may notice that I am inheriting Texturing on top of my shaders but I do not use any textures. That's true! I just wanted to show you that it is possible to create your own custom textures and implement more complicated and fancy effects, such as blur, for example.

Link to the archive: download

Inside "Alien: Isolation" - Graphics

"Alien" franchise is by far one of my favorite sci-fi movie series ever! That's why I was so thrilled when a video game "Alien: Isolation" came out in 2014. Guys from Creative Assembly have done an awesome job because the game looks and plays not only well but also recreates the vibes of 1979's cult classic "Alien".

Some time ago I found Adrian Courrèges's blog on the Internet with his "graphics studies" of different games and I wanted to do something similar since then. At the time of discovering new stuff in computer graphics, I decided to open up a popular profiling tool RenderDoc and perform some analysis of "Alien: Isolation" graphics. But I should mention that it was my first experience with profiling such a big game, so don't expect anything deep and meticulous! All I wanted to do is to dissect some frames and see how things were rendered. I will also mention some techniques that Creative Assembly used in the game, and that were then shown in the article by AMD.

First of all, the game uses Deferred Rendering which allows putting numerous light sources into the scene. I have touched that topic a quiet ago, so you should have a basic understanding of the process. One cool thing that the developers of the game came up with is working with multiple materials. We all know that if we want to use different BRDFs it can be problematic to do with Deferred Rendering. But "Alien: Isolation" has a nice solution: stencil buffers mark objects with unique materials and then lighting uses multipass while doing some culling for these objects. So basically if some materials are out of the player's FOV, they are just not used in the pipeline.

Let's see how the lighting is rendered in the same scene step by step. If you played the game, you know it's full of projectors, flashers and other sources of light, despite the fact that it is set in dark environments most of the time. Everything looked amazing because of indirect lighting, radiosity lightmaps and emissive surfaces (displays, buttons, LEDs and etc). Sorry, I'm too lazy to implement a decent slider script, that's why look at the GIF I made and try to remember the details :)

Shortly the process is following: emissive surfaces -> data from the lightmap -> more data plus the sunlight (or maybe not the sun, as we're in the deep space!) -> volumetric fog -> and after some post-processing - a final image.

Simply gorgeous! I really like it! And I hope you as well:) After the scene has rendered everything in it, passes for UI are coming. I haven't noticed anything special in them: just some textures with HUD and stuff like that. So that's it for this scene. But! Let's move to another one... Of course, I'm not going to repeat everything again, because I'm only interested in one classic and geeky thing from the franchise - a motion tracker!

Obiously, nothing but only overlayed lines and elements that are mapped to 256x256 textures with emissive light, lol. But mind that it's interactive, with motion detection. In that case, it is a little bit more interesting in terms of game mechanics (not graphics though) as it adds a moving dot on the display that tells you where the danger is. At that moment, there was simply no alien around...

The game also has a DLC "Crew Expendable"  that features the characters from the original film, including Ellen Ripley. Let's take a quick look how her model was rendered on the screen. I should state that a proper skin rendering was definitely used, because, for example, a subsurface-scattering pass is there.

I already told you that the lighting in the game is incredible. Besides, VFX are also stunning! Sparks, fire and smoke particles are physically-simulated on GPU using DirectCompute. Thousands of them can be rendered simultaneously with different properties. For example, in the scene below there are 1092 vertices in the pass to render these fire particles (of course, flames themselves are just textures).

I suppose that is everything I wanted to show except the shadows. I haven't talked about Contact Hardening Shadows that were used in the game, but you can find an original paper if you are interested. You can also read about the texture compression that was mentioned in "High-Tech Fear - Alien: Isolation" article by AMD. Oh, and check out the game, of course, if you haven't played it yet:)

Links:
1. https://community.amd.com/community/gaming/blog/2015/05/12/high-tech-fear--alien-isolation

Spherical Harmonics in Graphics: A Brief Overview

Visual representation of Spherical Harmonics

 

Global Illumination is a very popular and demanding topic in computer graphics. In particular, real-time graphics developers have been struggling with it for over 10 years now, still optimizing and implementing new techniques for modelling indirect lighting. Today I want to take all the articles, blogs and papers together and review one very efficient method of storing light information - Spherical Harmonics, or more precisely, the method that actually encodes the light data. For example, there can be a common situation when light probes should gather data and SH will help to compress it (i.e. Unity is working with the same approach).

Example of a scene with light probes

Spherical Harmonics are quite difficult to understand without proper math background, so that is why I will try to describe them as easy as possible here. First of all, you may ask why the heck do we need these harmonics at all? Let's take a look at the main rendering equation:

Arrghh... You said "easy"!!!

Don't worry, it can be simplified for our real-time purposes to something like this:

Lambert diffuse without shadows

So we need to solve this integral by finding L(x,w) and max(N*w, 0). That's where exactly SH are going to help us. They can be used as projecting inputs for our 2 functions from the integral above. But how? Let's find out! This intensity function can be estimated with Monte Carlo Integration. I'm not going to dive into the details (its a topic of probability), but the overall idea is that we can approximate our integral by calculating a dot product of 2 SH coefficients. Actually, this property of SH  makes them so efficient!

Now let's quickly talk about Spherical Harmonics themselves to get a better understanding of the process. They come from solving Laplace's equation and describe a set of orthogonal basis functions in spherical coordinates. Basis simply means that these functions can be scaled and combined together to approximate an original function (yeah, like Fourier). Orthogonal (polynomials) have a unique feature: when we integrate the product of any 2 similar of them we get a constant and when we integrate 2 different - we get 0. There are several polynomials of this type, but we are going to work with Associated Legendre Polynomials (just read about them on the Internet). So in general SH then look like this (where l is an integer (band index) that is >=0 and m is 0<=m<=l):

In addition, I want to mention that in real-time graphics 2 and 3 bands are enough, as they already give us desirable results.

That is it! Finally, we can compute our approximated integral using the equation above. Actually, there is a lot more to mention about SH, for example, rotational invariance, zonal harmonics or the fact that this method is only good for low-frequency compression. This time I've only tried to make an overview of Spherical Harmonics! So if you are interested more, you should definitely read the original posts and publications down below. I will also attach the link to a series about another similar, but newer, approach - Spherical Gaussians used in The Order: 1886 by Ready at Dawn. Now check out the shader by Morgan McGuire:

                                                          

Links:
1. http://silviojemma.com/public/papers/lighting/spherical-harmonic-lighting.pdf
2. http://simonstechblog.blogspot.nl/2011/12/spherical-harmonic-lighting.html
3. http://www.ppsloan.org/publications/StupidSH35.pdf
4. http://www.geomerics.com/blogs/simplifying-spherical-harmonics-for-lighting/
5. https://mynameismjp.wordpress.com/2016/10/09/new-blog-series-lightmap-baking-and-spherical-gaussians/

Experimenting with Shadow Mapping (SSM, VSM)

This time I'd like to show you how shadows are made:) Especially I'm going to focus on the technique called Shadow Mapping. Well, this method is quite old (it was introduced in 1978), but still very popular in modern real-time graphics even these days.

Demo scene. FBX models by dhpoware

Basically there are 2 steps of rendering the scene with Shadow Mapping. First of all, we should render our scene from the viewport of the light in order to create a depth map and store the depth values of the occluders. That depth texture will store the distance from the light to the closest vertices that are in the view of the light's "camera". And secondly we render our scene normally with projecting the depth map on it. After that we should be able to calculate the distance between each vertex and the light source for defining whether the vertex is in the shadow or not. If the distance is greater than the distance from the depth map, there's another vertex between our taken vertex and the light, so our current vertex is in shadow. It sounds not very complicated in theory, but in practice we will face several problems. Our shadow map is a texture, so logically it has the limited size and precision. That is why our calculations of the distance can be inaccurate. Look at these ugly artifacts below:

Shadow acne

This phenomenon is called shadow acne and is a precision problem. We can easily fix it by subtracting some value (that is called bias) from the actual depth. Our results will be much better, but keep in mind that a bigger bias value might cause another problem - peter-panning. (Actually, DirectX hardware has solutions for the bias problem, such as Slope Scaled Depth Bias.)

Acne is fixed, but now shadows are "detached". (Peter Panning effect)

Ok, we got an idea of how shadows are made, but there is still one big problem. Remember that we are working with a projected texture? That means that we are going to deal with nasty aliasing. Although we can increase the resolution of our shadow map, this approach is not the best solution for removing the jagged edges as they still won't be smooth. (even for modern GPUs)

So we have to think about filtering the shadow somehow. Luckily, there are a lot of different algorithms for smoothing shadows, like Percentage Closer Filtering, Exponential Shadow Maps, Variance Shadow Maps, Convolution Shadow Maps and etc. This time I'm going to mention only two techniques: Percentage Closer Filtering and Variance Shadow Maps.

PCF is a quite old algorithm that had been popular for many years. The idea behind it is following: we sample an area of texels and calculate a shadow percentage. For example, for 2x2 texel region we average 4 values for smoothing the shadow. (You can read more about PCF in GPU Gems, Chapter 11). So in our HLSL shader we then have something like this:

You can change the region size and get better results. Compare 2x2 and 5x5 PCF. In addition, besides standard PCF you can use more sophisticated methods of filtering, like Poisson PCF, Gaussian Blur or something else. 

2x2 PCF

5x5 PCF

But what if we want to improve our shadows quality using an easier approach with less samples? Well, then we should consider a different shadow map technique, Variance Shadow Maps. I won't get into the details (you can read the original publication), but just give you an overview. The implementation is pretty straightforward. We will work with mean and variance of distribution of the texel and then estimate the probability of being in shadow. So for this we will use two channels of 32-bit texture (red and green) to store depth and depth^2. Previously we used only one channel in our pixel shader. Now let's create 2 moments M1 and M2, where E(x) is the expected value of distribution, x is depth and p(x) is the filter weight. Then we can get the mean and the variance:

   

  

After that we can calculate an approximated upper bound for our probability P(x) using Chebyshev's inequality that was suggested in the original publication:

Variance Shadow Mapping

On the pictures (by Matt's Webcorner) you can see the difference between Simple Shadow Mapping and Variance Shadow Mapping with filtering. Mind the fact that you should use a different filtering algorithm with VSM, such as Gaussian Blur, to achieve the proper results.

Simple Shadow Mapping

This method avoids the problems with bias, such as acne and peter panning, but, unfortunately, has its own disadvantage - light bleeding. It happens when there's a big depth range between occluders/receivers, which results in apparent artifacts on the shadows. Light bleeding can be fixed by increasing the shadow value with exponent, which is a relatively heavy operation as it requires high precision. Anyway, in general VSM gives very good results despite it's costs. In conclusion, here's the shader code snippet that can be used for implementing the technique.



Links:

1. http://graphics.stanford.edu/~mdfisher/Shadows.html

2. http://www.punkuser.net/vsm/vsm_paper.pdf

3. https://mynameismjp.wordpress.com/2013/09/10/shadow-maps/

4. https://developer.nvidia.com/gpugems/GPUGems/gpugems_ch11.html

5. https://developer.nvidia.com/gpugems/GPUGems2/gpugems2_chapter17.html

6. https://developer.nvidia.com/gpugems/GPUGems3/gpugems3_ch08.html

7. http://www.dhpoware.com/demos/xnaShadowMapping.html