Travis Brown: Good afternoon everyone. I'm Travis Brown, graphics and imaging evangelist. You've probably seen enough of me by now. But welcome to session 212 which is cutting edge OpenGL techniques. And we sort of have this as the last session, content session at graphics and imaging track this year because this is really sort of an interesting culmination of the GPU theme that we have running through our content at this year's conference. What we're going to do in the session is we're going to invite hardware partner ATI to really the latest techniques they use in terms of programmability and various different special effects that they can accomplish with their Radeon products.

They sort of really let you know what the outer boundaries of what you can do with visual effects, what the state of the art currently is. Because it's also very interesting, we've had sessions on programmability from vertex programming and fragment programming, and that was sort of an entree into this whole topic. The interesting thing is we're fortunate enough to really gain all this programmability and fantastic graphical power through the efforts of our hardware partners like ATI, who are really, truly out there creating the silicon that does these incredible digital effects. So it's my pleasure to invite Alex Vlachos to the stage to take you through the presentation. Thank you.

All right, good afternoon. How's everyone doing? Excellent. All right, so where are we? Sorry, my bad, we're already at that slide. So my name is Alex Vlachos. I am from ATI. I'm part of the 3D Application Research Group. My primary role at ATI is I'm the lead programmer and general team lead of the demo team. So we're responsible for making all the demos for the latest Radeon cards, the 9700 and beyond in the past. On my left is Rav, Rav Duraj. He's part of our Mac development team, and he's responsible for doing the Mac-specific code We're going to be showing a few different demos today running on OS X. And basically our demo engines, like some game engines, are cross platform, cross API. So our demos run on both -- on three different sort of combinations. We run on the Mac through OS X. We run on the -- and OpenGL. And we run on the PC through OpenGL and also on the PC through DirectX. So all our demos run on all three platforms consistently, which is interesting. So the things I'm going to be talking about today are these three topics. The first thing I'm going to talk about is Normal Mapper. So in the Radeon 9700 Car Paint demo, we used what we call a normal map and we have this normal map tool which will be available on the Mac soon, sometime in the next week or two. Then I'm going to talk about a few video effects and more scene post-processing effects. So as graphics and hardware gets more and more interesting and the APIs are there, you post-process your scene and do some really interesting things. So we're going to show some post-processing effects on live video. We just have a camera set up that was all handed out, the nice new camera. So we're going to be showing the effects on live video. Then the last thing we're going to talk about are the shadow techniques we used in the Animusic Pipe Dream demo. How many people have seen the Pipe Dream demo?

Okay. How many people have seen the Car Paint demo? A few. Okay. So we're going to talk about how we did the shadow techniques in there. We didn't just brute force do stencil shadow vinyls. We did some more interesting things. So we'll talk about those three things. All right, so the normal mapper tool. The normal mapper tool basically is-- it's a tool to generate high res normal match. It's a way to make your low res models look really high res. And so why don't we jump into the demo now, to the demo machine. And we got the--we're on demo machine two. Okay, thanks. So this is the Carpane demo. It's a nice Ferrari, the car I want.

Yeah, I think Carmack has this car. It's nice. So what you're looking at is this model looks very high-res, when in essence it's really not that high-res. Can we jump to hit number two? So this split screen shows the before and after. So the model on the left is if we just use the vertex normals of our model to environment map and to light this car. What's on the right is instead of using the vertex normals, we use our normal map, a bump map. And so it makes it look like your model is a lot more high-res than it actually is. In essence, this car has a couple thousand polygons, maybe 3,000 or 5,000 polygons, give or take. The model, what it looks like, it's generated from a model that's over a million polygons. So as we watch the hood of the car sweep by here, as you watch it sweep by, so on the hood you can really see the detail jump in. You'll see all this interesting shape that's actually not there. Great. Okay, so let's jump back to the slides. Actually, one second. Can you turn on wireframe for a second, Rav? Never mind. Let's go back to the slides. This is our beta build of the demo engine. We just got this port up and running last week, I believe.

OK, so normal maps. Normally when you generate a normal map, your artist will author a height map, the one on the upper left there. Just a height map, gray scale. Black, darker pixels refer to low height. Lighter pixels are high height. Based on that, you can pass a filter over and generate a normal map, which is an RGB map which basically encodes vectors, your normals, in the RGB channel. That's generally normal maps. That's how we're used to doing normal maps. Now there's this new method of doing normal maps, which a lot of games are starting to do that will be released soon. And there are a few games out there already that use this technique. But essentially what you want to do is your artists generate this low res model that's on the left here, which is about, this is our car model that's a couple thousand polygons. That's what we meant, that's what we're going to render at runtime.

That's the thing you just saw running. What's on the right here in the middle is this high res car model. That has all the detail. That model is like a million polygons, probably over a million. I forget the exact number. But what we do is based on these two models, we can generate a normal map, the one on the right there, that encapsulates all the detail from this high res million polygon model and we can then use that normal map that has all that detail in it on our low res model. And so by combining the first and the last thing there at runtime, we get what the car paint demo looks like. It's really this high res model that's not really that high res. So the normal mapper tool is basically this. We have this command line utility that will be available for OS X sometime the next week or two. In about a week or so. In about a week or so. It'll be on the developer website. Right. Okay. So it's basically a command line tool. And the tool takes in as input essentially this low res model, this high res model, and is going to output this texture. Now, a few points about the low res model is the one that we're going to render at runtime. So it needs, obviously, positions. It's going to need normals. And it's going to need one set of texture coordinates, which define how that normal map is set up, how it's actually mapped on there.

The high res model just needs positions and normals, because we're not going to actually texture map that. That's not for real time. That's just for this pre-processing stage. So essentially, we have that as input, we can output these normal maps. Now, here's some background on sort of how it works. So, we have this low res model, which is represented by these two blue lines here.

Just imagine this is sort of looking at edge on to two polygons meeting up here, right? And this red line is sort of what a high res model might have it in. So, the idea is that we basically interpolate the vertex normals of the low res model, right? Those vectors coming off the blue lines. We interpolate those vectors and then we treat them as rays and we cast rays out and intersect the high res model. And wherever we intersect that high res model, we then take the normal, that surface normal of the high res model and encode that into a texture. So this is what happens internal to the tool. One of the things you'll notice here is, again, this blue is the low res model, the red is the high res model, is it does support what we call negative rays, right? It doesn't have to, your low res model doesn't have to be, doesn't have to be inscribed in your high res model. They can intersect in different ways. So basically as you cast a ray out from right here, it actually, where it intersects is behind where the ray originated. So that's fine. That fully works. The other thing that we're going to do is we super sample these rays. So your normal maps look nice and anti-aliased, basically, right? You shoot out, we can do up to, I mean, we've done hundreds of rays per pixel. Essentially, you know, 36 rays per pixel, I mean, is roughly good enough to get some good results.

The tool runs pretty fast. And this slide is here just so anyone who's doing their own normal map development, making their own tool, this is how we sped our tool up briefly. If you think about the amount of rays, the amount of calculations that have to happen, brute force, it's a lot. If you have, think for a minute about a, if your low res model essentially has, doesn't matter how many polygons are there, but the texture that you're going to output is, say, 2,024, 2K on a side, 2,000 pixels by 2,000 pixels roughly, right? That's going to give you roughly 4 million individual pixels. For each one of those pixels, you have to cast a ray and intersect every polygon in the high-res model, which is, say, a million polygon model. That's a lot of rays. That's 4 million rays that have to do ray intersect tests with over a million polygons. It takes a long time. So we've done some optimizations to basically say for each polygon in your low res model, which polygons in my high res model potentially can I possibly intersect with? So we basically just came up with this technique to use sort of a skewed beam in a sense. Instead of having a beam, so a volume of three clip planes, we had these six clip planes that define this volume. So we just have this optimization there that for each edge in your low res model, you pair the edge with each vertex normal and come up with two planes. You do that for all three edges and you end up with these six planes. And any polygon in your high-res model that lives inside that volume, we actually test again. So that gives us a huge perf boost. So anyone writing their own tool for this for whatever reason, that's a good technique. Now, normal maps, some developers are using this to actually generate normal maps for their characters' faces, right? It's not just for cars and other things. They actually use this for their character models.

And for faces, the thing they realize is they don't want their artist having to model every little bump on the face, every little scar. They don't want to have to model all the micro detail. They want to use their normal maps and their normal mapper tool to generally get the overall nice smooth shape, to get the nice shape of the face without having to have all those polygons at runtime, right? So they want to use the normal mappers, the normal mapper tool that's built out of normal map to do sort of this more rough, sort of nice smooth looking geometry, right? This nice smooth looking bumps. And then they want to combine that with a height map. And in the height map that artists are used to painting up, they can put in there their scars, the little holes, whatever kind of problems, like little micro detail basically. And our tool combines these two things and gives us our final normal map that has this nice sort of different, like the micro detail, the rough detail that gets mixed in.

So here's a note for artists. Any artists that are gonna use Any tool that does normal mapping has to be aware of this. This is one of the most important points for artists, is that if you look on the left, our low-res model. Now, these black vectors coming off here are the vertex normals. Now, what happens is since we cast our rays based on the interpolated normal on our low-res model, if there's no normal that gets interpolated across a part of the high-res model, you'll never encapsulate that piece of the high-res geometry. So that blue circle up there actually gets completely missed. those pixels, that part of the high-res model never gets raycast into because there's no normals to raycast into it with.

What you want to do on the right here is have a shared normal up top as opposed to an unshared normal. Shared normals will allow us the actual tool to raycast into the entire high-res model and encapsulate all that geometry and all the curvature there. Another note for artists is texture map spacing. So normally when artists page textures, it's normally done for sort of the color of your character.

You want to have one texture on your entire character, and they page it so you have the head here and the arms there and whatnot. And having them be not too close or far apart isn't that big a deal because it's just color and it bleeds and it's not that terrible. When it comes to normal maps, though, it's really bad. You have to make sure that there's enough space in these textures so if your textiles bleed, the color of those textiles bleed, you're going to change your lighting equation So those are your surface normals that you're going to pull out of there. It's not just a color. It's a vector you're pulling out of there that's going to then have math applied to it. So essentially, the tool does basically this. So the artists have to leave sort of these gaps in here. If you look up here in the upper left corner, the back of the car, there's a lot of space there. On the right, this is the after. So we pass a dilation filter over this, and we basically dilate out. We basically grow out all the normals in the image to all the unused area. And this is important because as you fetch a textile from your texture, you're going to get a bilinear fetch, right? So whether you like it or not, you're going to be pulling outside of the actual area that is truly texture matte by the artist because you're going to get your bilinear fetch. So what we have to do is grow this out. So the filter on the right is our dilation filter. We take this filter, pass it over the image that the tool spit out, and grow all those areas in. So if you look at the back part of the car here, you can see on the upper left this big hole gets mostly filled in there. We pass this filter over a couple times. And this helps everything look right in the end. The last note about the normal mapper tool is tangent space. The tangent space calculations that you need for bump mapping, the tool has to match. However the tool does tangent space, calculates the tangent space, has to match exactly with how your engine calculates tangent space. If they're not exactly the same, you're going to have, it's going to look wrong. It's going to look like there's errors everywhere. We burned a good, I think I burned a week of my life trying to figure this one out. I mean, there was literally, we did one thing where we had one 64-bit float typecast to a 32-bit float somewhere, and it gave us a little bit of inconsistency between the two implementations, the normal mapper tool and our actual engine, where we would get all these little artifacts. So with the tool ships the source code for it.

So you can either use our tangent space code or drop your own in, because they have to match exactly or it's not going to work. You're going to get errors. So we basically talked about from the normal map what these low and high res models are used for, how we do the ray casting, some optimizations and whatnot.

The next thing we're going to talk about, the next section, is some post-processing effects. We're going to talk about four different effects. One is sepia tone. Sepia tone is a way to do sort of antique pictures, basically. The next one is going to be Sobel edge detection. Edge detection filters are really useful for a lot of things, especially NPR rendering, non-photorealistic rendering. Then we're going to show posterized effect, which is an NPR that will also use the edge detection. And I'm just gonna touch on for a minute real-time FFTs, Fast Forward Transforms. So, sepia tone. Can we jump over to the demo machine?

OK, so here's some live video of me. Great. So this is basically what the camera is. We've got to apologize for the resolution. We had to use this camera because it's so cool. But it's only at 640 by 480. So there's sepia tone. So I can just sort of switch back and forth. So sepia tone is meant to look like a sort of antique picture. And this can be used in games. There's a lot of times where you want to do flashback scenes or whatnot to make it look like-- you don't just want to go grayscale. That's kind of boring. But there's a way to-- instead of going grayscale, it's a It's a sort of sepia way of doing things. So we jump back to the slide deck.

Okay, so here's the thing about sepia tone. If any literature you read about sepia tones, they always talk about this magical lookup table. For every RGB, for every, basically the idea is to take your image, make a grayscale, and then based on that, you map it into this different color space. Now everyone talks about this color space and this lookup table that exists somewhere.

It's really hard to find. We've searched high and low, and we can't find this lookup table. No one can. It's sort of always talked about, this lookup table. I'm sure it's in literature somewhere, but the point is having a lookup table is expensive. You have to store the lookup table, you have to fetch into it and all that stuff, and there's memory bandwidth issues there. So what you want to do is we found a way to do sepia tone without having the need for a lookup table. And what we do is we utilize color space changes. So we can change from the RGB color space. We do color space conversion from RGB into YIQ space and then back into RGB. Now, YIQ space is basically a different color space used in broadcast video and whatnot. not. And so the interesting thing about doing color space conversion is that it's just a matrix multiply. You have a matrix that transforms your RGB value into a YIQ value and you have a different matrix that takes you from YIQ back to RGB. So that's really interesting. So we're going to do -- so here's the algorithm that we do. So we -- the interesting thing about YIQ space is you can -- if you hard code certain values into the I and Q components, you can get some pretty interesting looking output.

What we do is instead of computing, we take our input color, our RGB color, and instead of just transforming the whole thing into YIQ space, we just calculate the Y component. And the Y component takes into account the RGB values. And then the I and Q components we hard code to be.2 and 0. And by doing that, it basically gives you sepia tone. And then you take that YIQ value and transform it back into RGB space and you get sepia tones. It's that simple. This is something fun to play with because if you hard code different values in there, you'll get different effects. And you'll get some interesting different looks that you might want to use in your game for different effects if you get the power up gun or whatever. You can do different things. So as we optimize it, as we look at the math to do all this, it turns out to just be a multiply and an add. It's really simple to actually do this. So here's the code, the shader code for it. the fragment shader code. Essentially, if we look down after the sample texture, this text op right dead center there, right below it is a dot product and an add. Those are the two operations that we need to generate sepia tone based on an RGB value. It's really simple. We do a dot product with a constant vector that's up top somewhere and the actual color value, and then we add a different constant to that. These slides will be online, so you you can rip through that later. The next effect is Sobel edge detection. Sobel edge detection. Can we jump over to the demo machine?

All right, so do you want to explain the difference between sort of the vertical and the horizontal? Yeah, so basically there's the Sobel edge detection, when we flip the whole thing on, the final result, is a way to look at your image and find all the edges. We have this camera, it's a little noisy right now, so you've got a lot of extra little dots up there. But essentially it passes a filter over, two different filters, one to find the horizontal edges and one to find the vertical edges. Why don't we jump in and just show the horizontal edges. Okay, so the horizontal edges, if you look at that thing right behind me there, you see just these sort of horizontal lines here at the top and bottom. If you go to the vertical edges, you get those vertical, the two left sides. And then when you have both of them up there, then you get the whole, all the edges basically in your scene. Can we go back to the slides, please?

OK, so edge detection is just-- this Sobel edge detection filter is really simple. It's just two kernels, your horizontal kernel down here and your vertical kernel. So for the pixel in the middle, if you're trying to calculate what the value is there, you take into account the three pixels above and below it, and you weight them with 1, 2, negative 1, negative 2 appropriately.

And you just combine those pixels, and you end up with your horizontal kernel-- sorry, your horizontal edge detection value. Do the same thing for your vertical kernel, and you get the vertical value. And here's sort of the output again. If you look at the picture down here on the lower left, you can see just the horizontal, sorry, the vertical edges versus the horizontal ones up top, and they get combined. So this is used a lot in NPR-looking effects.

And here's the Sobel code. I mean, it's pretty straightforward. You basically do eight texel fetches up front, down here at the bottom, and then based on those eight fetches, which are the eight surrounding pixels around your main pixel, you can then take the top and bottom three the left and right three appropriately on the right and combine them to do, to come up with that value.

Okay, so the next effect is an, it's an NPR posterized style. Now we're going to use the edge detection on this, and this is sort of, this is sort of inspired by this movie Walking Life. There's this edge preserving, it's called a Kuwahara filter, that basically does, gives you this nice look, right, that gives you this sort of NPR posterized which I'll talk about in a second. What we're gonna do is we're gonna composite our edges on top of that.

So, can we jump over to the demo? Here we go. All right. All right, so here's the non-posterized output. We're gonna turn it on. Gonna have to apologize. It's gonna be hard to see. The lighting conditions aren't ideal. There you go. So unposterized, posterized. Can do a couple different passes to sort of-- so if Alex can just sort of jump around, you can kind of see him. Jump around.

I'll pass on the jumping. There you go. That's posterization in real time. OK, so if we go back to the slides, So this is the post-release output when we have really good lighting conditions. This is part of our Mac driver team coming out of a meeting, and we snagged them and said, stand there, we need an image for our slide. So that's what the output looks like. Okay, so a core Hara filter is this. Essentially, for the pixel in the middle of this kernel here that we're trying to calculate, here's what we have to do. We take four quadrants of pixels. Each quadrant is a 3x3 pixel area. The upper left quadrant is sort of the pink area, and we have the green area, the, I guess it's orangish, and blue. And they all overlap by one row of pixels. So each 3x3 kernel, each 3x3 quadrant there, what we do is we calculate the mean and the variance for each one of those. The mean color and the total variance of color in that 3x3 quadrant. Once we have those four values, for each one of these quadrants, we have the mean and variance, we then look at them and say, And we figure out which one of those four quadrants has the smallest variance. Whichever one does, we just use the mean color from that section. And it's that simple. We just look at those four quadrants, figure out which one has the smallest variance, and pick that one's color. Done. It's that simple. So here's basically, that's the idea. The implementation is basically that. We do this two-pass effect, and essentially the way we speed this up, instead of doing these brute force calculations, is we basically do two passes over the final image. The first pass calculates the mean and variance for each unique three by three area in the whole image. That's one pass. We store that off and we render that into an off screen texture. Then we do another pass through it and we fetch from that texture to get the four quadrants out, those four mean and variance values. And in our shader we then do the comparison and just write out the right color. So we can get this down to two passes, really simple. It's all image space, it's a constant overhead. The interesting thing about post-processing effects is that it really is just constant overhead.

It doesn't matter what you rendered that scene, whether you rendered one polygon or two million polygons, it's the same overhead, 'cause you're just doing the same math, it's a fixed number of pixels. So, and it's great for games, you don't have to worry about the complexity of your game, as long as you have a few frames to spare, you can do that kind of stuff. So here's the fragment shader code. which basically shows how do we compute the mean and variance. You can rip through this later, the slides are online. But basically you just kind of go through, fetch your full 3x3 area of pixels and do your computations.

And then we have the variance selection, how we actually fetch from those, make those so dead center here. You have those four text operations. We fetch four pixels from that off-screen buffer, and then we do the comparisons down at the bottom half here and say which one has the lowest variance. And then we use the color for that to output. but pretty straightforward.

Okay, the last section of this, the last part of this middle section is FFTs, Fast Fourier Transforms. These are really useful. Anyone that's vaguely familiar with image processing knows that this is used a lot. And this is something that we can actually now do in hardware, the Radeon 9700 family products and other similar products that can actually do full floating point, that has full floating point precision and able to render out to a texture, 32-bit float textures, you can do some really interesting stuff now. So here's basically a test pattern on the left here, a very common test pattern for FFTs. And this is pasting. This is output from one of our in-house demos. We haven't released this one yet, but it's coming soon. And this basically shows the two different -- the frequency domain and the spatial domain of the FFT. That's all I really want to say about it. It's something that we're working on that we can actually start to apply. So this is sort of stuff that's in the works right now at ATI. We're starting to apply this, use FFTs to do some image processing effects.

Okay, so we talked about those four things, sepia tones, edge detection, posterization, and FFTs. Which brings us to the final section, which is the shadow techniques we used in the Pipe Dream Animusic demo. How many people have seen this demo already? A few of you. Okay, why don't we jump over to Demo Machine 2 with audio.

Do we have the audio hooked up for this? OK. Great. So this Pipe Dream demo, this is something we first saw at SIGGRAPH Electronic Theater 2001, I think. Yeah, 2001. And we were watching that, and we looked-- sitting there in the audience and said, we have to do this in real time as soon as we can. And as it turns out, the next trip we were doing, we kind of went back to the office and thought about it. And I realized, you know what? I think we can do this demo in real time. And so here's it running on OS X.

This was one of the launch demos for the-- this was the main launch demo for the 9700 product. Family-- I'll let this run for a minute, for those of you who haven't seen it. the beta build of our demo engine on OS X. So we're pretty much feature complete at this point.

So if you notice all the shadows that are happening, the dynamic shadows and the shadows globally in this scene. So we wanted to implement this and do this as one of our main launch demos. And one of the things we realized was the first thing we asked ourselves was how are we going to do the shadows in this scene?

Because the geometric complexity is really high. This scene, on average, we render about 400,000 polygons per frame in this demo. It peaks at around 550,000 polygons per frame. So there's quite a bit of geometry in here. And yeah, here we're at about 400,000. And so we're trying to figure out, how do we do the shadows in this? We wanted to, as writing graphics demos, we wanted to sort of keep up with what games will look like in a year down the road as developers start to take advantage of these latest features. So we looked at a few games that are in development and games out there and said, let's stick with full-scene stencil shadow volumes. How many people are familiar with stencil shadow volumes?

Okay, so basically it's a technique that lets you render hard shadows globally, right? In the past, developers have used shadow textures, light maps, right? That sort of bake in your lighting globally. But it's sort of soft. It's not very exact. It's not hard shadows. And there's a way to do dynamic shadows, dynamic objects that cast shadows that look nice and hard and crisp.

And that's sort of how games are going to be looking for the next couple of years. So we wanted to stick with that. So we first went through this. We got the scene up and running. And we turned on stencil shadow volumes globally. And our performance was somewhere around two frames a second. and we realized we can't do that. There's just way too much geometry. The overdraw on any given pixel on average was near 100 at points, 'cause of all these different volumes, all these different pieces of geometry. These things up here are just insane, the amount of overdraw, especially if you look at that from the side, you get a lot of overdraw there. So I'll talk a bit about that in a minute or so. Rav, can you sort of pause this or come out of the fly-through mode, yeah, and fly over here to the left. Okay, there we go. on this area here. So here's what we do. We have, yeah, zoom out and so there's two different kinds of geometry in our engine. There's something we call static geometry and then dynamic geometry.

So static geometry, so this is the dynamic geometry. This is the geometry in our world that actually moves, that's animated in some way, right? So these are the only pieces of geometry that cast shadows dynamically. This is our static geometry. This is all the geometry that never changes. It's baked in. It's not moving. There's nothing animated about it. So this, we can do something different with for shadows. We have two different shadow techniques that we combine to do our overall shadow. So if you zoom in here, Rav, on the left, if you look at that shadow area right up here. So we use this technique that I'm calling shadow cutting. So what we do is we actually cut the shadows directly into the scene. So if you turn on wireframe here, these are static shadows here. This is geometry. We actually cut the shadows directly into the scene right here. Hit page up a bunch, or just press A to pause the lighting. So you'll see we actually, the artists don't do this manually. They would, they'd quit if I made them do that. So we automatically generate this stuff. We figure out, we basically do shadow cutting, which I'll explain now. Can we jump back to the slides, please?

Okay, so static shadows, like I said, it's a great opportunity to not do stencil shadow volumes globally because their shadow volumes don't change. They don't ever change. And a lot of those shadows won't cast geometry onto dynamic objects. So you don't need their shadow volumes, right? So for a lot of those things, you can optimize this out. So here's where we just showed the before and after, right? This is what you see, which looks like shadow volumes, which I'll get to in a minute. But instead we're actually cutting the shadows straight into our scene geometry. And so the advantage is, like I said, is it looks just like stencil shadow volumes. This is what some very well-known, soon-to-be-released games are going to be doing, global stencil shadow volumes. So it's going to look hard shadows everywhere, which is what we wanted. But we obviously have to cheat a lot. We're not going to do full shadow volumes. We need to run more than two frames a second. So the interesting thing is that since we separate out our geometry, so you're probably thinking, isn't it more expensive? You have more geometry now that you diced your geometry up, right? Isn't that slower to render? The answer turns out to be no, because if you compare it to shadow volumes, you're rendering these huge volumes to your back buffer that have a lot of overdraw. Instead, we're just going to have a few more vertices to process, which is the same, but a lot less than the shadow volume verts. those polygons that we know are in shadow, we can actually draw with a simpler shader because we know that none of our dynamic, our dominant lights really hit those polygons. So we don't have to do all the math that's required for those polygons that are in shadow permanently. Um, So here's how we do it. We do it with shadow cutting. So it requires beams. So here's just some basics on what a beam is for those of you who aren't familiar.

A beam is basically this. So a beam is sort of a pyramid-shaped volume, in a sense, with three sides. And so you have a light position, this yellow thing up here. That's a light source. And this polygon, this sort of horizontal edge at the bottom, if you think that the polygon is actually popping out of the screen, right, out of the screen here, the volume is basically the position of the light and you calculate four total planes. One plane is the point of the light with each edge. So each of the three, three of the four planes are calculated by the light position and the edges of that polygon. And the fourth clip plane is the actual plane of the polygon itself. And that gives you, so depending on which way you look at that polygon's plane, you can have a near beam, which is basically anything in that triangular volume closer in between the light and the polygon is a near beam. A far beam is anything that falls beyond that polygon. So the shadow cutting algorithm is this. We do this brute force. I tried, I can't even tell you how many different methods and algorithms to try and do shadow cutting.

And all of them had, I mean, anything in literature has a lot of different issues. They're all trying to optimize certain things. When it comes down to it, doing the brute force approach is the right thing to do. And you can optimize it so it runs pretty fast. For the animusic scene, globally, the algorithm takes about 20 minutes for the entire scene to cut those shadows in. I mean, that's a pretty complex scene. So that timing's not too bad. So basically, here's what we do. For every polygon in our scene, one at a time, we're going to dice it up. We're going to take all the other polygons in the scene and cut the shadows into it based on that geometry, one at a time. So what we do is we'll call that polygon A. Polygon A is for each polygon in our scene. We're going to find all the polygons that live between that polygon and the light source. So everything in that polygon's near beam we're going to say has to cut into that polygon and dice it up for shadows. So for each polygon A, we have a bin of polygon Bs. For each one of these polygon Bs, we take its far beam. So for polygon B is between. So in the case of if this is our light source here, and we're trying to cast shadows onto this table surface, this microphone here has to cast shadows onto it. So for a given polygon B here, we're going to beam into polygon A, which is the table. And we're going to take its far beam and cut into this table with those three planes, which is what's happening here. Polygon B, this blue one being the microphone polygon, down into the table. And those end up cutting it into three different polygons. And what happens is on the right here, you can then tag each resulting polygon as in or out of light. So if it falls in polygon beams far frustum, far beam frustum, we tag it as in shadow. Otherwise, we just leave it alone. And it's that simple. You just keep doing that over and over and over.

and you just have an optimization step that optimizes your mesh every step of the way. And you end up with, you know, what we just saw, the button here, the final shadow. So here's the interesting thing is a lot of the literature, a lot of different ways to solve shadow lines, to do shadow cutting in a sense, can't solve these cases. Cyclically overlapping polygons are the root of all evil for anyone doing research in this topic. It's polygons that overlap each other, three of them, right? There's no order. You can't possibly sort these three polygons and figure out which one's on top because they're each on top of each other, right? And so this breaks a lot of ways, different methods. This brute force method of just cutting everything up works fine. You end up with what's on the right there, exactly what you want to see, everything cut up perfectly.

And there's the results. So those are our static shadows. We cut them into the scene, nice and simple, right? Now we have to deal with our dynamic shadows. And in a second, we're going to talk about how we combine both of them so they look right. They look like they belong together. So let's talk about the dynamic shadows. So dynamic shadows are basically done with shadow volumes. Let's take one -- can we jump into the demo, please?

Okay, so, okay, why don't you pull the camera back a little bit? Okay, that's good. And now, turn on just objects. There we go. So here's our dynamic objects, and if we turn on wireframe, these blue wireframes are the shadow volumes. So a shadow volume is basically, given our piece of geometry, we can generate a volume that represents what's in shadow. So anything that falls into the symbol, for example, here, anything that falls into that symbol's volume, this volume generated, is in shadow. We tag that as in shadow.

We can do tricks with how we render these volumes into the stencil buffer to make it look like, to actually tag the pixels properly if you're saying what's in shadow and what's not in shadow. So those are basically what a shadow volume is. If you turn, actually if you turn wireframe off and go back to the full scene and object geometry.

D right there. So if we zoom in up here over the drum machine, there we go, and just sort of page down briefly just to get those other instruments up there. That's fine there. I mean, even with the balls flying, you can see those volumes intersecting with the wall and generating those shadows. So here you have a lot of shadows here. We'll come back to this in a minute. Can we go back to the slides, please?

Okay, so here's just a wireframe shot of these instruments going over the drum machine there, and here's their shadow volumes. And you can see where their shadow volumes intersect the wall, those pixels are tagged as in shadow with sense of volume. So here is essentially, here's what a shadow volume is. Shadow volume basics, for those of you who aren't that familiar with it, is given a light position and, say, a sphere, what you do is you figure out the silhouette edges from the light's point of view. What you want to do is you want to basically break that sphere in half, right? Keep anything that gets hit by the light in place and take the bottom half of that sphere and shoot it down far away from the light and at the angle of the light as well. So what you end up with is this sphere turns into this sort of this weird looking pill shape, this full pill shape. And that is your shadow line, which is a closed volume and it's sort of -- and that -- you can use that to render into the stencil buffer to render to mark each pixel as in or out of shadow.

Here's just sort of a brief look at one way to do this in hardware. Essentially, you have to... So there's this method... How many people here are familiar with this method of doing shadow vimes with degenerate quads along the edge of primitives? Not many. Okay, so the idea is basically this.

For each... So you have this model, this sphere, right? For each edge in your model, what you want to do is you want to stick a degenerate quad, an infinitely thin quad there, just two triangles that connect that edge. So you want to take your sphere, basically separate out all these vertices, and add up two polygons along each edge, that is basically a quad. So the idea is when you take this sphere and you split it at the seams, and then stretch it apart, those quads along those edge, those silhouette edges, get stretched. And so if we go back one slide to this, over here, these represent sort of the quads, one of those vertical areas represent a quad that was originally stuck on an edge, right? And when we split it apart, it's like putting like Elmer's glue there and you just pull it apart and it sort of just fills in the gap or you just stretch it open. So the idea is this. So we stick these infinitely thin quads along these edges. So here's two polygons in our given model. And that shared edge there, we stick these infinitely thin polygons right in the middle there.

Now, they're actually right on top of each other. I'm just spreading them apart so you can see, so you can visualize it. And what we do is we stick the face normals of each polygon. So polygon A, we take its face normal, sorry, oh, we take its face normal and we embed it into the three vertices of polygon A. And we take polygon B's face normal and embed that face normal into those three, three verts. And the idea is this. The idea is that in your vertex shader, as you're running a vertex shader, you want to basically be able to shoot each vertex back one at a time. You want to either keep it in place if it's front facing to the light, or you want to take the vertex and shoot it back to generate this volume. So the idea is that since you have your face normals embedded in your vertices, if you shoot back the lower left vertex for polygon A, you're going to end up shooting back the other two as well, because it's the same normal, it's the same face normal you're doing the math with. You're going to basically do a dot product with the light vector and that face normal. So if one shoots back, they all shoot back. And in the end, you end up basically either each primitive stays in place or it gets moved back one at a time. This This is sort of a brute force approach to doing shadow volumes in hardware, fully in the hardware. There's no CPU work happening here. All happens in your vertex shader. So we render this into the stencil buffer.

And so they're in there. So now we can render these shadow lines in there, and you have this in your stencil buffer. You can basically test it to say which pixel's in shadow, which pixel is out of shadow. So which brings us to the question is how do you get those shadows into the color buffer and make them look good and make them blend in with the shadows that we have baked in, right?

You have these baked in shadows. You want to make sure that you don't double darken pixels. If something's already in shadow from the scene geometry, you don't want to have it go into shadow again. It's going to look bad. It's going to look obvious that you're doing something different for dynamic geometry. So, we're at the stage right now where we have, we essentially have our full scene drawn into the back buffer with no dynamic shadows. Everything is drawn without the shadows. And our stencil buffer lives each pixel tagged as in or out of shadow.

So now we have to combine these, the back buffer, with our stencil buffer in some way to get them to look right. So what we do is, one thing you could do is just draw a full screen quad over your scene And for each pixel that's in shadow, just dim the color by 0.5. Just dim it in half. You're going to get some really bad artifacts like that. It's not going to look right. So what we're going to do is we use Destination Alpha. How many games here? Who uses Destination Alpha for anything? Actually stores real values in Destination Alpha? Two. Two.

Perfect. So-- four, sorry. So Destination Alpha, you can think of this as a whole other channel you can store data in. As you render your scene, as you render each polygon, you're writing to RGB, also write something to alpha. You can write something useful to alpha and then use it later in a separate pass. So this is a way you can do a lot of interesting things. So we're using Destination Alpha to do our shadows. And the idea is this. Instead of drawing a quad over your screen, to just dim each pixel by some constant value, we want to basically, as we draw our scene, write out a value to destination alpha that is a dim factor.

That means it basically represents this. If this pixel falls into shadow by something dynamic, how much should I dim this pixel? So for things that don't have any real lights hitting it, you don't want to dim it at all. For lights that have a really dominant light hitting it, you want to have a fairly high value in there to say, I want to dim this pixel a lot. So the way we do it is we want to write a value out there. that we can just simply multiply this dim factor into our color and get the right value out. And the idea is to bring you as close back to ambient lighting as you can. So you basically have this scalar value. So here's what Destination Alpha looks like. Here's a screenshot. Can we jump back into the demo, please? OK.

If you zoom back and just sort of, yeah, what is it? G, is that the color? Yeah, I go two more times. There we go. So here is what the scene looks like. Now, you're going to notice it looks counterintuitive. Where there's no lights up at the ceiling, there's no real dominant lights hitting that, you have white. What white means is it's just one, right? You're going to basically take, if you multiply that pixel by one, it's not going to change, which is what we want, right? If that falls into shadow up there, it doesn't matter because there's no light already hitting it, so there's nothing to mask out. So you want to just multiply by one and leave it alone. For the pixels in here, in these dark areas, these dark areas here, those have a lot of light hitting them. So we want to subtract that. We want to dim those pixels a lot. So when we scale those pixels there, we're going to take this mid-gray color, say 0.5 or 0.2, multiply it by the color value. It's going to dim those pixels. So where any one of those balls hits the wall, has their shadow hitting the wall, you get that nice dark circle, it's because we multiplied this value by the current color value. And you got those shadows. One thing I want to point out is if you go back to the color mode, if you can zoom in dead center there to the left. So here's one thing about our engine. Let me see. It's one thing. So we want to go here. Zoom in just right on there. So here's how our engine basically works. We have these dominant lights. Let's back up, back up, back up.

Okay, so here's it starting. We have one light in this area. If you notice right here, dead center, we have just one dominant light on. And we have geometry here for this is where the shadow cutting comes in. Now, in a second, we're going to have this other light turn on. Where it's coming? Let's go. Turn on.

And what you're going to see is we have, we're going to, and these are baked in shadows here on the wall. We now have two different lights casting shadows in this area. Okay, if you turn on the wireframe quickly. So this is going to show you, we actually cut in, in our shadow cutting, to back up for a minute, we have, we cut into our geometry shadows from multiple lights. And we can preserve which polygons fall into one light source, two light sources, or are shadowed by one light or two lights. So if we turn the wire frame off for a minute, go into the desk alpha, view desk alpha there.

There we go. So if you notice, this is what we write into dest alpha for this area. So the interesting thing is for the pixels right here in this sort of darker gray, there's no baked in shadows. This is just a pixel being hit by the light full on. The pixels down here that are sort of lighter gray are being hit by just one light.

right, because the other light is casting a shadow there. The brighter white stuff is actually being cast. Those are the pixels that have shadows from both light sources. So there's no light to really subtract out of it. That's why it's white, right? If something, if a shadow volume intersects those pixels, go back to the full color view, it's already fully in shadow. There's nothing to shadow there. So we can write out-- when we preprocess and do our shadow cutting, we tag each polygon and we know how many lights is already being shadowed by, or how many lights is being hit by. And we write out the right value. So what we end up with-- so can we go to the right over the drum machine on the left? Yeah, right there. So let's page down a little bit and get these guys up here.

OK, good. So I want you to notice, this is the whole reason for doing this dim factor thing with this sort of dest alpha As these shadows go out of this light source, they dim and they blend in perfectly. This bar here, this big horizontal bar, that shadow is baked in. That's from our shadow cutting, right?

And the idea is that those shadows blend in with that bar fine and they disappear nice and smoothly. You never know that right when it gets out of that volume, that disappears. We don't actually draw that shadow line because we do culling based on our light sources. So we end up with these nice shadow volumes, these nice shadows that work perfectly, that just integrate with the scene well. So can we go back to the slides?

OK. So the shadow quad. So now that we basically have our dest alpha drawn, everything, all the contents of our destination alpha and our RGB channels are filled with all the right data. We have our stencil buffer filled with the right data. Now we have to basically draw a quad over our full screen to get the shadows for the right pixels. And so what we do is this. This is our blending. In OpenGL, you set your source color to zero and your dest color to dest alpha. Sorry, you sort your source factor to zero and your destination factor to dest alpha. So the equation you end up with when you plug this into your alpha blending algorithm is you end up with, this turns out to be zero, and you end up with this dest color times dest alpha. Dest alpha is your dim factor. So you just end up with the equation that you want. And you turn on your stencil test as well, and you only write to the pixels that are in shadow. And it's that simple. And you get those shadows in the end. So to summarize on the shadows, we basically have talked about the differences between the static and dynamic shadows, went into some shadow cutting algorithm, and some of the shadow volume basics. So and that's how we do shadows in Atom Music. So I'm looking at the clock and I have another, I have a few minutes to burn here. So I want to show one other effect, which is if we could go back to the demo machine in Atom Music.

So, if we press 1, press 1, start this over. If you watch this, the balls in that music are actually motion blurred. When we first got the DVD in our hands from SIGGRAPH to watch this video, I stepped through every frame and I watched everything they were doing in the full video. And the only thing they motion blurred were the balls, because those were the only things moving fast. Everything else was moving slow. So we said, well, how do we do motion blur on balls that make it look realistic? And so we came up with this vertex shader to do what looks like motion blur. So if we press A to pause the animation. Okay, so here's the pause. Here's the actual geometry. That's good. Actually back up just a little bit. Okay, so the ball right in the middle there down on the bottom is almost a full ball shape. And that's moving pretty slow. The ones that are moving faster look like hot dogs, right?

So what we do is we hit wireframe. One more time. Okay, so it's going to be hard to see. Turn off, press D, turn off the--one more time. Okay, so the ball shape here looks like a pill. What we basically do is we take our ball and we split it at one of those seams, and we generate--it looks like it's motion, but we have the right shape for it, right? And what we do is we then use--when we fetch from our environment map, because these things are all environment mapped, we lower the MIP level bias, the LOD bias. So when we fetch the textiles from our environment map, it's blurrier. We also change the opacity value on those pixels. So the more rounded it is and not moving, it's a solid ball. As it becomes more and more stretched, we make it more transparent. So it looks like you can still see through it, right? 'Cause if you keep it solid, it's gonna look like you're launching hot dogs, right?

which isn't good. So you wanna make it look like, you're gonna be able to see through it. 'Cause motion blur, you can sort of see through anything that's motion blurred. So now you're probably wondering, well this is a cool technique, but not a lot of games have balls launching out of machines and instruments. But you do have other projectiles. You have rockets, you have someone's gonna throw a grenade over something, or whatever things that are gonna be moving in your scene in your games, you can apply this technique to it, right? Especially things, even things that are like a, You can do the same thing with shadow volumes, where you have these degenerate quads stuck in there, and you split it whatever the appropriate seam is. Based on how it's rotating, you can split it based on its direction vector, its velocity. And based on the velocity, you split it more or less. And so this technique can be used in a lot of different ways. So that's motion blur and then music. All right, can we go back to the slides please?

OK, for more information. So here's some references. These will be on the web. These are some books I highly recommend as far as current graphics go. And some links to-- the normal mapper tool will be on ATI's developer website sometime in the next few weeks. And here's us. That's me, Rav. Marwan is another guy in our office that worked on some of the video shaders. And if you have any questions, feel free to drop us an email. And that's it.