Good morning everyone. My name is Travis Brown. I'm the graphics and imaging evangelist. And I want to welcome you to session 208, Fragment Programming with OpenGL. And this session is our second session in programmability in terms of sort of our focus on exposing the capabilities of GPU to do interesting things like vertex operations, and in this case doing per-pixel operations, very advanced per-pixel operations at incredible speeds.

A lot of you saw some demonstrations using fragment programs earlier this week in the graphics and imaging overview. And what we're going to do in this session is really sort of drill down and focus on fragment programming. And we're really looking forward to seeing what you the developer is going to be able to do with this incredibly new technology. So it's my pleasure to invite James McComb to the stage to take you through the presentation.

Basically, this little program that you upload to the graphics card, it's ran once for every color lookup during the rasterization. : Basically, whenever the rasterizer is going across and filling in the pixels on the surface of the polygons, normally it goes through a fixed function which can get pixatexels from the different texture units and blend them together.

Well, this allows that process to be totally programmable. So, yes, you could go and get the data from the texture units or you could just return any color you want based on a mathematical function. James McCombe : Now, so, again, the output of the fragment program is good. It's got a very well-defined output, single RGBA color, and optionally you can specify its position in the depth buffer so as it can be culled by later stages in the GL pipeline.

In regards to inputs to the fragment program, The texture coordinate channels which you would normally set up in the vertex program, you'll note that when you write an output vertex, you can put into the texture coordinate channels and you have the number of channels is equal to the number of texture units on your graphics card.

The values you write in there in the vertex program, those are interpolated across the surface that the fragment program is running on, so you get that interpolation for free. So that's one of the inputs. You also get the interpolated vertex color and the position of that fragment in the window, in window coordinates.

Also, with our fragment program, you can access all of OpenGL state, which is certainly very convenient. That means you get access to the matrices and also the program parameters as well, which I'll discuss. And then the other important input is you can sample text. So you can sample text from any of the texture units, which is important also. Let's take a look at specifics of our fragment program.

Great news here. If you've programmed our vertex program, the language is totally parallel to that. It was intentionally designed that way, so if you know our vertex program, you know our fragment program as well. The only difference is that instead of dealing with XYZW components to specify a vertex position in 3D space, you're now defining red, green, blue, and alpha components for an output color. So the same thing applies. All these instructions, there are a few exceptions. 90% of these instructions are vector instructions, meaning if you do an add, it does the add for all four components.

There are some of these instructions, which we'll talk about later, which are considered scalar instructions, and that is, for instance, the POW instruction, to the power of. That's a scalar instruction. So if you want to do a power for all four components, you need four instructions. That's one thing.

So I've grouped these together. Those are all your arithmetic instructions. You've got some logic instructions like other languages. Unlike programming a general-purpose CPU, don't be expecting to be able to do bit shifts and things like that. That isn't going to happen. So you won't be able to do shift left, shift right, any of that. But you do have some basic logic to determine if something's equal to zero or not, or it's greater or less than zero. So you can do that. Less than zero.

The other thing, then, is these are the important ones that our fragment program adds. They're the texture sampling instructions. These allow you to get things from the texture units, with text being the most common one. And then you have some miscellaneous instructions like vertex program. You have a squizzle instruction, so you can reorder your components in a vector. And you have this kill instruction.

I haven't actually used it yet in the fragment program. But what it actually is supposed to do is stop a fragment from continuing down the OpenGL pipeline. It actually literally just kills it off. So there might be some possibilities for you with that instruction.

Let's switch over to the demo machine and let me show you Shader Builder here.

demo two, actually. Great. So what we have here is like the screenshot I showed you, on the top right of the window, we have the rendering view, which, you know, I can move it around. At the moment, it's just showing a simple quad, which is really what you want if you're just writing a fragment program. You want to see the pixels.

You don't really care about the geometry so much for a lot of the things you'll be doing. On the bottom right, again, you've got the texture units inspector. So here, if I click here, this shows me that in texture unit zero, I have this rock texture loaded. In texture unit one, I have the water texture, which is what's currently being shown. Now, let's look at the code here.

We can see basically what's going on here. This is a very simple pass-through fragment program. It's doing nothing special. Remember, the program's running once. I beg your pardon? Oh, it looks okay here. It must just be clipped. Sorry about that. Is that better? Okay. Sorry about that. So, what it's basically doing, this is running once for every pixel, let's just think of it as. And what it's doing is the text instruction, basically that acquires a color, it gets an RGBA color from a texture unit at a specified texture coordinate.

So, we can see that I have a temporary variable declared, which I've just called T0. And I am sampling from texture unit 1, that's the second argument, from texture unit 1 into T0. And this fragment.textCord that you can see, that is the interpolated texture coordinate that it's using to look up at. And then I'm simply, in the last instruction here, I'm moving T0 into result.color.

Result.color is just, it's sort of a fixed thing in the language which defines the output color you're writing. Let's look at, that's something I can do. So, for instance, this texture 1, that's the texture unit that I'm sampling from. So, as I change that, you can see it's changing. That's very straightforward. Let's look at something a little more interesting. Let's implement, say we wanted a fragment program that would allow a program parameter to adjust the brightness of the image and do it all in hardware. that. Okay. Let's create a new line of code right here.

So what we've done is we've declared a parameter p which is bound to a program parameter, meaning that there's now a GL entry point that I can call with index 0 and I can pass in a floating point value which is sent to the card and run in this program. So let's implement a brightness control. Pretty straightforward.

A simple multiply will do that. So let's do a multiply. Remember it's like assembly here. If you've written vertex programs you'll be familiar. It's the destination first and then the arguments. Multiply takes a destination and two arguments. So we multiply t0 with the x component. That's the first one last little change to make.

Hi, I'm James McCombe, I'm the editor at OpenGL. I'm going to show you how to create a 3D visual effect with the X component of P. Right now, the brightness is at the bottom, so we're seeing it as black. But watch this. If we move over here, you notice P is in this identifier list. I'll select that. I will open up an inspector right here, it's the symbol editor right here.

This allows me to change those values. So remember, the brightness is stored in the X component. So I'm editing P, and you can see right here where my mouse is, I can change the minimum and maximum value this slider will go to, and I want to set the range to go from zero through to two, just to keep reasonable values. And as I slide this right now, you can see it's sending that value to the graphics card, and the graphics card, there's a brightness control implemented in hardware.

That's pretty straightforward stuff. Let's now show, okay, I want to implement a contrast control as well. I'm going to do a pretty cheesy implementation of contrast here. I'm going to use the, I'm just going to basically raise the color to a power where that power is the contrast. And as I, if you may recall, I mentioned that the power instruction is a scalar instruction.

Note, I only had one multiply instruction, yet it multiplied all the three red, green, and blue components for me. Power will, the power instruction will not work like that. So I'm going to need three of them, and I'm going to need to do it for each channel. So I'm going to do that. So I'm dealing with the red channel here. Red channel dealt with.

And you notice that Shader Builder is keeping, like before, as I type it's all real time. It's updating on the fly and it's showing me all the syntax information, it's highlighting the line that the error is on, it's just great to work with. Okay, we now have a contrast control. If I now take the Y component, I set the range like I did the other one.

0 through 5 happens to work well here. It's totally low contrast right now, hence white, but notice as I increase the contrast here, it's becoming more and more and more contrasty. And again, the brightness control works as before. So this is a simple example of writing a fragment program and showing how you get program parameters into it. So let's head on with the presentation here. Back to the slides. I'm going to try and keep the rest of this presentation pretty example centric and get you thinking here.

I'm going to open up the other example which I have laid out here.

: Switching to the Fragment Program, it's a pretty straightforward Fragment Program. It does much like what I described if you follow the comments. Let's take a look at what's in the Texture Units. In Texture Unit 0, we have this rock texture, which is the texture I'm going to displace, I'm going to warp it.

And then in Texture Unit 1, I wrote a little program that basically just generated a spiral sort of effect and it encoded those vectors as colors and put them into positive space, which is what you're seeing in the Texture Unit right there, in Texture Unit 1. So my first instruction here is the text instruction, which samples the displacement map.

I'll just highlight that right here. And then it does what I described, which is scaling it from color space into a vector, which has negative and positive. And then I have a program parameter, a bit like what I showed in my last example, where that allows me to control the magnitude of the effect. So by multiplying that vector with a normalized magnitude, I can control how much of a spiral effect is going on.

I add it to the original. So I'm displacing the texture coordinates here. That's how I'm doing the lookup. I'm actually changing the texture coordinates for each fragment. That's how you do the warping effect. And then finally, when I have a texture coordinate. I sample from the rock texture map in Texture Unit 0. So let me show you what it looks like. So here, I select the magnitude parameter in the list.

I open the Symbol Editor again. And as I slide this, you'll see there's actually a spiral effect going on. It might not look as good as the Photoshop filter. But the reason for that is simply because the quality of the displacement map that I put in. It was crudely generated.

It was a simple program. You can imagine here how this same code is totally generic. You could create a displacement map that was a sine wave filter or any number of other displacements. And this same fragment program, all I'm doing is sending four values, four floating point values across the bus. And this is all running on the graphics card. I think that's pretty exciting.

So, let's see now, okay, yeah, let's move on. Back to the slides again.

: So basically this configuration started off as three of those little arpentamino configurations I described. You can see it's sort of entered a stable state right now. There's a little bit of movement, but it's totally stable right now. So that was life running on the graphics card. It's just kind of interesting.

Can we cover it at the end? Thanks. Okay. Now, let's head back to the slides here. And let's look at how does optimization apply to this new environment. That fragment program I showed you was hideously unoptimal. It was long. It had lots and lots of texture sample instructions. These are the very things that you really don't want to be doing in a fragment program. Because, yes, it's very quick. It's very, very quick.

But it could be quicker. And remember, I was only -- I had a small window. And I was only texture mapping the face of one polygon. If I was in a complicated 3D game environment, which I'm sure some of you work on on a daily basis, performance really, really matters. And we're going to look at how you can make big improvements in your fragment program performance. ShaderBuilder is going to help you here a lot, I think.

So, let's look at some of the tricks that we can do. The first thing is, if possible, why -- instead of calculating things in the fragment program, if it's something that isn't absolutely necessary for -- to be calculated on a per pixel basis, remember, you've still got your vertex program running. That's running for every vertex.

Now, for that example, there were only four vertices defining a quad. But many, many, many fragments. : Why not calculate it on a per-vertex basis if it doesn't need to be as high resolution? You can then pick up that data in an interpolated form in the fragment program as those values will be interpolated across the surface.

The other useful thing is using lookup tables at this time can bring performance benefits and I'm going to show you how to do that. Instead of doing a calculation in the fragment program, why not pre-generate at your application launch time a table of numbers stored into a texture map and then be able to sample that in your fragment program and there you have a lookup table, just like you're familiar with.

The instruction set is very rich, so I would encourage you, and I know no one likes to read the manual, but it's a good idea in this case to read through the instructions. Shooter Builder has a nice instruction browser. Go through them, learn your toolkit, learn the instructions you have because you could be implementing something in four or five instructions and you realize, oh, there's just a strange instruction I've never heard of that actually does all of this in one instruction.

That's going to run faster and as good as it can be. It's going to make your code shorter, too. So let's look at optimizing life specifically. How did I go about that? This part I'm a little concerned about because it's a little hard to explain, but I'm going to do my best how I did it.

This is a little clipping out of the grid that Life was running on. It's that R-Pentomino thing I was talking about. One of the things that added a lot of instructions to that program that I showed you when I was running through it was the process. Remember there were eight texture sample instructions and that was sampling the state of all the surrounding cells.

Well, that's a lot of texture sample instructions. If we could do that in less, it would be great. Not only do we have to sample those texture, those eight times, we also needed to calculate the surrounding coordinates to do that. We needed to calculate the surrounding texture coordinates and there were eight of them.

Well, I'm going to show you a way that I went about it that actually has allowed me to drastically cut down the instructions here. If we look carefully at the Life configuration, we realize that it exists only in one color in. It doesn't require the three color channels. So let's take advantage of that.

We realize that in Fragment program, the instructions, 90% of them will deal, will operate in one instruction like AlteVec almost. It will operate on all four components at once. So if we can pack more data into those channels and execute one instruction, we can do the work of like three or four.

So what I did was outside of the Fragment program, just before I did the feedback where I was feeding it, where I was doing the feedback, I actually took the sort of one bit image and I took it and I offset it at one pixel to the left and I stamped it into the red channel.

Then I offset it at one pixel to the right and I offset it and I stamped it into the blue channel and then in the center it was in the green channel. So I end up with, I've magnified it, but you can see what I've got here on the right hand side.

And that shows you how I've packed the data in. This is really good because now if I do one texture sample in the Fragment program, by accessing the red, green and blue channels independently, I have got the state of the left and right pixels. Meaning to get all the surrounding neighbors, all I need to do is read the, is three texture samples. Instead of just one.

I only need three. And then by using the red, green and blue channels, I've got all the pixels. That's a useful thing. The other important thing is remember that nasty logic that I implemented which was to choose whether the cell would be on or off? Well, that was like six or seven instructions. Let's get rid of those with like one instruction. Let's create a lookup table. James McCombe Here is a texture map. It's a two by eight texture map.

And basically what we do is on the Y component, on the Y, in the texture coordinate Y component, we just put in a one or a zero. Is the current cell on or off? And on the X component, we have the number of neighbors that we calculated. And by simply doing a texture sample with that lookup value, it's a lookup table.

We get back. So, we've got the color of the output cell. That's another pretty useful thing to be able to do. After doing that, here was the benefit. Remember, Shader Builder had that nice thing that would show you the number of instructions and the number of texture samples that were going on on the graphics card? Well, I just created a chart to show you what I was able to get it down to. And it runs exactly the same. It looks no different.

Except it's faster. If you measure it. So, I'm going to measure it. What we see here is the instructions went down from 36 instructions down to 9 instructions. That's a pretty big deal. Texture samples, those are expensive to run on the graphics hardware. Got those down from 9 down to 4. Temporary variables, well, not 13 of them anymore. Got those down to 5. And then this last one is interesting.

A DTR that expands that for you. It's a dependent texture read. What a dependent texture read is, so I've shown you the text instruction which allows you to take a texture coordinate and look up and get a pixel back from a texture unit. Well, imagine that the coordinate that I used to look that up was actually derived from a previous texture sample. You can imagine that you have dependencies between them. Well, for every one of those dependencies, that's known as a dependent texture read. The graphics hardware has support for four of these.

I am now using one of them. It's still a huge performance improvement. Dependent texture reads, if you're going to use lookup tables in your program, you're going to inevitably use these. You've got four of them. I find that is more than enough for anything that I'm going to implement. You can look into that after.

Now, this is my final part that I'm going to look at. This part has had me most concerned because it seems that a lot of -- when I look at other demo programs of people who have maybe implemented per-pixel lighting models using not necessarily our fragment program but other extensions, I find it personally very hard to understand what they're doing.

I've looked at their examples. Yeah, they look cool, but it's like I'm not really sure what they're doing. So what I've done here is I just threw a lot away and started from scratch, implemented my interpretation of a lighting model, which I think is correct, and it looks good. And I'm going to just explain step by step.

I'm going to break it apart for you, show it how it's done, and try and dispel any of the mystery there. So there's certainly some 3D math involved here, but I'm sure it should be quite understandable. Before explaining it, let's look at what the contributing entities to this are.

What are the sort of inputs to this fragment program? The first one is the position of the fragment of the light source. Where the light source is relative to the fragment that we're currently calculating the color of. Well, so we're going to call that LP. The other thing is I have support for variable brightness lights, so that's another thing.

For the light source, we have another, just a scalar, which is how bright it is. The next important thing is where is the fragment in 3D space? Where is it located? That's important to know as well, because we're going to use it to calculate a vector from the light to the fragment.

It's important to know that. The other important thing is the normal of the fragment. This is the key part that makes per-pixel lighting models look so cool. And that is that the normals, unlike the OpenGL traditional lighting model, where normals are only exist on a per-vertex basis, and they're interpolated across the surface of the polygon, here the normals are on a per-pixel basis, meaning we can create these amazing detailed surfaces.

And the way it's done is if you recall back to when I showed you my displacement map where I had encoded a two-dimensional vector into a color, this is exactly the same thing, except we're encoding a three-dimensional vector into each color. So not only an XY, but an XYZ.

So we're able to encode a three-dimensional normal into a texture map. And by sampling that in the fragment program, I can get the normal at that point, which is the large reason why this looks so good. And then finally, we have the fragment -- we're calculating the colors of the fragments on the surface of a polygon, so we need to know the normal of that entire polygon so that we can transform the fragment normal. We'll cover that in a minute. Another useful thing I did to make this program simpler was I wanted to calculate everything in model space.

So the -- in your 3D game, your light sources, you're probably going to want to define them in world space or eye space. You're going to want to define them relative to your world. And then what I did was in the vertex program, I multiplied by that -- by the inverse model view matrix to transform it into model space so the fragment program had something easier to work with. So I've already actually covered the normal map, kind of what it is, but this is what it looks like. That's the base texture map that I'm going to use. And you can see the one below is the normal map.

Well, these psychedelic colors are the -- are what define the bumpiness of the surface. Again, the same little trick applies. I have to do this sort of -- you know, normals are obviously they range from -- they're normalized. They range from minus one through one. I have to squash them and push them up into positive space using this super simple equation.

And I'll unpack them in the front. So I've got a little bit of time. We're going to get into the next step here. I'm going to go back to the program. I'm going to go back to the fragment program. So quick run through before I demonstrate it, what's going on here.

We calculate the vector from the light to : Light vector incoming, LVI. Then, when we've done that, it's pretty trivial to calculate the distance, just a scalar, the distance from the light to the fragment. If we know the vector, we can get the magnitude, and that's the distance. And that we use to attenuate the light later on, to get attenuation in here also. Then we acquire the fragment normal, like I described, we sample the normal map, and we get the normal.

fn, then we transform that relative to the surface of the polygon. I explained that in a moment ago too. Then the good part, if we know the fragment normal, and we know the incoming light vector, we want to calculate it after it's reflected. So it comes in and we get it coming out like that. We do that for every fragment. That's important also.

And then when we have that, we do a dot product between that and with the incident light ray, and then the direction of the light source. So in this lighting model, you could actually not just necessarily have the light pointing directly down on the surface, but you could have it sort of going along the surface. You can change it. I've actually made it fixed right now, but there's no reason I couldn't move it around. I just wanted to have not spent so much time here on this.

Then the next thing is to do the attenuation. Remember, LD, we calculated the distance from the light to the fragment. We're going to use that to attenuate the light intensity, which will create sort of a fall off, which is important also. Like for instance, if I hold a light here, a little light here shining on this, it's not going to have very much effect lighting up the far wall. That's the attenuation.

And then we modulate that with the light brightness control. So quickly, a little bit of animation here. This is the surface over polygon, FP. That's the current fragment we're calculating. Here we have the FN, the fragment normal, which I acquired from the texture lookup. It's encoded in the red, green, blue channels.

Here's our incoming light vector, like so. Then we calculate after reflection with the normal. And then LD, that was the distance I was talking about. So I think it's probably very clear in your heads now what I'm going to do. And I'm going to move over to Shader Builder now to illustrate this in action. I'll show you how really pretty cool it looks. So let's close our previous example.

is that in range? Can you see it? Is that good? Okay. So if you can see this, here's the fragment program. I'm not going to go into it on a per-instruction basis on this. It simply would take too long. This, again, this exact thing I'm editing right now on stage will be available as a demo afterwards. So you can go home, you can run it on ShaderBuilder on your new G5 that you buy, and then you'll be able to see this. You'll be able to see it, play around with it.

What I've done here is I've actually commented out. I designed the program in such a way that the exact order of the stages I described to you are the exact order of the instructions in this program. And I put a comment that matches that exactly. So when you look at it and you look at my corresponding slide, you can walk your way through this program after. So let's take a look. Texture unit one, there's our normal map in the bottom right corner of the screen.

In texture unit zero, I have the base texture, which is the gargoyle face. In the rendering window right now, what you're seeing is for every fragment, you're seeing the color, you're seeing the incoming light vectors displayed for you, encoded and decoded as a color. So if I go here and I open up the symbol editor, This will allow me to move the light source around.

So here I'm moving the light position around, and you can see the vectors are updating. Let's pull the light back from the surface. You can see it. Move it close to the surface there. It's right at the light. It's right on the plane now. You can see a little singularity there. So let's put it back. Now, let's start uncommenting some code and watch this effect build up.

Let's calculate the vectors after they've been reflected off the surface. And this also depends on the fragment normal, which comes from the normal map. So let's uncomment those lines. These are the light vectors after reflection. Again, move the light around, you can see them updating right there. Let's now do the dot product that I described with the light direction vector, which is a constant right now.

Move it around again. You can see it moving around. And then let's do the attenuation. And you'll see this makes a big difference. That looks better. And then the next stage is we multiply it by the color of the light. The moment the light is just white light. Well, we want to be able to have that controllable, so we do that.

We've got a colorful light. And then finally, we modulate that with the base texture map, which we now have. That's that. Now if we move the light source around, you can see how good this looks. I'm going to make this window bigger so you can see it more clearly.

We pull the light away from the surface. You see it's getting far away. The light's so far away now that it isn't even showing up. Bring it in really, really, really close. So I think it looks pretty good. And then we can also, of course, we can change the color of the light. That's another program parameter. If I adjust that, I can edit it as a color. You see it's changing the color as well. So multiple parameters as well is supported in Shader Builder. So that certainly should be fun for you to be able to look at.

So, at the end of, I've reached the end of my presentation now. I hope you all find this enlightening. And I hope that you'll go home afterwards and we'll run the new shader builder and look at these examples. Because I think there's a lot to learn here and a lot of scope for implementing these things. So, thanks a lot and I think I'm going to hand back to Travis here.

Thank you, James. If you have any questions on the information that you've seen today, feel free to contact any one of these email addresses up there. I'd like to actually add mine to it. I'm Travis at Apple.com. And again, any questions about the graphics technologies or various things you've seen relating to 2D and 3D graphics at WWDC, feel free to email me with your questions.