Hi. And welcome to the session of OpenGL ES Shading and Rendering. My name is Luc Semeria and joining me on stage will be Alex Eddy. We are both part of the Embedded Graphics Acceleration team here at Apple. Here is the agenda for today. We're going to start by talking about shaders and how to write shaders in OpenGL ES 2.0. Then in the second part of this presentation, we'll talk about advanced rendering techniques to make your games look even better.

So if you have been writing applications over the last year for the iPhone and the iPhone Touch using OpenGL ES 1.1, you are going to learn today how to write shaders and particularly making use of OpenGL ES 2.0. We're also going to learn in the second part of the presentation how to make advance rendering effects in both OpenGL ES 1.1 and OpenGL ES 2.0. If you've been working on OpenGL on Desktop, a lot of this may look like a review. You probably have written shaders already and have used a lot of the OpenGL state techniques as well. But we'll find out some of the key differences between ES and Desktop.

And, you know, along the way you're probably going to learn some new tricks. So let me start by a review of OpenGL ES 2.0 and the programmable graphics pipeline. So here is the overview of the pipeline. If you've been to the presentation, you have already seen that slide. Well, this is the OpenGL ES 1.1 fixed function graphics pipeline. Let me just go through it quickly and just highlight a few of the points that are relevant for OpenGL ES 2.0.

So you start with your model that's composed of, you know, primitives. Triangles typically. And the corner of those triangles are vertices. So in the first stage, you break those triangles into vertices and you pass them to the vertex processing stage. That's particularly those, the transformation of the coordinates from model space through eye space into window space.

That's also, where you compute your textual coordinates. That's also, where lighting is typically done. Once you've done all this, the vertices with the correct windows positions go through the primitive assembly block that puts those back into triangles. Then those triangles get rasterized into separate fragments. Those fragments correspond to pixels that are going to go on your screen. So each of those fragments are sent to the fragment processing stage. And in a fixed function pipeline this is where you do your textual loads, combine your textuals. This is where you do your fog computation and alpha testing.

Then what you end up at the end of the fragment processing stage is a color that you can dynamically display on your screen. What changes with OpenGL ES 2.0? Well, OpenGL ES 2.0 is a programmable graphic pipeline. What it means is that really the two boxes that are highlighted here, the vertex processing box and the fragment processing box, are now programmable. And the way you program those two stages are using shaders. So we have our two programmable shaders and they look like regular code. Almost looks like C. And this code gets compiled and linked to run as a program on the GPU.

So just like you would, you know, write C code, and compile anything in that C code to run on your CPU, well, in OpenGL ES 2.0 you write GLSL, which is a programming language we use for those shaders. And we compile and link those shaders to run on the GPU. It's accelerated on the GPU. And this allows you to do more effects than what you can do on OpenGL ES 1.1. Here are some examples. You can do tangent space bump mapping, especially if you do skinning.

You can do better on the cubic environment map. You can do better effects like reflection. You can do better image processing and, you know, this is programmable so it's really up to you. You can put your own algorithm in those shaders and compute the effects that you want. So that was an overview. Now, let's dive down and see how you actually write one of those shaders. So there are two kinds of shaders as we have seen.

There is the vertex shader and there is a fragment shader. Well, first we are going to look at the interface in the overview of both the vertex shader and the fragment shader. Then we'll dive down into really how you write this GLSL code. So the vertex shader, as I mentioned, replaces existing OpenGL ES 1.1 fixed functionalities. So namely, the vertex shader is where you compute your vertex and normal transformation. Where you go from model space into window space.

This is also, where you do transformation on your fixture coordinates, and you can pass those fixture coordinates to the fragment shader to start drawing things. This is also, where lighting equations are implemented if you do textured effects lighting. But you can do way more things in the texture since it's programmable.

This is where you can distort geometry, as you have seen in the first illustration. This is also, where you can do skinning easily and many more effects that you can come up with What is the interface of the vertex shader? The vertex shader has two kinds of inputs.

The first kind of inputs are called attributes. And attributes are specific to each vertex. These are the characteristics of each vertex in your model. So an example of attributes are the coordinates of each of those vertexes in model space specifically. The normal to each of those vertex tangents or whatever your model has.

Now, there is another kind of inputs, which are called Uniforms. And the reason why they're called Uniforms is because their value is uniform throughout the model. So these are constant data as far as the model is concerned. An example of Uniforms that you may want to pass to the vertex shader are your matrix transformation, your transformation matrices. The light coordinates, because the light is going to be fixed for all the different vertexes in your model.

So based on those inputs, the attributes and the Uniforms, the vertex shader is going to compute a set of outputs. And the set of outputs of a vertex shader are called varyings because they vary from one vertex to the other. The varyings are defined for each vertex, and they get interpolated and passed to the fragment shader to be used in the fragment shader.

So an example of outputs, the first one is the position in window space of the specific vertice. Picture a coordinate that you want to pass to the fragment shader. Colors, you name it. How does that look inside in the pipeline? Let's take a look. So we have our vertex shader that's following the primitive processing block. So we receive a bunch of vertices and the vertex shader is going to run on all of those vertices.

We have a set of generic attributes coming into the vertex. So if you are finding out with writing shaders on Desktop you will notice here that we only have generic attributes. There are no predefined attributes. Likewise, we have a set of Uniforms. They are all generic Uniforms. No predefined Uniforms in OpenGL ES 2.0.

And based on those two inputs, the attributes and the Uniforms, the vertex shader is going to write varyings. Again, there are a lot of generic varyings that you can use for anything you want. Colors, texture coordinates, so on and so forth. There are two predefined varyings. The first one is GL position. This is a position in window space of your vertex that's computed in the shader.

If you are drawing points, you can also set the point size in the vertex shader. You must always write to GL position. You must always set the position of your vertex so that it can be used in the later stages of the pipeline so you can render something on the screen. Let's now look at the fragment shader. So the fragment shader also replaces existing fixed function blocks in the fragment's processing units in OpenGL ES 1.1. So things you would do in a fragment shader? This is where you do your texture loading.

This is where you can combine textures. This is where you can do your fog computation. This is where alpha testing happens if you need alpha testing. But you can do a lot more things. Again, it's programmable. So this is where you could do per pixel lighting, per pixek bitmapping. You can animate pixels.

Anything you want. Again, the fragment shader has a set of inputs and based on those inputs compute a set of outputs. The inputs are the varyings. The varyings are well defined in the vertex shader. They get interpolated in the rasterization stage and passed to the fragment shader. So these are the inputs of your fragment shader. Example of such varyings are the positions, the texture coordinates, the color.

You know, whatever you put in those varyings. The other kind of input to your fragment shader are also Uniforms. Uniforms that are going to be constant throughout all the fragments in your model. So things you would pass here. Your fog characteristics, which texture you need you want to load from.

Any of those characteristics you want to pass in. Based on those inputs the fragment shader is going to compute basically one output, which is the color for the pixel that corresponds to the fragment on the screen. You also have the option to discard that fragment so it doesn't get drawn on the screen.

Well, how does that fit in the pipeline? The fragment shader follows the rasterization stage, and this is where bearings are getting interpolated from the vertex shader to the fragment shader. There are all the generic varyings that you defined in your vertex shader that can be read in the fragment shader. There are three predefined bearings coming into the fragment shader.

The first one is the fragment color, and then the chosen ones are the facts whether or not the primitive to which the fragment belongs is whether this primitive is pointing. Is FrontFacing or BackFacing, so this is in GL FrontFacing. And if you are drawing points, the point coordinates within the points that you are drawing.

The other sets of inputs are the Uniforms, right? There are Uniforms for all those. That are constant for all those fragments. And you can also load from multiple texture units. Based on all those inputs you can compute the fragment color. And again, if you want to draw the pixel on the screen if you don't discard it, you must write to FragColor.

So we've seen the overview of what the fragment shader and what the vertex shader does. And we've seen the interface, what's coming in and out of those shaders. Let's now look at how we can actually program one of those shaders. The way you program one of those shaders is through a language called GLSL. The OpenGL shading language. And GLSL is select, you know. It has a main function, it's procedural.

And you get it compiled and linked to run on the GPU. It doesn't have any of the kind of dynamic features of scene, no pointers, no goto's, no recursions. It's just simple C code. But it has several extensions to make it usable for graphics. And in the last few slides, I just want to highlight some of the specific extensions or features of GLSL. But first, let's start with a very simple example.

This is probably the C+ shader you can write. Here what I'm doing is I'm just writing a constant color, white, to all the fragments in my model. And so if I have the good old teapot it's going to be all white and flat on my screen. This is kind of the matching vertex shader. This could be even C probably. This is pretty simple and typical vertex shader. What we do here is we have two kinds of inputs. We have the attributes that corresponds to the position of each vertex in my model.

And then we have the Uniform. That's my model view projection transform matrix that I'm going to use to transform those position coordinates from model space into window space. And so in the main function all I do is I multiply my 4x4 matrix model view project matrix by my for valued position vector.

And I end up with the transformed position that I passed as the following stages of the pipeline. So let's look at it in little bit more detail. First thing, you notice at the top are the inputs and outputs definitions that are part of the global definitions. So you have a set of storage qualifier that you can use to define those inputs and outputs.

So you have the attributes of storage qualifier to define, you know, your core vertex attribute inputs. The Uniform qualifier to define these other Uniforms of the fragment shader or the vertex shader. The varying storage qualifier that's used to define the outputs of your vertex shader and the input of your fragment shader.

And if you have constants and you want to let the compiler know that those variables are going to be constant and can be optimized, you can use the const storage qualifier as well. Any language as datatypes. So GLSL has all the basic datatypes that you're familiar with. Floats, integer, Boolean. It also has vectors. It can have two component, three component, four component vectors.

And they can be of type floats, integer, Boolean. GLSL ES has also square matrices that are always for input. So if on Desktop you have non-square matrices here, we have only square matrices. All those datatypes are either logical or arithmetic datatypes. So they have the usual operators that are defined for those.

And what's interesting here is multiplier operator all these different matrices. So you can use multiplier operator to do matrix multiplication. You can use the multiplier operator to do a matrix by a vector multiplication, just like we did in the original shader. Now also datatypes, such as the central datatype.

The central datatype is used to pass in the information about which textual unit you want to load from. And there are also standard datatypes, arrays, structures, and you use them just like you would them in C. In addition to the operators I mentioned, GLSL has also a wide variety of built-in functions. So here is a list. It's not complete, but it's already quite a few, built-in functions of GLSL.

I'm not going to go into details on one of those but, you know, you can recognize your favorite arithmetic built-in functions. You know, sine, cosine, power, absolute value, so on and so forth. I want to highlight some that are specific to graphics. So if you have a vector you can normalize it using the normalize function. If you want to implement reflection or refraction, you have built-in functions to help you in your algorithm.

You have specific functions to do your textual lookup. And we also support the derivative functions that are part of an extension in GLSL. So you can do derivative in the X dimension of the Y dimension. With all this, you can start writing more real shaders. So here is also a fairly simple example that does textual mapping. Here is my vertex shader. It looks pretty much the same as the one I showed before.

There is one more attribute we have here. It's the texture data that is part of my model. And in this case in the main body of the function we are just passing that through to the fragment shader. I'm also outputting this textual coordinate as a varying so that I can pass it to the fragment shader. And I have my usual, you know, position model view defined by the model view projection matrix multiplied by the position.

If you've seen shaders on Desktop you've probably seen that people use ftransform everywhere. Well, ftransform was not part of my initial building functions. So in GLSL ES you need to explicitly do your computation. This is part of the fact that we don't have predefined Uniforms here. They are all generic Uniforms, so you just define which model projection matrix you are going to use.

So here we computed our texture coordinate varying that gets passed to the fragment shader. And here in the fragment shader I'm going to load from two different textures and just combine them by adding the colors. So I have 2D Uniforms. Texture1 and Texture2 that are defined as sample2D Uniforms.

And I use the texture2D function to load from those two textual units. And then I just simply add those two colors and assign that to my fragment color output. If you look at this shader a little carefully you will notice two things I haven't mentioned yet. Highp and lowp. These are GLSL ES specific precision type qualifier. Well, this is something that doesn't exist on Desktop. This is GLSL ES specific.

And what they mean is that the way you use them is you use them to specify to the compiler that the variables that you defined using those qualifiers have a reduced precision. So you can use lowp. Low precision, medium precision, high precision. And these are just compiler hints to let the compiler know about this limited range.

So for example, if you add the color and you know the color is not going to go out of the range -2 to 2 and you only 8 bits of precision for that color, well, you can use the lowp qualifier for that color variable. But what happens if you don't use any precision qualifier? Well, in your vertex shader you don't have to. This is defined for you. There is a predefined default precision, which is high precision.

So you don't need to worry about precision in your vertex shader. This high precision. But the OpenGL ES specification leaves the default precision in the fragment shader as undefined. As a result you must define a precision for all your floating point variables in your fragment shader. But the way you do this is simply by adding one statement typically at the top of your shader that says, my default precision in my shader is going to be, in this case, high precision for all my floats. And then the rest of your shader can be just a generic shader and you don't have to worry about precision.

Notice that if you define a default precision for float that applies to vectors, matrices, arrays, floats and so on and so forth. So putting this all together this is how, you know, a typical fragment shader would look like. At the top you have your default, you know, precision setup.

And in the rest of the program it's a pretty much a generic shader here and we just use lowp here for colors, but you don't have to. So this gives you an idea of what a shader looks like, so now you know how to write a shader. Express your computation, your algorithm in one of those shaders.

How do you actually use it inside your application? Well, the way you do that is, as I mentioned, you have your fragment shader, your vertex shader. And they get compiled and linked into a program. And the way you do all this is through the typical OpenGL APIs. So to compile a shader, like the vertex shader, the first thing you do is you create a name for the shader. You create a shader object. Then you attach your source card to that shader object you just created. Then you can compile your shader. Make sure the compilation has succeeded. You do the same for the fragment shader.

Now you have both fragment and object shader that are compiled. You can create your program. You attach the compiled vertex shader and fragment shader to your program. Now that you have attached both compiled fragment and vertex shaders you can link your program. Verify that link has succeeded. And then you can use the program and start drawing things on the screen. This is something you want to do in this initialization step of your application. You don't want to compile and link your shaders while referring. This is part of the initialization part of your application.

This is good. Now you have your program ready. But how do you actually set the inputs so that something gets computed, right? So the way you do that is you set your attributes and Uniform values, and typically you do that for every frame. If things move, Uniforms may change for every frame or attributes may change for frames.

And now a couple of things you want to do still in the application, in the initialization of your application. After you link the program you want to query the location of your attributes and Uniforms. And then in the main body of your application after you selected which program you want to use, then you use the location that you just, you know, queried in this initialization to set the Uniform values. Then you can set all sorts of attribute values. And then you draw things on your screen. So that's pretty much all the steps that are involved to create and run the shader. So with that, let's go to the demo machine and actually see how it looks.

So here what I have is a simple Xcode project that's basically the template. If you have used OpenGL ES 1.1 over the last year, you probably have seen this template. And you can find the usual files here. The one thing I changed here is instead of using OpenGL ES 1.1, I switched to OpenGL ES 2.0.

So let's go and look at where we initialized our GL context and see how we compile and link things. So I'm jumping to the function in it with color. This is where GL is being initialized. And if we scroll down a little bit we can see that here I am creating my OpenGL ES 2.0 context. Make sure that the context gets created.

If I'm on one of the iPhone 3G or iPod Touch, then this would fail. Then the next thing I do is I call the function that's going to compile and link my shaders, so let's jump into that function. This function is using a separate function that does the compilation part, so let's look at the compilation part first. That's the function compile shader here. So what I do here is for the vertex shader and the fragment shader here I'm creating the shader object. I get a name. Then I attach the source code for that shader that was initially stored in a file that I'm reading.

Then I can compile my shader. And then for debugging, say if I did the syntax there in my shader or any other error, I need to check the info log because this is where all the warnings and errors are being stored. So in this log here I'm dumping all this onto the console. And this works both in the simulator and on the device. After I checked and printed those error and warnings messages, I make sure that the compilation has succeeded by checking the compilation status. And then I return my shader.

Now in my main compilation and linking function, you can see that, you know, I compiled both the vertex shader and the fragment shader. Now I can create the program. The way I do that is I create the program. I attach the two shaders, vertex and fragment shaders. And in this case here I am buying my attributes, and then I link the program. Same thing here for debugging. I want to print my error and warning messages.

This is what I do here by checking the info log. And I make sure that linking succeeded. Then I can query the Uniform locations that I had in my model. And I'm going to use this later in the main loop to set both the attributes and the Uniforms. Then I can use the program.

So let's take a look at in the simulator how this looks. So here we go. This looks pretty much like the simplest shader I showed you before, right? Here we have the good old teapot, flat color. Let's make it more interesting. So here I'm bringing up the source code for most of my vertex and fragment shader. Let's start with the vertex shader. What I'm doing here is just computing the position based on the model view projection matrix multiplied by the position.

Let's add some lighting to this. So I already set up the input so I can compute lighting here. And the way I compute lighting is I take the normal attributes that I have in my model, and I have the light direction. And here what I am doing is just to use lighting. Some very, very simple lighting. And I just do the dot products of the normal for the vertex with the light's direction. That's Uniform because the light is not changing for every vertex. And if both align then I want to make things brighter.

And if they are in opposite directions I make things darker. So I just compute the dot product here, and I assign this to the light bearing and pass this to the fragment shader. While I'm there I'm also going to pass the pixel coordinates to my fragment shader. Let's now look at the fragment shader.

Here in the main body of the code we can see that we set the color to a constant value, yellow. So this is why everything is yellow. Let's now use lighting. And the way we do that is we just modulate the light, the color, by the light component that I just computed.

Let's run that and make sure the lighting works right. So I can check that it's lit properly. In animated, it works fine. Let's make it a little more interesting. You know, as I said, you know, you can do texture loads in your fragment shader, so let's use those texture coordinates and do some texture loads here.

So I have a Uniform coming in that tells from which texture I want to load my data from, and I just use my texture coordinates to load the color for that fragment. And then to make it slightly more interesting I use that color and use that color to lookup into a palette to make things more colorful. And that's going to be my final color.

So let's run this. And now we have our little teapot that also has a texture on that to it. And I can also, in this case, I passed in a Uniform so that I can animate the way I look up in my textures. So this is a little plasma effect. I can use this to do also some simple water effects.

Also effects that you would see in the second session. So with this, you know, I'd like to hand it over to Alex who's going to show you more interesting rendering techniques.

OK. So, so far you've seen an overview of the OpenGL ES pipeline. And Luc gave you an overview of GLSL, the shading language.

What it is. What you can do with it. And how it works. Now, for the rest of the session I'm going to be talking about some advanced rendering techniques. How you construct and use all this information to make some pretty pictures show up on the screen. And I'll be touching on a couple of different topics here, starting with some 3D effects.s=tylY Then I'll talk a little bit about 2D and image processing.

And finally I'll wrap up by combining 2D and 3D together. And I'll be talking about how to achieve these rendering techniques and effects in both ES1 and ES2. Now, that's because, as you heard on the keynote on Monday, there are now 40 million iPhones and iPod Touches out there. That's a very large installed customer base, so this device is running ES1.1. And that number is still growing. We are still selling iPhone 3Gs and the new iPhone 3GS supports both ES1.1 and ES2.0. So we think that ES1.1 is still important and relevant to your application.

And as we'll see a lot of these effects can be done in both ES1 and ES2. But why are we talking about this at all today here? Why do we care about rendering techniques? Well, it's a pretty simple answer. We just want you to make your application look as best as it possibly can. OpenGL is about more than just drawing flat textured quasma screen.

There's a lot of depth and functionality in this API. And there's a lot of silicone there in the GPU waiting for you to use it. So we want you to use that hardware acceleration to its fullest, and make your gaming application look really great. Now, in a little more detail OpenGL provides multiple attributes to you, and you can combine these in different ways. You can also use multiple texture simultaneously.

And a really key point to achieving a lot of these effects is learning how to combine textures together in interesting combinations. In ES1 that means using APF features like the Texture Matrix, or the Texture Environment combine modes. In ES2 of course you're going to use program mover or text and fragment shooters. We can do all these things and a lot more. Now, to give you an idea about what type of effects we're talking about let's go to the iPhone and look at a live demo.

Now, here you can see I had a pretty good looking 3D scene. What are we looking at specifically here? Well, I have an interactive 3D skybox, and I have an anti alias good looking tesilated model here that I can interact with and zoom in and out. And you notice, if you look at this carefully, that the object is reflecting the environment around it. It's not just a flat textured object. The object is responding to the environment, and it's not just the reflection.

There's actually a little bit of dynamic part to this reflection, so it reflects around differently when you put in how it's oriented. So this effect is an environ map using a fresnel term that dynamically changed how the environment is reflected. This is an example of the type of effect we're going to build up today step by step.

And you can do these effects in both ES2 using shaders and in ES1.1. Now, there are several other effects built into this demo. Just to give you a brief look at the type of effects we're talking about, I want to go through a couple of them quickly. This effect has a couple models also.

If you like car games here's how that same thing looks on a car. And just to give you an idea, that fresnel term by itself kind of gives you this edge highlighting effect so we see the edges are lighter than the middle of the body of the car. If you combine that with some lighting we're going to get a couple interesting backlighting and room lighting effects.

Or we could use that fresnel term for different purposes like blending. And we can make some interesting actually budding effects, which may change interactively to achieve a variety of blending styles. Or a couple other ideas for effects here. We could do some none photorealistic renderings such as toon shading. Again, this can be done with hardware acceleration in ES2 and in ES1.

For a more complicated version of this you could do some anisotropic lighting effects and maybe I'll show the good old teapot here again. You can see you can get a kind of nice looking brushed metal effect. So this is just a very quick overview of the effects we'll be talking about in this session. I'll go back to the slides.

Now hopefully you're asking yourself how can I do that in my application? And let's be honest here. The first step is to learn OpenGL. To do these kinds of advanced rendering techniques you're going to have to learn OpenGL and get comfortable with these in the pipeline. Now we don't have time to teach you everything about OpenGL in this session, but I want to point out a couple of those hidden knobs and switches in the pipeline to show you how you can start taking control of the pipeline and make these kinds of interesting effects.

We'll also be looking at an example of how you can redefine your problem, or your rendering technique, in terms that your OpenGL API can understand, and the hardware can conceal and accelerate. In ES1, again that means using texture coordinates and features like the texture matrix and the texture environment combiners. In ES2 the same thing can all be done with shaders. We'll also be looking at breaking complicated effects into multiple pieces using multipass and rendered texture.

So let's get started looking at the pipeline. Now you've already seen today an overview of the pipeline. I want to go into just a little more detail in a couple key parts. In each part I'll show you how the pipeline is typically used, and how you can modify the behavior a little bit to get some more flexibility out of it. So starting with vertex attributes here, anytime you draw something with OpenGL you're defining a bunch of data that OpenGL is going to transform. Those are the vertex attributes.

Things like the position and the color and the normal and so on. And there are a lot of ways that you could lay out that data for OpenGL to consume. But here's a diagram of a pretty typical way a bunch of different attributes can live in a single vertex buffer object. Now you can see the attributes here on things like the XYZ position, the normal.

Maybe there's a texture coordinate ST or a per vertex color. And when you want to tell OpenGL to use this data, you first tell it what the data is and where it is. With these APIs like VertexPointer or TexCord Pointer, you point GL to the data and you say it's 2 floats or 2 bytes or whatever it is. And then you call draw erase or draw elements and that kicks off the pipeline, pushing this data into the transformation stage so the OpenGL starts processing it.

So that's the next stage. Vertex transformation. Now again, this is a pretty standard overview of the pipeline here where the positions are coming in on the left, and they're transformed through a series of spaces, eye space and clip space and so forth. Finally ending up in window coordinates.

And I want to point out here that your inputs into this, the way that you can control this pipeline, are these matrices. The modelview matrix and the projection matrix. This is your control to tell OpenGL how to transform the data. To scale it or rotate it or project it. How do you want to change the data so it shows up on the screen directly for your application? You're probably all familiar with this already, but I'll point out another part of this that you may not have used before.

We also have texture matrices. The hardware has multiple texture units, and each of those units has a matrix associated with it which works just like the other matrices. Except that instead of transforming the position data, it transforms the texture coordinates coming in. And they work just the same way. You can do the same transformations like scale or rotate or projection. Anything you can do with a 4x4 matrix. As we'll see this is extremely useful for influencing and addressing effects.

The first is just a little thought experiment for you to think about. You'll probably recognize this. This is the Touch Fighter application that we showed a year ago with the iPhone SDK 2.0 launch. And if we remember you're flying around this 3D planet while you're battling these aliens in space. Now that planet is defined with some 3D geometry. It's a sphere.

And as the game plays the planet is rotating towards you. By rotating those positions by the modelview matrix. Now if you think about it, you could achieve the same type of effect by using the texture matrix. Instead of rotating the position if you leave the positions where they are, and instead rotate the texture coordinates around the geometry.

It's exactly the same rotational transform what you do on the textual matrix instead of the modelview matrix. Now why would you want to do that? Well, it depends on your application. But here you can see that you can't see all the planet at the same time. If you're never rotating it you're never going to see the back part of it.

And in fact, in this picture you're never going to see the bottom half of it either. So maybe you can get away with defining less geometry. If these are texture matrices this way you could save yourself some video memory, and you could save yourself some transform costs because you have less vertices to push into the pipeline to transform. So there's a simple example. And now we've talked about basically all of the vertex transformation stage. Finding attributes and transforming them.

So that takes us to the fragment part of the pipeline. Now Luc showed you how ES2 has a really flexible fragment pipeline with the programmable power of GLSL. In ES1, you're a little more constrained to what you can do. And the fragment pipeline is really defined in terms of texture combiners. So here's a diagram showing you what type of things you can do there. You have a couple of inputs coming in which are things like the primary vertex color.

That's the lit color that was processing the vertex stage. You also have textures coming in that you're sampling from on the current unit. And with those inputs you can do some very simple math operations like modulate. That's a multiply or an add or a subtract. And a couple of other simple operations like interpolate or dot product which are really just multiplies and adds combined into a single operator.

After these operations are done, you can additionally scale the result by 1 or 2 or 4, which gives you a little bit of extra flexibility for creating effects. And I want to point out here that the color data that you're working with is all clamped to [0, 1]. So that's both this vertex color and the texture color. This can be a little problematic for some effects which seem to work with a wider range of data.

But in the 2D section of this talk I'll show you a way how to get around that. And I also want to point out here that again the hardware has multiple texture units. So you can combine these texture environments together to create more interesting complicated effects. For example in the first unit you might do a multiply between the primary color and the texture. In the second unit do an add from a different texture. For example, add a highlight. Now the number of texture units depends on the hardware you're running on. On the MBX base devices you have 2 units.

And on the new iPhone 3GS you have 8 units available. This kind of limit is a thing that you can query out of OpenGL dynamically at run time in your application so you can be sure that your algorithm will work in available hardware. Now what happens if your algorithm doesn't fit in the units that are available? You query it and you find you have 2 and that's not enough for your effect. Well, you can break your algorithm down into smaller pieces using multipass rendering.

There are a couple of ways to do this, but a simple way is simply render the first part of your effect to the framebuffer normally. Then in the second pass change the state you're using. Maybe use different textures, different combiners, and render the second half of the effect using blending to add the result into the framebuffer. Drawing the same object for the same position, but with different state and blending turned on.

It's a pretty simple trick, but it works in built-up complicated effects. Another similar approach is to use render to texture. Here is the same idea, except instead of rendering the first pass to a framebuffer directly, you create a temporary texture to capture the intermediate results. And that texture can then be fed into second or possibly multiple additional passes to build up more parts of the effect or do post processing or image filtering.

And the final result is drawn back to the real framebuffer. So that's the points I wanted to touch on the pipeline. Just a bit of summary there comparing ES1 and 2. In ES1 you only have a couple of attributes available to you. Things like the normal and the toucher coordinate and the positions. And the transformations you can do on those are fairly limited because it's a fixed function pipeline.

Although I showed you how to use a texture matrix to get a little more flexibility out of it. And again in the final stage you're pretty limited in what you can do because you only have simple math operators. But there's a very large number of ways that you can combine those operators together to create interesting effects.

And because there's only a couple of texture units available on the MBX based devices, you often have to break rendering techniques into multiple passes. By comparison, in ES2 you have a lot more flexibility. You have more attributes available and they're completely generic for you to define and use it anyway that you want. You also have the full power of GLSL, the programmable vertex and fragment shaders.

So you can do all of the things you did in ES1 plus a whole lot more. Now let's use all the information I just described to build up one of these effects step by step. And we'll look at that environment map sample that I just showed you at the beginning.

So let's start out by making a completely reflective object, and then we'll improve the technique a little bit to add that fresnel term and make it look a little more realistic. Now the concept behind this kind of technique is that we have a 3D model which= has a bunch of attributes defined for a vertex like such as the normal.

What you can see here the normals are pointing out in all directions from this model. We also have a texture which in this case contains a spherical projection of the environment, as if you are looking around the scene from the center of this object. Now the basic idea is that we want to take the normal in one of those vertices, and use it as a texture coordinate.

And to do that the normal will be transformed and projected into 2D space to lie on this texture. Wherever that projected point lines up on the texture we'll use that color for that vertex of the model. And if we do that for all the vertices it ends up looking like the object is reflecting the environment around it.

Now here's how you can achieve that in ES2 using the shader. So for starters the vertex shader. Now you can see at the top there I have an attribute coming in which is the normal. I also have a Uniform defined which is that environment matrix. Now this matrix is calculated by the application during every framed animation to match the current rotation of the object as its being moved around.

That way the normals are transformed the same way that the object is rotating. And the body of the shader you can is pretty simple. The first line of code is simply using that matrix to transform the normal, and it's passing it out as a texture coordinate to be used in the fragment stage. The rest of this is very trivial. It's simply passing through the vertex color that was defined in the model.

And it's doing the standard transformation on the position to go through eye space and clip space. Here's the fragment shader. Now start out with the model is completely reflective, so this is very simple. You have the texture coordinate coming in which is just computed in the vertex shader. And you can see that I used the texture sheeting function to look up that environment color in the texture, and I simply assign it to the output and I'm done.

The entire model is reflected using an environment map. Now the slight improvement on this is to add a fresnel term. In this case I'm using the mix function which is built into GLSL to interpolate between that environment color and the original vertex color of the model. And I interpolate in here by the alpha channel of the texture that I just looked up. In this case the application has repaired the alpha channel with the fresnel term, which is simply a gradient ramp that falls off from black to white. So that's how you can achieve this kind of effect in ES2 using shaders. You can do the same thing in ES1.

Just a little trickier. So going back to this partially the pipeline that I touched on look at the vertex attributes. Now normally when you have texture coordinates you tell OpenGL to point the texture coordinate pointer at that texture coordinate data. Now here's a trick. Don't do that. Instead, point texture coordinate pointer at the normals which are already defined in your vertex data. Now those normals can be treated as if they were texture coordinates and you can transform them by the texture matrix. So here's the texture matrix stage.

And the idea here is that the normals were defined in unit space so their 3D normal is pointing in all directions on the unit's sphere. The first thing that's going to happen to this is they're going to be rotated by the modelview matrix, so the thing matched the orientation of the object for the current frame of animation. Additionally we need to scale and bias that normal data to fit in the range 0 to 1 so that we can use it as a texture coordinate to look on the 2D environment map.

Here's the code in ES1 that sets that up. Now you can see the first thing that happens is I set the matrix mode to the texture. After that it's using the same API that you're already used to, doing some translates and scales and matrix multiplies, but they're going into the texture matrix.

And in the simple case where the model is completely reflective, the texture combiners are simply set up to completely replace the color with that texture. A small additional improvement of the fresnel term changes a couple lines of code here to set up an interpolation operator in the fragment stage.

And here again you can see that it's interpolating between that environment texture and the primary vertex color. And it's interpolating by the alpha channel of that same texture. So this is a pretty simple effect, but it ends up looking good. Now to really drive this home, let's go back to the iPhone and build this up step by step.

So here we are back in the 3D scene. And if I back up a couple of steps here you can see now that I'm visualizing the normals which are defining this object as colors. And in these two this kind of a debug visualization is really easy to do, because you have the normals coming in as attributes and you can simply assign them directly to the color.

It's really easy to visualize the data and how it's coming in before you do any operations on it. In ES1 this is a little harder, but it basically uses the same tricks that I just showed you using the texture matrix. To do that in ES1 you can define a couple of special debug textures like this which are simply ramps to go from black to red and black to green and black to blue.

Now you can see how the normal data defined in the model here is mapping to the texture colors. So for example, all the normals are pointing downwards on this plane are being mapped to the green at the bottom of this first texture. And all the normals that are blue on the right side of the plane are being mapped to the right side of the second texture. So that's the basic setup. The next step of this effect was to transform the normals by modelview matrix to keep them in the same orientation relative to the camera.

So if I go there you see the coloration has changed a little bit. Now regardless of how I rotate the object around you see the normals are always pointing in the same direction. In this case blue is always pointing towards the camera, and green is always pointing down regardless of how the object is being rotated. Once the state is set up, either in the shader or using the texture matrix that I've just shown you, all you have to do is replace these textures with the real environment map.

Now the object looks like its completely reflective. Small, simple addition to use in the fresnel term in the alpha channel looks like this. In here, you can see the alpha mask and the alpha channel in the same texture is simply the gradient in the sphere which goes from dark in the middle where the object needs to be using more of the original vertex color as on the parts that are facing you. To lighter and then actually white on the edges of that sphere for the edge parts of the model that are perpendicular to the camera are going to be completely reflective.

Now with a little more work it's possible to dynamically change the context of this texture to change how the environment mapping is happening. I make it completely reflective again, or I can change the falloff term to get a wide variety of artistic effects, depending on the type of object and the type of an effect you're going for. So that's an environment map using a fresnel term. Now. Thank you.

Now we don't have enough time in this session to go through this much detail on all the other effects. But I want to go through them again to show you how these basic concepts can be built upon and expanded to build up a variety of other interesting effects. So if you look at just the fresnel term again, here again you can see that it's kind of highlighting the edges of the model.

And if you combine that with something else that's not an environment map, for example a regular vertex lighting, you can get kind of this backlighting or room lighting effect. Here you can see it's kind of highlighting the edges of the model regardless of the orientation. And this kind of backlighting effect has been popularized recently in a couple of console games you guys may be familiar with Or another idea is use the same kind of fresnel term for something other than lighting.

And again you could use it as a blending factor. So here you can see that the parts of the card that are directly facing you are transparent, while the parts that are perpendicular are being dynamically blended together. And again you could dynamically change the content of the texture to change how the blending looks and build up a variety of interesting effects.

Now the same type of technique using the texture matrix can get you these kind of effects like toon shading. Again here, you know, I can interact with this a little bit to move the light source around. Luc showed you earlier the very basic lighting technique of n.l. That's a normal dot product with a light. Well, you can do that with the texture matrix.

You're feeding in the normals of attributes you can put the light vector into the first column of the texture matrix and have it do that dot product for you. Once you have that lighting term, that can be used to lookup in this kind of compared texture with a couple of grayscale ramps in it and make this kind of none but realistic effect.

And of course it's all hardware accelerated. The more advanced variation on that is to do both diffuse lighting term and spectral lighting term. So it's n.l, nn.h, where h is the halving the vector between the light vector and the eye vector. So here again with a specially prepared texture you can achieve a wide variety of anisotropic lighting effects. Now take a second and compare this to the vanilla OpenGL lighting. Here's what that looks like with just the diffuse and spectral lighting terms.

Now what do you want your application to look like? So hopefully you can see how if you use the features that are built into the API you can really dynamically increase the quality of the rendering in your application. This is what we want you to be able to do with OpenGL. Now so far all these techniques that I've shown you are implementable both in ES2 with shaders and in ES1.1. And of course with ES2 you can go much further than this.

You have shaders, so let's take a look at an example of a couple of effects that you need shaders to do. So here, and now deforming the teapot, and this is doing a custom transformation of the vertex shader. As Luc pointed out there are a lot of map functions built into the shading language. Things like sine and cosine.

So it's actually quite easy to make this kind of custom transformation. This actually looks pretty good on the car model. And if I change the environment terms there you can see how it's deforming both the vertex position and also the normals, so the environment map track the positions correctly. So, very easy to write with GLSL and the ES2.

Here's another example using the fragment shader. So here this is the same environment map effect, except at the end of the shader where I have the color calculated I'm going to use that color as a texture coordinate into a special kind of texture like this which contains a palette.

And again, a Uniform could be used to animate this coordinate around to change the values a little bit every single frame to make this kind of thermal imaging effect. Now this is what we call an independent textual lookup or a dependent texture read. This type of thing cannot be done in ES1.1. You have to have ES2 for the shaders to do this kind of effect. So that's about all I'm going to talk about on the 3D section of this talk.

We'll go back to the slides. So I showed you a bunch of different effects, a lot of which can be done both in ES2 and ES1. There's lots of more examples. You can come up with your own by combining all these techniques together. In ES2 there's even more. You have all the power of GLSL to you, and I'm really interested to see what you can do in this next year when you get your hands on the iPhone 3GS. And to change topics let's talk about 2D and image processing.

These iPhones all have cameras built into them. It's pretty natural that you might want to take a photograph and then manipulate the pixels in the image a little bit. For example, maybe you want to make a grayscale version or change some brightness or contrast, or rotate the user round. These types of simple image processing can be accelerated with OpenGL. Again, both in ES2 using shaders and it turns out also in ES1.1. So let's take a look at a simpler filter which is a grayscale conversion.

There are a couple different ways you could do grayscale conversion, but the simplest version is to just simply make an average of the red, green and blue channels. So if you multiplied them all by .33 and add them together, you get the average illuminance value. And here you're looking at GLSL fragment shader to do that kind of effect. And the key part of it here is this dot product operation.

That dot product is doing its multiplies and add's. And you can see that I first took a sample out of the texture and I'm doing a dot product with these weights which are just simply constant values like .33. It's a very simple shader. You just use grayscale in this processing effect. Now you can also do this in ES1.1, although again it's a little trickier.

And here the texture environment gives you an operator which is a Dot3 to do these kinds of dot products. But that operator is really designed to do bump mapping using normal maps. So if you want to use it to process regular colors and image, like a photograph, you first have to bias the data into the proper range. Now you can do this by using an interpolation.

So you'd have your original image there. You can use an interpolation towards white to squish the data into the proper range of .5 to 1. Once you have done that you can then use an additional unit to do a Dot3 operator and find the average color converting the image to grayscale.

So that works into our accelerated and actually runs pretty quickly. Here's the code showing you how to set up in ES1 and I'm not going to go through all this. But you can see the basic idea is that the first unit is doing interpolation. The second unit is doing the dot product. And you notice there on the top the weights that are being used have also been biased and scaled into the proper range.

Instead of being .33 they end up being .667. Now that's one filter. For a little more advanced image processing theory there's a nice little set out there, a Graphic Obscura, with a couple articles that I thought were pretty good. And one of them points out that you can do a variety of imaging processing effects by using simple interpolation and extrapolation operations.

So in this table you take the center 1.0 column as the original image. You could interpolate towards a degenerate version of the image, like a grayscale copy. And you could achieve all the intermediate desaturative values as you interpolate towards it. Or if you extrapolate away from it you get the opposite effect, increasing the saturation and making the colors more vibrant.

Or another example, if your degenerate image was a blurred copy you can get all the intermediate and blurred values for the interpolation, or get the opposite effect of sharpening the image by extrapolation. Now this is an interesting topic because it's not immediately obvious how to map this acronym to OpenGL, especially in the confines of ES1.1.

The left-hand side of the table is just interpolation which turns out it does map really directly to the interpolate operator which you can do in texture environment. But as I pointed out earlier all those terms in this texture combiner have to be in the range of 0 to 1.

So that includes not just the pixels and the texture, but also this interpolator value T here that we're using. So how to you deal with this if you want to extrapolate where T is bigger than 1? Well, it's actually pretty easy. You could use a little bit of algebra. You can substitute in 2T for example.

And then factor out that 2 to redefine this equation. Now all of the terms including T are back in the proper range 0 to 1. Now remember the texture environment can multiply the result of an operation by 1, 2 or 4. So this all maps to the hardware to be accelerated, even in ES1.1. Now let's go back to the phone and take a look at that to see how it performs.

So here you can see I have a pretty looking butterfly wallpaper with a couple of modes at the bottom of the slider. And it works pretty much just as you expect. I drive my finger on the slider and the effect updates. So I'm going to change the brightness here or other effects like the contrast. Decreasing contrast or increasing it. And hopefully you notice here that this is completely fluid and interactive. This is being accelerated with OpenGL, and the image updates just as fast as I can move my finger around the slider.

This is running full screen at 60 frames a sec`ond. Here's a saturation operation. Desaturating to grey or increasing the saturation to increase the vibrancy of the colors. You can do this in both ES2, of course, or in ES1.1. And it turns out that you can do this in ES1.1 even on the first generation iPhones at completely fluid frame rates if you use OpenGL.

Here's another effect which we haven't talked about which texture rotation. Now rotating colors in OpenGL is really just like rotating vectors. It's the same math of the matrix side of the vector, except that instead of XYZ for position it's RGB. Well, matrix multiplies nothing more than a couple of dot products, and those dot products map to the Dot3 operator which is available in a texture environment. So again this can all be accelerated.

And finally, another example interpolating towards a blurred version image we're extrapolating away from a desharpening image. And again, this is all completely fluid and interactive, and a full screen 60 frames a second even in OpenGL ES 1.1 So there's an image processing on static 2D data. But if we stay on the phone here for a second it turns out that we can do the same types of things in 3D. So here we're back in the 3D screen with this object again.

Now watch this. Now I've run a full screen image processing filter on my dynamic 3D scene. And you'll see it's still nice and interactive. So what's happening here is I'm doing the first pass rendering the 3D geometry into a texture. And then additional processing is happening. So you do that exact same grayscale conversion filter that I just showed you in 2D. And of course there's a whole huge variety of effects we could do here.

For example, here's a bloom filter. Everybody's favorite filter maybe. Where a blurred version of the image is being added back on top of the original scene to get that kind of dreamy, bloomy, glowy feel. Or you can do more in mass processing which use the 3D depth information of the scene that you've just rendered. For example, here's depth of field. Now here you can see that as I move the object off into the distance it becomes blurry. And in the midground it's in focus.

And in the very close foreground it becomes blurry again. And this is being done by using all the tricks that I've just shown you in this session. In this case I'm transforming the window coordinate position, the Z value, by using the texture matrix transforming into texture coordinate. And I'm looking up into this specially formatted texture which contains a focal point. And again, I can animate what's in that texture to dynamically change the focal point of the scene. I can bring the near part of the plane in focus, or the midground or foreground.

So there's some advanced image processing techniques. And of course you can combine all these things. So you can go through all these effect with image processing turned on. And then you do the most complicated version in this demo, combining the vertex deformation using the shader with a full screen processing.

And you can see it still runs just fine.

So go back to the slides. Let's talk just a little bit about image processing in full screen. And the trick there is really to use framebuffer objects. As I pointed out earlier you can do rendering texture operations with framebuffer objects.

And the basic idea is to render the 3D scene in that first pass into a temporary texture. That texture should test the framebuffer object. Secondly, after the texture has a content in it we can use that texture as input to additional passes of processing such as post-screen imaging processing.

And the final result gets written into the framebuffer. Now it turns out that you already know how to use framebuffer objects. You're using them all the time, any time you draw anything on an iPhone. And as was shown in the first session this diagram is showing how framebuffer objects interact with the EAGL windowing system, just to get it kind of on the screen on the screen in the first place.

The only change to this setup you have to make to render to texture, is instead of using a color render buffer, you use a color texture. And this texture is just like any other texture that you already use. You gen it, you bind it, you define the format and the width and height with GL tech image.

And after you have the texture created you attach is to the framebuffer object with GL framebuffer texture. At that point that FBO is ready for you to render that first pass of content into. So here's an example of how that bloom effect is built up. Is using multipass of render to texture.

And that first set was to render a 3D scene into a texture with an FBO. And then additional processing is done to it. And the first step I did was to downsample it, and this does two things. First, it blurs that scene a little bit which is good for bloom. It also reduces the number of pixels that we're going to be processing, because the more and more steps that we have of course there are performance overhead there.

So downsampling is good to reduce the flow rate. Secondly, I can do a real blur operation to really fuzz that image out, and also darken it a little bit so that only the bright parts of the original scene will end up contributing to the overall bloom effect. Finally, once I have that version of the image I can simply add it back on top of the original frame.

And that combines to form the final bloom effect that you saw. Now there are a lot of other examples here. I showed you depth of field, probably the most complicated one using all these tricks to change the focal point interactively. And that's really about the end of this session. So to wrap this up the key points here are that to get the most of OpenGL you're going to have to learn OpenGL.

There's a lot of functionality here, and we touched on some key points which in ES2 is all about using the vertex and fragment shaders. In ES1 it's about using the capability you have. The attributes that you have and the fixed function transformations you have doing things such as using the texture meters to give a little more flexibility to that pipeline. You can also use texture combine to combine multiple textures together in interesting ways. And we talked about breaking down complicated rendering techniques in the multiple stages by using things like render to texture.

Now there's a whole bunch of tutorials out there on the Internet where it can teach you all these types of things about OpenGL. But I want to point out the one that's been there all along which is OpenGL.org. There's a section there on OpenGL.org which has tutorials and sample code. And in particular they've collected a bunch of these scene graph presentations from previous years which are about the advanced OpenGL rendering techniques. Now those sessions predate OpenGL ES, so they won't work directly on your iPhone, but the concepts there are still good.

And with a little work you can port those examples onto the phone at OpenGL ES. Also for more information about programmability and the ES2, there are a couple of good books which have come out recently. There's the new OpenGL ES 2.0 guide which is the gold book, although it's supposed to purple and not gold. There's also the big orange book, the Open GL Shading Language reference. Both of these have a lot of information about GLSL and the programmable pipeline. So if you have interest in that please check these out.