My name is Dan Omachi. I work for the Apples GPU Software Team on the OpenGL Framework for Mac OS X, as well as the OpenGL ES Framework on iOS 4. I hope you guys are here today, because you'd really like to add some stunning visual effects to your Mac or iOS applications. Perhaps you thought of adding some shadows, reflections, or refractions into your app.

Maybe you've heard of some advanced techniques, such as parallax occlusion mapping, tone mapping, or deferred shading. I'm not going to be talking so much about those advanced techniques today, however, I am going to be talking about some essential design practices that you'll need to consider in your applications if you want to add such advanced techniques, or invent your own techniques using OpenGL.

So what is OpenGL? So many of you know OpenGL as a 3D graphics API. The OpenGL specification, which is the definitive document on OpenGL, actually has what I think is a slightly more accurate definition. OpenGL is a software interface to graphics hardware. In other words, it's an interface with a graphics processing unit.

Every device that ships with iOS 4 and Mac OS X has a pretty capable GPU on it. So what does this GPU do? Well, many people believe that the GPU is just there to make your graphics look good. Actually, you can make some pretty high quality renderers using just the CPU.

Movie studios do this all the time. They make very high quality renderers, and you see some great special effects. However, their renderers take many, many minutes to render a single frame. This isn't so good for an interactive application. So what does the GPU do? It accelerates your rendering. And when you're talking about interactive frame rates, this matters. Faster rendering equals better image quality.

Drawing efficiently, allows drawing more: more models, more vertices in those models, more pixels, longer shaders, better special effects in your application. All at an interactive frame rate. All right, so what will you learn today? So let's say you've got a Formula One car and just because you've got it you've driven to work every day doesn't mean that you're going to be winning any Formula One races or even qualifying for them, even though you've got this very advanced, almost 1,000 horsepower machine. You need to know how to use it effectively. Same thing with the GPU in OpenGL. It's a very complex machine. You need to know how to utilize it and use it efficiently in order to harness that power.

So I'll tell you a little bit about how OpenGL works under the hood. Like any good Formula 1 driver, he knows exactly the strengths and weaknesses of his machine, where it excels, where he needs to work at it more. I'll talk a lot about this process called state validation.

This is where OpenGL translates API calls into GPU commands. Now this is actually a CPU intensive operation, and a lot of applications stumble on it. So you need to be aware of what happens there. I'll also give you some fundamental OpenGL techniques to avoid these CPU bottlenecks, and efficiently access the GPU.

Now it's important to note that these techniques apply equally to iOS 4 and Mac OS X. So the codes you write on one platform and the knowledge that you've gained here, you can leverage on the other platform. They've got pretty different architectures, very different GPU's, very different devices. But the OpenGL software stack is quite similar, so you can leverage that knowledge pretty easily.

So let me talk a little bit about OpenGL's design. OpenGL is designed to be a low level API to allow it direct access to the GPU's capabilities. Allow you to get in and get out, not interfere, not have the software stack interfere with your code. Really lean. However, it's high enough level to drive many heterogeneous GPU's. On the Mac we support many GPU's. On the iOS we support many as well.

So there is a fair amount of work to translate from these API calls to GPU commands. Not the lowest level way to actually get to the GPU. I mean we could make an API that's to the metal. However, what this allows, this high level allows for you to do is write code that's portable, and also write code today that will run on devices that have changed drastically underneath your application tomorrow. So as architecture changes, your code remains the same and works quite efficiently.

So OpenGL is a state machine. And this state maps roughly to the GPU pipeline. If you were to look at Apple's implementation of OpenGL, what you'd see is this gigantic C Struct. OK? And what would, be in this struct you could actually look at the OpenGL specification. At the back you'd see this pretty large table that says, OpenGL state. And it has things like, you know, what stuff is enabled. The various GL enables in OpenGL, whether it's on and off. It has things like what's bound at certain points.

And so that's basically what's in this context. All right? So much of OpenGL's live time is just spent tracking state that your app makes. So your application, there's a certain class of calls that OpenGL has that just exists to change this context state, this state becker. [phonetic] So for instance, you call glEnable, some state changes. You call glBindTexter some more state changes. You call glUseProgram, and again, the context is updated with new state.

So at this point we're actually doing nothing that's GPU specific. It's all GPU agnostic. There's no hardware specific command getting generated. The real work happens when you make a draw call. This is when API calls get translated into GPU commands. All of that work is deferred until you draw.

So for instance you call glDrawArrays, and look what happens here is we're munching all of this context state, taking that draw call, and creating a GPU command. So one thing to note about this is that this translation state is a CPU intensive operation. We need to do a lot of processing on the CPU to figure out exactly what your application has done, what it wants, and translate that into a GPU command.

All right, so now we've got this GPU command, what happens to it? Well, GPU commands are inserted to a command buffer that's allocated by the kernel. So we've put that in there. And as the command buffer fills up, or your application calls glFlush, we flush it to the GPU. The GPU now can process it. But actually there's one step that needs to happen first.

We need to actually transfer all the resources that are needed for those commands. For instance, we need to put textures that are used for those draw commands. And we need to download those to the GPU. And that can make this process a CPU intensive process. In addition to state validation, this is another potential bottleneck your application could incur.

Now this is why we recommend not calling glFlush for really any reason. There are some very specific reasons where you might want to call glFlush for multicontext, multithreading rendering. But for the majority of applications, you're only talking about a single thread, and you should never need to call glFlush. It's an expensive operation.

Now you've got the command buffer on the GPU, and the GPU can begin processing it. And it starts by fetching vertices, and then it pushes that data down the pipeline. So it's important to note that there are many potential bottlenecks on the GPU. Any one of these stages could be a bottleneck.

However, a very common bottleneck is actually the CPU. All of those stages that I just talked about. So a key point, the GPU is another processor that runs in parallel to the CPU. Like any good multithreaded apps, you don't want one thread blocking and locking up the other thread.

You want both cords being running at the same time, really maximizing the use of the CPU. Well, same thing happens with a GPU. As OpenGL pushes commands into a command buffer, the GPU fetches commands that have already been submitted to it. So, let me show that again. OpenGL is simultaneously adding commands to one command buffer while the GPU is fetching commands from another. So you really don't want to let the GPU wait for the CPU. That's just wasting the GPU's resources.

You could be rendering 1,000 enemies in your game, and maybe you could be rendering 3,000 if you're not, if you are utilizing the GPU fully. If you're stalling it, well, you're not utilizing the GPU as much, and you're not rendering its cool effects or as many effects as you could be.

Also, some applications tend to use the CPU to do some graphics work. Well, as I mentioned before, the CPU is good at a lot of things. And you can do some high quality rendering with it, but it's very slow. And you really want to use the GPU for this work, really offload the work.

There are certain calls that; so this is what happens when you let the GPU wait, it just sits there. There are a certain class of calls where the CPU may need to wait for the GPU to process. Anytime the GPU needs to process something for the CPU and give it to the CPU, the CPU will sit there waiting for it.

These things are like readpixels, queries, and fences. Here's what happens, CPU sort of just sits there waiting until it gets the data it needs from the GPU. Now there are ways you can use readpixels, queries, and fences efficiently where you're not stalling the CPU, and I can go into that a little bit later.

So one thing I really want to reiterate here is that for each draw call, context state is translated into GPU command. This is a very CPU intensive operation. We call this state validation. Here's what happens you called glDrawElements, the CPU processing goes way up and we translate this GPU command here.

If you were to profile your application and what you would see here is that the draw calls, in this case glDrawElements takes a substantially more time than any of the other OpenGL calls. OK, here you see it takes order of magnitude more than any of the other OpenGL calls that you see here.

And what's important to note about this, is that there is some cost in making a draw call. It's actually not the draw call itself that is very expensive, or causing this expense, it's state setting calls that your app has made before the draw call. So you enable something, you bind something, those look cheap on a Shark profile, but that cost does show up.

It doesn't show up in that particular call, but it shows up in the subsequent draw call. So state sending calls look cheap, they're not really as cheap as they might appear. So I'm going to over a couple of techniques, which you can use to reduce your CPU overhead. The first of which is to use OpenGL's various objects.

The second is to manage your rendering state. Sort it, so that you're not making redundant state changes. State changes that incur more CPU costs, unnecessary state changes, and batcher state. Reduce some state by combining objects into one.

So objects. Whenever you use an OpenGL objects, some of that state is cached in that object and pre-validated. It's not validated at draw time; it's validated up front.

So when it's bound it's easily translated into GPU command. So let's take texture objects for example. You call glGenTextures, and you create this texture object. All right, you call tech image and you see this object state here. It's this little state vectors that the object itself has. And that gets updated and validated. You call tech parameter and that state gets validated and updated again.

And finally, when you call bind texture, that state is merged with the rest of the context state, and cached much more easy to validate, much more, much less CPU processing required to create that GPU command. Now it's important to note that this process of setting up objects is actually somewhat expensive. So you really want to do this upfront, when you're app, your level, your document is loading when the user isn't expecting some high frame rate. Don't wait until your middle of your, you're in the middle of your real time run loop. Do it up front.

So now I've described one way in which you can reduce CPU validation. I'll also describe a few other ways. And I'll also describe some ways in which not using objects can be an inefficient use of the API. The first thing I'll describe is how fixed function, vertex and fragment pipe, how that fixed function, vertex and fragment pipe, can be an inefficient use of the API. And instead, how using GLSL programmable objects can be a much more efficient use.

So modern hardware doesn't have any fixed function vertex or fragment pipeline. OpenGL needs to convert all of your fixed function calls, glEnableLight, etc., into a shader internally. And this can be pretty costly. Here's what happens. You call glEnableLight, some context state changes, as it usually does. GL text in another fixed function call. And that state gets updated.

glDrawArrays. Here's what happened, we generate this large shader under the hood for your application. Now you don't have to pay attention to exactly what's going on. We're just emulating what you've told us, what the state that you'd like to use. And this can be a pretty expensive operation. We actually cache the shader away, so we don't generate shader and complete it every single time that we use fix function.

But we cached it away in the cache table. And when you use this, you use fixed function, we need to generate this cache key, which is somewhat expensive. And then go inside of our cache and pull out your shader and set it. This can be pretty expensive. So instead what we'd rather have you do, and which could be a much more efficient use of the API, is to use program object, shader objects.

This is a most efficient way to set up the pipeline. So you specify shader code, you compile and link it into a program object while you're document your level, your document, your app is loading. Before you venture your real time run loop. And then when you want to use that pipeline you set it, and bam, easily set on the GPU. No looking into some weird hash table to pull out the fix function shader.

Very efficient use of the API. So some people would like a little bit more understanding of how vertex and fragment shaders work. They are kind of a complex piece. So I'm just going to go through and describe a little bit about how they work. So the vertex shader is the first part of the programmable pipeline.

A shader is executed every time for every vertex. So let' say you've got a model with 1,000 vertices. And you draw that model. Your vertex shader will run 1,000 times. OK, now this is great, this is fine, because the GPU's are very efficient at that. It's designed for this.

It's a high bandwidth processor specifically designed to do this sort of professing. OK, so let's talk about the inputs. Inputs are per-vertex attributes that are specified outside the shader. Things like pre-transformed positions, pre-transformed normals, untransform texture coordinate. Specified outside the shader. Now there are two classes of inputs.

The first is a position in clip space. OK? This is the glPosition built-invariable. You've assigned this, glpresent variable. And this clip space is used for OpenGL to map, map to the screen. OK? It's to map your 3D model onto a 2D screen. So that's one output of the vertex shader. The second are one or many bearings. And so the type of data that these bearings usually consist of are colors, normal, and texture coordinates. Values that you'd like to read in your fragment shader. Now these values are interpolated across any rasterized triangle.

So, let's say this triangle was generated by three vertices. And on one vertex you signed 0.4 to one of these varying. One the other you would sign 0.8. You output that in your vertex here. OK, so halfway through the pixel generated on this polygon will have a value of 0.6.

One-quarter of the way down it'll have 0.5. Three-quarters, 0.7. So in other words, it's distributed across the polygon from the two varyings that you output in your vertex shader. Now it's not it's not linearly output like it looks here where it's evenly distributed. In fact, if the polygon is facing you it will be.

But if it's a little bit on edge you'll have this sort of perspective correct interpolation. Where values that are closer to you will be nearer, or actually be a little further apart. That's to give it this 3D effect that you need. OK, so let me give you an example of a pretty simple vertex shader.

So here is a varying that we define, varTexCoord And this will, this is, this works just like the variances I just described that are distributed across the polygon. And here is the main body where all this work is going on. On the first line what we're doing is we're multiplying the incoming glVertex, the pre-transformed position, and multiply it by the modelViewProjectionMatrix.

And then we output it to glPosition. So now this position is in clip space, because we transformed it. And then the second step here is we take in a texture coordinate that is specified outside the shader. We take the two components, S and T components, and we assign it to this varying. And this varying, again, it works just like that diagram I just showed you. And you can read varTexCoord in the fragment shader. The second stage of the programmable pipeline. Now there are a couple of things to note about this shader.

We're using these built in variables here. Now these are sort of a throw back to the fixed function days. And they're only available on the Mac. They're not available if you use ES 2.0 on iOS 4. They're based on, sort of, the legacy pipeline. And actually we would prefer that you do not use them for a couple of reasons.

There needs to be some mapping performed in order to use them. And they're not forward compatible. OpenGL ES doesn't have them as I decided. And code without them isn't the future of OpenGL. In fact, there are a number of extensions that would ship on Mac OS X where you can't use these variables.

There a whole set of these variables. In fact, on the right hand side are varyings and attributes that are, you know, these legacy fixed function based variables. And on the left here, so excuse me on the right those are the varying attributes. And on the left here we've got uniforms.

ModelViewProjectionMatrix, lighting, points, stuff that just doesn't exist anymore in the programmable pipeline. But OpenGL has them for a legacy reason. So instead of using these variables, you can use generics. Here's an example of a shader, which does the exact same thing as the previous shader. Except instead of using built in variables we use our own. We define our own, so that we have portable code and OpenGL can map them to the programmable pipeline much more easily. So we define an input position and an input texture coordinate. Well don't use the built in glVertex or glMultitexture cord. We use our own.

And we define our own model view projection matrix. We don't use glView projection matrix. And we sat that outside the shader. And just as in our last shader, we have bar, TexCoord another varying that we output to. And in the main body we multiply our model view projection matrix by our input position that we've defined.

And we output it to glPosition. Now glPosition is a built in variable, but it's not one of these legacy throwback variables. You still need to use it. And then again we take our input text TexCoord variable and pass it through to varTexCoord. So that it may now be read in the fragmentator.

The fragmentator. So this runs once per pixel produced by each polygon. So let's say you've got a model with 1,000 vertices, and it draws 100,000 pixels. This shader will be run 100,000 times. Again, GPU's efficient at processing this, so that's great. But in general, if you have some processing that could be done higher up in a pipeline in the vertex shader, instead of the fragment shader. You probably want to do it up there, because that will be run less times. A little bit less processing, a little bit less work for the GPU.

The cool thing about this programmable pipeline is that you can render effects that aren't possible using the fixed function pipeline. So here we've got this tune shading effect. OK? And here's how that works. Per fragment we calculate this edginess factor. So if the pixel on the polygon, if the polygon on which it lies is more on edge to the user, we'll give it a value that's lower to zero. That's closer to zero. OK? However, if the polygon is facing the camera, the user will give it a value that's closer to 1. So we now know whether this polygon is on edge or facing.

And then we could use this edginess factor to give it a color. So, for instance, if it's on edge, it's probably we want to give it the sort of tune effect, you know, as the silhouette of the teapot hat, so we'll give it a black value. And otherwise we'll give it a blue value.

And then we assign this color that we've determined the glFragColor, another built in variable, which is the ultimate color that will be rendered for that pixel. Let's talk about the inputs to the vertex shader. There's this call, glVertexAttribPointer, which points to data that can be fed to the vertex shader. Input position, for instance, or texture coordinates, things like that.

Here's how it works. You allocate some memory in your application; in this case we're using malloc. And then you load data into this buffer that you've allocated. You call glEnableVertexAttrib to let OpenGL know that hey, I'm going to use a vertex array. And you give it a position at index, or some index 0 to 16 that maps to the shader. And then you call glVertexAttribPointer and you give it this position data that you've allocated. OK? Position data is basically tells OpenGL hey, my vertex data lives here.

So there are some issues with using OpenGL this way. Because you're allocating the buffer on your end, you're not telling OpenGL to allocate it itself. OpenGL has to copy this data into its own stream. So CPU's cycles will be required to copy that vertex data. Here's what happens. You call glVertexAttribPointer, OpenGL now knows about this buffer that you've allocated, but then when you call glDrawArrays or draw with this buffer, OpenGL copies it into the command buffer. And there's a double whammy here. Because the CPU's also needing to copy, there's some CPU cycles being incurred.

But also we're filling up the command buffer much more quickly. And then a flush will occur. Flushes will happen much more often, a second whammy. So instead of doing this, we'd like you to be able to just cache that vertex data on the GPU. And here's how you do it. You can store it in a VBL. You call glBufferData and BufferData allocates some space on the GPU, and then loads your vertex data into the GPU.

Then you'll call glDrawArrays, a command is created and it simply references this data that's already on the GPU. You would call BufferData probably when your application loads before you're in the real time run loop. There is some cost to it. But if you do that it'll be cached ready to be used in a real time run loop. So here's how it works. You call glGenBuffers, that creates this vertex buffer object. You can call bindBuffer to bind it to the context. Tell OpenGL, hey I'm going to work with this object now. And you call glBufferData to allocate and load your data.

Then you call glVertexAttribPointer the same call that you made before with client side vertex arrays. But this time instead of giving it a pointer, you give it an offset into the vertex buffer object, where your vertex data lives. So in this case we give it 0, which says, hey my vertex data is at the beginning of this buffer.

You can actually store many attributes within a single vertex buffer object, so it doesn't have to be 0. Let's say you've got color data 50 bytes down, then you give it a value of 50 for the color attributes. You may want to modify your data for animations or some other reason.

If you have a constant number of animations, if you have few of them, few enough of them, just create multiple VBOs. Let's say you've got 10 frames, 10 models that you want to animate for your character. Just make 10 VBOs. That way all 10 of them are cached on the GPU. However, you may generate data on the fly.

Maybe you'll load it from disk. Someway that OpenGL may not know about it. In this case you can call glBufferSubData Or MapBuffer to modify this cached vertex buffer object. Here's how this works. You call glBufferData as you normally would, but instead you give it this glDynamicDrawHint. And that says to OpenGL, hey, I'm going to modify this buffer, so put in someplace that's easily accessible by the GPU, but can be easily modified by the CPU.

Then you modify this data that you want to update in the vertex buffer object. And you call BufferSubData with this update data pointer, and it's loaded into the vertex buffer object. There are some caveats. There are some potential problems where if you're updating buffer a lot, buffers a lot, you can have some, encounter some problems.

You can force the GPU to sync with the CPU. So all of a sudden you're running full out in parallel, and then you call BufferSubData and then lock, one depends on the other. And you don't want that. So instead what you can do, well so, basically what will happen is this CPU will wait for the GPU to finish drawing the buffer before it updates the buffer. OK? Both the CPU and GPU can't read and write from the buffer at the same time.

So this can happen whenever you use glSubData or glBufferSubData or MapBuffer. You can use a double buffering technique to avoid this problem. And let me explain how that works. So, you have two vertex buffer objects. On an odd frame you'll load, you'll bind and load an odd buffer. OK? And you draw with it just as you normally would.

But on an even frame you bind and load this other buffer. This way the CPU is loading this even buffer, while the GPU is reading from an odd buffer, a different object. They don't need to synchronize, because they're not accessing the same object, the same data. OK? And then you draw with an even buffer as you normally would.

And so what you would do is you'd ping-pong between these two buffers updating one, while the other is being drawn. Vertex array layout. So vertex array layout. So VertexAttribPointers are really important to GL call. Because not only does it tell you, tell OpenGL where your vertices live, it tells it the vertex layout.

So you call glVertexAttribPointer and you give it some data, like what kind of data is it? It's a floating point in this case. The size of the data. It's probably, it's a position, so maybe it has an X, Y, and Z. So it needs a value of 3. Describe the number of bytes from the one vertex, one attribute to the next. So in this case there's 16 bytes between to the next attribute 0, OK, in the array. And offset, where it lives within the vertex buffer object.

Call glVertexAttribPointer again and some more data is updated for a different attribute. OK? So wouldn't be nice to cache this in the GPU so that you're not always having to call the glVertexAttribPointer. Well now you can, because now you have a vertex array object. And the way this works is you call glGenVertexArrays. And this creates this vertex array object. And any subsequent vertexAttribPointer call actually changes the data within this VAO.

So it's all cached right there. Let me show you some code. You call glGenVertexArrays, create the vertex array object. You bind it to the context, tell OpenGL, hey; I'm going to work with the vertex array object. You call glEnableVertexAttrib just as you normally would, and glVertexAttribPointer. But instead of this getting set in the context, it's set in the VAO.

You can set multiple vertex attributes and it'll all cache within the VAO. And then you can call glBindVertexArrays when you want to use this VAO to draw with. You don't have to call VertexAttribPointer many, many times to set it up, to set up your model data to be drawn. You just call BindVertexArray once and it's already cached ready to go to be drawn.

Framebuffer objects. These are pretty cool objects. So with EAGL and OpenGL ES you must always use an FBO in some form. The EAGL IT guide requires that you use and FBO to allocate your backing store your store from, to which you will render to. However, you can do some pretty cool effects by attaching a texture to a frame buffer object. So, here we've got this little demon character.

And what I've done here is we've rendered this demon character to a texture. All right? And then we bind that texture and textured this plane that you see here on the bottom. And this plane is the image that we've rendered to that we're now mapping to this polygon. So you can do all sorts of reflections, refractions, shadows.

Some pretty neat effects with a renderable texture and framebuffer object. The way this works you call glGenTextures, create your texture as you normally would, bind it to the context, and then you allocate some data by, with glTexImage2D. In this case we're making a 512 x 512 texture. OK? And then we can create a frame buffer object, called glGenFramebuffers. And bind that to the context.

Later when the texture is no longer bound to the context, we can attach it to this framebuffer object, which basically says, any drawing that you do in OpenGL, draw to this texture. Don't draw it to the screen. OK? And then later on you can bind that texture to the context, and you can read it and map your rendered image to some other object.

Some notes about objects mutability. OpenGL objects can be changed at any time during their lifetime. So in this case we've created a texture, and we've given it GL Linear Filtering. And then later on maybe in a real time run loop when the user is expecting an interactive frame rate, we can do something like glLinearMipmapLinear and change that texture object. Change the way that it works.

I would really avoid doing this, because this forces OpenGL to revalidate the object the next time it's used. If you need an object with 2 different states, just create 2 different objects. Set them up at fronts, don't change them, because then OpenGL needs to do some revalidation, and that's going to cause a stutter. Or it's potentially going to cause a stutter in your application.

So OpenGL objects aren't actually pre-validated when they're created. Instead they're validated when they're first used to draw. Now there's some reasons for this. Shader objects can't be compiled until their first use of draw; because the compiler needs to know some context state that that shader's going to be used with. For instance, they may need to know that FBO, VAO, or textures that are bound, the blend states that is used in conjunction with that shader in order for it to do a good job compiling and optimizing your shader.

Texture objects and vertex buffer objects, memory resources don't get cached in VRAM until they're first used to draw. Now this lazy validation step that I just spoke about, can cause some hiccups in your run loop. And there are some methods to get to avoid this. So you can avoid this validation, this hiccup by pre-warming your objects. And the way this works is, you bind the object and draw with it to an offscreen buffer. Maybe the back buffer and you don't present that to the user. And you use the state and other objects it's used with. So let's say you got a shader.

Make sure you use it, you turn on all the blending state you bind all the textures that that shader is used with, and draw with it first before you're in your real time run loop. Don't wait until you're in your run loop and cause a stutter. Hey, here's a little bit of pseudo code. Oh, so one thing I should note is really only consider this if your application experiences a hiccup. If it experiences a hiccup, particularly when you've bound an object that you've never used to draw with previously.

So here's some pseudo code for how this works. For every program in your scene bind it to the context. For every VAO that's used with that program bind that. And for every texture used with that VAO program, bind that. And then for every blend state use with all those other objects, etc., etc. You set that stake. And then you draw.

Draw to the offscreen buffer. This isn't to present to the user, it's just to warm up OpenGL. Note about object size. Know how much memory your objects take. All current graphics resources need to fit in memory somehow. And memory is limited. On iOS 4 there's no virtual memory system, so this constrains. On Mac OS X. Some of the devices have limited VRAM. So, you know, if you're using too much memory there's some cost to it. There's some paging that might need to happen, and you don't want to have that in a real time run loop.

Try to use compressed textures. Textures take up a lot of memory. There are a number of compressed texture formats that you can use. Also, don't use the data types like a 32-bit float for your textures, when you could be using an unsigned, 8-bit unsigned byte. An 8-bit unsigned byte.

Use what you need. Use the smallest state as possible that you need for your scene. We see a lot of time, some applications allocate this humongous texture. A 2,000 by 2,000 pixel texture for a little tiny model that's going to fill up maybe only 200 pixels on the screen. You're never going to see most of that texture. It's just a waste of resources. So fit the texture to the size of the model that's rendered.

Use a 256 x 256 texture in that case. And really to ensure a smooth frame rate, try to fit, the entire frames resources into VRAM so that we never need to page out a texture from VRAM in the middle of a frame. And if possible, fits entire level or scene's textures into VRAM. That way we will never need to page in the middle of your scene.

So, OpenGL's objects are a very efficient way to use the API. However, there is still some costs to binding an object. You really need to determine this costs through profiling, however. Batch or draw calls to reduce this binding of more expensive objects. OK? Let's say you've got, you've determined that some texture takes a really long time to bind, or takes a fair number of CPU cycles to bind. In two objects that use this texture. Don't bind it once then draw, bind it again, and then draw. We said bind it once and draw, and draw the second one.

Visibility. OpenGL processes every thing sent to it in some form. Even if it's ultimately not visible. You know, we're not going to draw something that's, you know, behind the camera. But there is some processing that needs to occur. The CPU needs to process the draw call. The vertex shader needs to run to determine whether or not the camera can actually see this object. So try not to send them to OpenGL. Imagine this is the scene of your application. Here's the view point, the user, the camera. Here's its field of view.

And here is the frustum OK, this includes the right and left clip planes, the top and bottom clip planes, and the Z near and far clip planes. So now it will iterate through your list of objects and determine whether or not they're visible. Anything within this area we'll send to OpenGL. Anything outside of this frustum we'll just discard, we won't send it to OpenGL.

And draw this robot; he's clearly behind the camera. We won't draw him. We draw this hero character; he's off to the right. We won't draw him. We draw this demon, hey, he is definitely visible, so we'll mark him as object 0. We draw this other robot, hey, you know, he's visible. So we'll draw him.

We check this other demon character; he's not visible way off to the left. We won't send him to OpenGL. This hero character, he's definitely visible. He' also in the frustum. So we'll send him to OpenGL and then this demon character also visible. We'll send him to OpenGL. OK? So now we have the set of objects that we want to send to OpenGL. You can put this inside of a visibility list.

Ok? Now one thing to note about this is you don't want to draw the objects in the same order they were determined visible. Ok? You want to sort it by render state. What would happen here you'd bind the demon's texture? Draw him, bind the robot's texture, draw him.

Bind the hero's texture, draw him. And then we'd rebind the demon sections that we bound originally and draw the demon. OK? Instead what you want to do is sort them, so that now the demons are together, one bind two draw. All right, so here's an algorithm that you can use to sort your rendering state.

It's called a state tree. So let's say you've got these four characters. You've got these two guys and they're clearly using the same texture. And you've got these two guys, and you've got this human character and he's got some metal armor on him and you've got this robot clearly all metal. So you want to have the shader that does some little shininess effect. They used the same shader.

You stick them inside of a tree and you traverse it in order. So starting from the top, you bind the first shader. You bind the texture that's used by these two guys. You bind the VAO, and you render the demon. You go up and you've already bound the texture, don't do it again. You bind the new VAO and you render this guy.

Go up to the top and now we bind this new GLSL program. This shininess program. And we bind the texture, bind the VAO, render this robot, go on up. You've already bound the GLSL program, so now we just bind the texture for this human guy, find his VAO and render him. Now it looks kind of like a binary tree here, but it would actually be an enary tree.

This is a very simplistic scene. You probably would have, you know, many, many shaders, which would make this tree much wider. Also, you might want to account for different rendering state. Like you might want to account for depth, blend, clip state, etc., which would make this a deeper tree.

OK? In this case I determined that the GLSL program objects I said, well those are pretty expensive to bind. So I'm going to put them at the top of the tree. We're going to set those first, so that we don't have to rebind them very often. OK? So the more expensive objects at the top of your tree, less expensive objects like vertex arrays towards the bottom.

One way in which you can reduce CPU overhead, the CPU overhead of draw calls, is to combine the draw calls. Basically make less draw calls. And there are two methods that I'll talk about. One is texture atlasing. And this is basically combining multiple textures into a single texture. And the second one is instancing. Instancing requires some special hardware that's only available on Mac OS X. My colleague Matt Collins will be talking about that a little bit more tomorrow and how to use that. I'm actually going to talk about texture atlasing.

Here you have these four characters, and you've got these four textures to map to these characters. In order to draw this in a naive way would be to bind this texture, draw to it, bind the second texture, draw to it, find the third texture, draw, find the fourth texture, draw. OK, you're binding four times for four textures.

Instead you can bind it into one uber texture. A texture atlas. Then you bind that one texture, draw, draw, draw, draw. One bind, four draws. Much more efficient use of OpenGL. Here's an example of a texture atlas used in the Quest demo. Here we've got a lot of different elements in a single texture. We've got some flags on the upper right. We've got a stone wall in the upper left. We've got stairs, we've got doors, we've got statue, all in one texture, one bind, and they can draw a ton of different things of their dungeon.

Multithreading in OpenGL. So because there is a fair amount of CPU overhead that OpenGL incurs, there are reasons to multithread it, so that you can amortize the CPU costs across multiple cores. This makes sense on iOS 4 devices as well, even though they only have a single core. CPU intensive calls can block.

And that means that while they're blocking, while you're in your main loop doing some OpenGL processing, you can't handle UI events, you can't handle audio, you can't do your app logic. Additionally, if you're doing a lot of stuff on the main thread, there is this is watchdog process that looks at your app and determines whether you're in some infinite loop, whether you're behaving badly, and it may kill your app.

So it may mistake you for doing something like in an infinite loop or a block and some sort of deadlock and just kill your application. So you really want, you could put some of this processing on anther thread and the watchdog won't do this to you. So here is the simplest multithreading technique I'll describe.

And basically you load a second, maybe third thread, both with, or one or two OpenGL contexts. And you can use these threads to load data. Load textures. Load vertex data. Compile shaders. A lot of CPU heavy lifting. Important things to know is once you're done with all that loading you want to kill your other threads.

You only want to have one thread running with an OpenGL context. OK? Because the other thread, two OpenGL contexts is running at the same time. There is some CPU over head that might be incurred, because there's some locking. And so two threads can block one another; two OpenGL threads can block one other. So just have one OpenGL thread running at a time.

Another more advanced technique is to use a producer consumer paradigm. So in this case the main thread produces data, it can produce, you know, the animation frame, which your characters are in. To position on the screen or position in the world, you can compute the visibility of your objects. Use that frustum culling technique.

On this thread you won't have an OpenGL context, you're just producing data to render by a second thread. When the producer thread is done you send all the render threads to begin. And then the render threads can take all that data, consume it, and render with OpenGL. It has the only OpenGL context on it, that render thread. You don't have two OpenGL contexts.

The main thread can then process audio input and other app logic in parallel while the render thread is actually drawing stuff. And let's say you still have some CPU overhead and it's not well balanced. You can move some stuff, like maybe the visibility test to the render thread.

Give it a more even distribution. So I've talked a lot about using the CPU, or the CPU and the GPU together. You need to also consider using the GPU with the display, the other device. That is important to consider when coding your OpenGL. One thing to note is you can only render as fast as the display can refresh.

It doesn't make sense for you to render at 200 frames a second if the user can only see 60 frames a second. OK? It just wastes battery power. You're doing a lot of processing that the user will never have any context into. Will never see the result of.

On iOS 4 you can use a CADisplayLink API. And we see a lot of apps using this NSTimer API to initiate their per-frame rendering. Instead, we use this DisplayLink API, because the NSTime is arbitrary when it's fired with respect to the display. What will happen is NSTimer might fire right before the refresh, and so there's going to be some latency between the time you draw and then you can see it.

Or it might be fired right after display has refreshed. So it's not going to be consistent with respect to display. And in some very pathological cases it can reduce your frame rate. On Mac OS X you can use the analogous API CV display link. Now we see a lot of games trying to control their main loop.

This is a kind of an outdated way of doing it. Because, again, you don't need to display, or don't need to render any faster than the user can see. Looping more than needed wastes power, particularly on these MacBooks where people want a fairly long battery life. You could implement some benchmark mode for advanced users or developers that runs on the main loop.

But under shipping game in a normal loops, or for a normal running you may just want to use CVDisplayLink to initiate your rendering. A note about coding for both platforms. So OpenGL ES is a subset of OpenGL on the desktop. So if you code using OpenGL ES 2.0, you can port your code that you've invested a lot of time on iOS 4 onto the Mac.

And vice versa. OK? So if you've got this Mac application, and if you stick to all the calls that OpenGL ES 2.0 provides, you can pretty easily port it to the iPhone OS 4. There are some things to be aware of. Clearly there are more memory and performance constraints on the embedded iOS 4 devices. So you need to consider that. There are different compressed formats. You will need to translate for that. And there are for some kind of silly reason, slightly different function names. Now the parameters of these functions are exactly the same.

On OpenGL ES we have this BindVertexArray OES function and it's exact same function as BindVertexArrayAPPLE, it just has a slightly different name. So you just need to rename your functions. The sample code that I provided for this session that you can find for the session's site, compiles for both and you can kind of use it as a template or maybe a starter for creating your application that you might want to port and ship on both platforms. It's this kind of cool little reflection demo. And it works pretty well on most platforms.

So in summary, there is a fair amount of CPU overhead that OpenGL needs to incur, and you can minimize this to efficiently access the GPU. Validation is where a lot of this CPU overhead occurs. You can use OpenGL objects to cache this validation and minimize state changes and draw calls to reduce the validation. For more information you can contact our Graphics and Game Technologies Evangelist, Allan Schaffer.

As well there's a ton of documentation on OpenGL at the OpenGL Dev Center. And there are a lot of engineers from Apple at these Apple Developer Forums, so you can get help and ask questions throughout the year at this devforums.apple.com. Also, I've posted a bunch of code snippets and the sample code at this link down here. So you can check that out.

There's a Q&A on the different variables that you shouldn't use in your GLSL program and some other information. This is just the first of six OpenGL sessions at this year's WWDC. And there's some great information for both Mac and iPhone developers, or iOS 4 developers. The first is OpenGL ES an overview on, for the iPhone.

And that's mainly geared for the iOS 4 developers. And the second is OpenGL ES Shading and Advanced Rendering. And this is a pretty cool one. My colleagues have come up with some great demos. And even if you're a Mac developer, a lot of these demos are easily portable to the Mac.

So they're going to be talking about some shadows, some reflections, some really cool techniques that you should probably check out even if you're a Mac developer. There's also an OpenGL ES Tuning and Optimization session. And this is going to be great for developers coding for iOS, but also some of the techniques that they mention you'll be able to use and leverage on the Mac. They're going to be describing a tool that's available on iOS 4 the OpenGL Analyzer.

And even though it works only on iOS 4, if you're a Mac developer you can use some of the same techniques to profile your applications. They're also going to be talking about some of the GPU bottlenecks. Now I talked a lot about CPU bottlenecks, but there is a whole class of bottlenecks on the graphics processing unit that you should be aware of and potentially could run into after you've optimized the CPU portion of your app.

And then tomorrow morning at 9 o'clock there's OpenGL for Mac OS X and my college Matt Collins will be talking about a number of the newly available features on Mac OS X instancing, texture arrays. He's also got some really cool demos that you might want to take a look at. And then finally there's this Taking Advantage of Multiple GPUs session, and they're going to be talking about OpenCL the other GPU API.

And it's used in conjunction with OpenGL and leveraging multiple GPUs on the Mac Pro to do some really cool processing with OpenCL and really great rendering with OpenGL on two different devices. So thanks a lot for coming. I really appreciate it. And I'm hoping you guys will be able to take these techniques and really efficiently use OpenGL.