Welcome, everyone. Hope you can hear me OK. Great. Well, I just want to say we are really, really thrilled to be doing today's presentation on 3D graphics for iPhone OS using OpenGL ES. And as we look out at the crowd here, it's really great to see all of you here.

As Steve and Bertrand and Scott have mentioned, we do believe this is the birth of a real new platform, brand new platform. And you guys are all the key to making that platform a reality. So we are looking at teaching you today about OpenGL ES. And we're going to talk about OpenGL ES on the iPhone. And to do that, we're going to take a brief look at our agenda.

So first off, first thing we're going to do in today's talk is talk about a brief overview of OpenGL ES, what it is, and how you'll use it, and how it compares to desktop OpenGL, which you've probably used before, or how it compares to some other 3D programming APIs, which you might be familiar with.

The next thing that we're going to do is talk about OpenGL ES on the iPhone. Now, in developing OpenGL ES for the iPhone OS, We implemented some new APIs, we created some key technologies, and we developed some key concepts that we want you to become familiar with as you develop your applications.

And the reason that we want you to become familiar with these concepts in particular is that some of them are unique to the iPhone OS, and some of them are simply unique to embedded development, developing for small consumer electronics products like the iPhone, and other products in this similar class. And this may be an area where you may not be familiar with if you've done desktop programming. Some of those key technologies are the areas of integration with the rest of the iPhone OS, and some of them are some new APIs. We'll talk about those today.

Next, we'll talk about the iPhone simulator and OpenGL ES support there, and how you can use it, how you can use it to develop your application, what it's good for, and what you need to use the device for instead. And last, we'll talk about our recommendations for best practices, some caveats with the device, and just how to use the device to be the most efficient and to develop the most highest performing and the best looking applications that you can.

So let's move to the next section of our talk today and begin by thinking about some of our assumptions about your background. So first of all, this is not going to be a talk to teach basic 3D graphics. We're going to assume you have some background in this area.

And judging by the size of the crowd here today, I can assume that that's probably true. There's a lot of folks here, and it looks like you're ready to do some very cool 3D graphics programming. So we'll also assume that you know OpenGL 1.5 and its extension mechanism.

There's a few other key technologies we'll make the assumption that you know at least a little bit about. The first of those is Cocoa and Objective-C, or Cocoa Touch today, as we've talked about in the iPhone, and Core Animation and UI Kit. Now, on these last two subjects, you might not be experts yet.

There's a ton of great information here at WWDC in the iPhone SDK on Apple's developer website for you to familiarize yourself with these concepts. And we won't be looking at them too deeply, but we do assume a little bit of familiarity to understand some of the code snippets we'll show. All right. With that, let's begin.

So as we mentioned, the first thing we'd like to do is start with an overview of OpenGL ES. And the very first thing you need to know about OpenGL ES is that it is a subset of OpenGL. So if you've done OpenGL programming on the Mac, on Windows, on other platforms, you'll feel right at home with OpenGL ES. The basic concepts are the same. The basic methodology programming the API is extremely similar. And you can use the same basic programming techniques for your 3D graphics.

OpenGL ES is an industry standard. It's an open standard, which means it has multi-platform support across a wide variety of hardware vendors. This list of hardware vendors includes many hardware vendors that maybe you haven't even seen before if you haven't done embedded device programming. But it's actually incredibly well supported, and there are just literally millions, if not billions, of devices using this API.

The reason that it is a subset of Open GL 1.5 is part of -- part of the reason is because it is focused on targeting an embedded class device. And what does that mean? Well, these are small consumer electronics devices. They typically run off a battery. The iPhone is a perfect example.

But others are set-top boxes, GPS units, other types of small devices. And the API is specifically targeted to try and drive this class of device. And as we talk about what Open GL ES is a little later in the presentation, you can see how certain design choices were made to favor these devices. Perhaps most importantly, though, OpenGL ES is a full-featured hardware acceleration API. So it fully supports 3D graphics programming with fixed function vertex and pixel processing.

So as we move on, let's take a look first at some of the API design goals of why OpenGL ES was created the way that it was. As I mentioned, it was specifically designed for embedded class devices. Now, really what this means in terms of the API design is that the API had to be designed for the implementations themselves to be smaller. Smaller in lots of ways. Smaller in the amount of power that they use, because these devices are typically on a battery or some other limited power budget.

Smaller in the amount of memory that they consume. Smaller in the amount of processor cycles that they consume. The typical processors and memory systems for these devices, while incredibly powerful, might be considerably smaller than what you're used to if you've done any kind of desktop or workstation class programming.

A second key point about the API design goals for OpenGL ES is that there was an intention in the design to take advantage of the fact that the API was new. And it was targeting a new class of devices, trying to bring this functionality to them. As a result, there wasn't a huge amount of legacy support that people had to worry about. And while they wanted to maintain a certain consistency with the API of OpenGL, there was not a need to retain exact binary or API compatibility.

And this gave us, the Kronos group, who developed OpenGL ES API, an opportunity to actually remove some obsolete functions, to clean up the API, and to think very carefully about what to include and what not to include in order to fit in some of these constraints I've just outlined.

And as a consequence, or perhaps another motivation for this type of design, So, the API was consciously designed with the idea that there does not have to be a software fallback or a means for the CPU to execute things that the GPU cannot. As a result, what's actually in the API is very, very likely to be hardware accelerated. And this is actually a benefit to you, because you can actually use it to run things that the GPU cannot. And this is actually a benefit to you, because you can actually use it to run things that the GPU cannot.

All right, so let's take a look next at the traditional 3D graphics pipeline. If you've done any OpenGL programming before, this will look very familiar to you. And this is the exact same 3D pipeline used by OpenGL ES. So you start with your traditional stages of geometry processing, rasterization, texturing, fragment processing, and frame buffer operations. And you'll see that as we walk through the features of OpenGL ES, that these same stages, the same pipeline applies.

So what I'm going to do next is kind of walk through each of these stages and show specific features in OpenGL ES, and particularly how they compare to desktop OpenGL. And this should give you a good feel for what's in the API and how you can use it.

So let's take a look first at the geometry processing stage. Now, the first thing I want to say on this slide in particular is that you can see from the sort of inclusion features here of what's included and what's been removed relative to desktop OpenGL that this is a full-featured API.

It fully supports geometry processing, vertex array-based rendering, buffer objects, vertex lighting, shading, et cetera. But there's a few key features that have been removed. And again, the reason they've been removed is because some of these features are inefficient, perhaps inefficient even on desktop systems. Some of them have become redundant, and then a few of them are not likely to be accelerated on some of these embedded class devices. Thank you.

The one you're going to notice here, probably if you've done any OpenGL programming before, that's probably the most key, is the immediate mode entry points. Specifying vertices one at a time is likely to be inefficient, again, even on desktop systems, and was removed from the API. Let's take a look next at the rasterization phase of the pipeline.

And here, once again, there's very full features. So you've seen from some of the demos that we've done so far that you can create really terrific looking 3D games and applications with OpenGL ES. In fact, you can see some of those games in Steve's keynote and some of the other examples from the application store.

Now, a few key features here have been removed as well. So if you look on the right side, you'll see quad and polygon primitives have been removed. And once again, the reason they've been removed is because they're largely redundant. You can actually do quads and polygons with triangles and other techniques.

Things like draw pixels and copy pixels have also been removed. But again, you can implement these with what's remaining in the OpenGL ES API. So this is all an attempt to make the API smaller to fit in the types of devices that can fit in your pocket or that are just generally constrained to run the OpenGL ES API.

In the texturing stage, here you have 2D texturing, multi-texturing, lots of different formats, lots of different filters, mipmap generation. So again, it's very full-featured. However, certain features, such as secondary color, and features that might require a considerable amount of memory, such as 3D texturing, or that might require a considerable amount of CPU processing, such as converting from one type of pixel format to another, have been removed.

And last, when we look in the fragment processing stage, when we look at this stage, we actually see that there's not very much that's been removed at all. There's a few blend equations that have been removed, but largely the whole feature set is there from OpenGL. And this is key, because most of your processing in these types of devices is in the pixel space, in the fragment processing space. And the API here is very full-featured.

And finally, if we took a look at frame buffer operations and some of the miscellaneous features of the API, we can see that there's actually only a few things that are probably unlikely to be used in this space that are actually been removed. So double precision is a good example. Because double precision is a feature that, once again, on the desktop is often not even accelerated and has been removed from the API. Or other features that require a certain amount of CPU-based processing, such as pushing and popping your OpenGL state.

I've gone through this in a hurry because we mostly want to point you at the official specification for OpenGL ES to give you all of the details. If you take a look at Kronos' website, as I mentioned earlier, Kronos is the organization that controls the OpenGL ES standard. This is an open standard, just like OpenGL, which they also define. They have a website that defines all of this. It provides a full specification of the behavior, and it also provides a different specification, sort of similar to what I've been doing here today.

Theirs is actually much more formal and much more detailed. So here's an example showing some of the features that are present in OpenGL ES and some of them that are absent. And you can compare this to the full OpenGL specification to get a good feel for what's in the API.

So in the interest of time, I'm going to move on from what's actually in the OpenGL ES core functionality set itself and talk a little bit about OpenGL ES extensions. So OpenGL ES extensions work just like OpenGL extensions. And this is important in a couple different ways. First of all, it's the exact same mechanism that you used in OpenGL. So if you've done OpenGL programming, you know that you have to query for the list of extensions that are supported by your implementation.

And that those extensions can be named to indicate whether they're supported by one vendor, multiple vendors, or some official extension by the Kronos group. Such extensions would have the arb or oes suffix on them. So let's go ahead and look at some of the extensions that we've used.

And this last point up here is key. Implementations will support different sets of extensions, perhaps across different driver revisions. So you need to make sure to always query for the extensions before you use them, and make sure that you are prepared to deal with their presence or absence at any given time if you choose to use extension features. Jeremy Sandmel So we won't talk about all of the extensions that we support in the OpenGL ES implementation on the iPhone, but we'll talk about a couple of them because they highlight some kind of key characteristics of OpenGL ES extensions.

So the first one I want to highlight here is the OES map buffer extension. I call this one out not necessarily because it's an amazing piece of functionality, but it's a good example of a feature which is actually a core part of OpenGL 1.5, meaning it's required. Every implementation supports it.

And it's actually an optional extension in OpenGL ES. The API is essentially the same. It has some suffixes on the function entry points and enumerants, but otherwise it's basically the same. And like all extensions, you need to query before you use it to find out whether it's present or not on your particular implementation. On ours it is, but this is a good example of this type of extension.

Another key extension that is a good example to show about certain properties of extensions in OpenGL ES is the IMG Texture Compression PVRTC extension. Now, this extension I highlight partly because we encourage you very highly to use this extension when doing iPhone OS programming for OpenGL ES, but also because it's a good example of an extension which is particular to a specific GPU vendor or perhaps even a specific GPU.

will talk more about this extension a little later on. The last one I want to highlight here out of this list is the OES Frame Buffer Object Extension. Now, we mention this because, first of all, we make heavy use of this extension on the iPhone OS, and we use it for configuring all of your frame buffer operations. So when you're going to decide what format your frame buffer should be for OpenGL ES or whether you should have a depth buffer or not, you'll be using this extension.

This might be a little different than some other implementations of OpenGL and OpenGL ES, in that some of them have other means of configuring the frame buffer. But here, this is what you'll use. So we'll talk more in depth about it in a few moments. Okay, so that's all we're going to say about extensions. But what we want to do now is take a look at some OpenGL ES code. And first, we'll start off by comparing it to the corresponding code in OpenGL 1.5, just to give you a feel for how they compare with each other.

So the first thing we have here is some OpenGL code, and it's basically setting up some state. It's going to set up some transformation state. And then at the end, at the last line here, it's going to do some drawing. Now, I'd like to take a look at what this code would look like if we converted it to OpenGL ES.

So as you can see, actually hardly anything here has changed. And that's the point, that with the exception of one routine here, which actually has a slightly different suffix because, again, there's no double precision floating point. There's a GL ortho call, which takes single precision arguments. With that small exception, this is exactly the same code.

We think this is really, really powerful because this means that if you're used to doing OpenGL programming, if you have a large body of code, if you have games that could run in OpenGL or applications, converting them to OpenGL ES should be very, very straightforward. Now there's some caveats and some tips and techniques specific to OpenGL ES and even further specific to the iPhone that we'll go over here today.

But this is a big takeaway point, is that you should realize that you're probably very familiar with OpenGL ES before you ever get started. And there's a few other minor code changes of this flavor, such as some extensions have slightly different suffixes, some features have become optional that used to be core, and there's this change from double to single precision, but largely it's very, very, very similar.

All right, so that's OpenGL ES. And now we want to talk about using OpenGL ES on the iPhone. And here, once again, we've got several key technologies and concepts we want you to become familiar with, and I'm going to go over them now. The first thing I'd like to say is that we have a full OpenGL ES 1.1 implementation on the iPhone. That means it's fully hardware accelerated, fixed function, vertex, and fragment processing.

And it's very highly performant. You can create really, really, really great games with the iPhone. And you've seen some of them already in the keynotes that have gone on so far, and in the game session that was earlier today. So you think there's just an amazing amount of potential here.

And I look at some of these demos, and I see the things we've shown in the keynote, some of the things we've shown that are going to be on the application store. And I'm still blown away. I can't believe this is a cell phone. This is a thing you usually are just answering calls with, and it has this really, really fantastic 3D implementation. Thank you.

The other thing we want to say is that it is-- our implementation for OpenGL ES on the iPhone is efficiently integrated with the iPhone user interface. Now, we do this in a couple different ways, but the way you'll probably run into it the most often is that Core Animation, which provides the key to the user interface on the iPhone, Core animation layers can actually serve as OpenGL ES rendering surfaces. This means that you can render directly into a core animation layer, and then use it as you would essentially any other core animation layer.

You can create some amazing user interfaces, and you can create some very efficient use of the 3D pipeline on the iPhone using these techniques. And we'll go over them in a few moments. But because you can put your content in a core animation layer, this means your content becomes a first class citizen. This is not just switch into OpenGL ES mode, do some 3D rendering, and then shut everything down, and then draw your user interface. They can be very carefully integrated, and they can be composited together.

This is key to providing the seamless type of graphics experience on the iPhone. All right, so let's take a look next at a little bit of background about how OpenGL ES typically interacts with its host environment or its host operating system, and then we'll talk about exactly how that works on the iPhone.

So there's three APIs we've listed up here, or really three types of APIs. The first one is OpenGL or OpenGL ES itself. And this provides the rendering API, the thing you're actually going to draw with. But it doesn't define anything about how you start up, how you initialize your context, or how you display anything. And for that, we turn to some other APIs.

We call the first one a platform API, or the OpenGL ES platform API. And it's basically responsible for the device initialization, the initial configuration, creating your rendering context. And there's some examples up here. If you've done programming on the Mac, you're probably familiar with AGL and CGL, or NSOpenGL, or Windows, it's WGL, or other embedded devices, it's EGL. And these are all sort of similar types of platform APIs in this respect.

And then generally, there's a native Windows system API that's actually responsible for displaying your content. On the Mac, that might be Quartz. On Linux or Unix, it might be X Windows. And it's going to be responsible for taking your content and putting it on a screen. So how does that look? Well, normally there's this sort of block diagram that takes your application code, where you'll manage your surfaces through your display, native Windows system API. You'll manage your context. You'll manage your context through this type of platform API. And then you'll do your drawing through OpenGL or OpenGL ES.

Now, if we take a look at this diagram as it pertains to the iPhone, you'll see that basically we have implemented the native windowing system API with core animation. This is the API that's going to be responsible for doing your display, for presenting your interface, and integrating the two. And we've implemented the platform API with a new API introduced in the iPhone OS. Called EAGL or Eagle. And we're going to speak much more in depth about both of those now.

All right, so first off, Eagle. Well, the first thing you're going to see is that Eagle serves the role of our iPhone OS platform API, and is specifically responsible for defining the interface in between OpenGL ES and the iPhone OS. What does this mean? Well, this means this is the API that's going to manage your device initialization, your rendering contexts. And it's also the API that's going to do the communication on your behalf with Core Animation, our native windowing system, to display your content.

And the last point is that, as we mentioned earlier, it's also going to use the OES Frame Buffer Object API for its drawable configuration. And we'll talk more about how that works. So the first part of the Eagle API that you're going to run into is the Eagle Context Object.

The Eagle context is an abstract data type that represents all of your OpenGL ES state, as well as your command streams. So you can have one of these Eagle contexts current at any given time for your thread. And this is because OpenGL ES, just like OpenGL on the Mac, is not a thread-safe API.

Whenever you've made one of these contexts current to your thread, all the OpenGL commands that you issue will then be done with respect to that context. Let's take a look at what that looks like. So in order to set up an Eagle context, you follow only really basically two steps before you can issue OpenGL commands. I've outlined them up here.

So the first one is you're going to allocate and initialize the Eagle context itself. And to do that, you basically follow this sort of standard Objective-C object initialization and allocation methodology. So we allocate an object, and then we're going to initialize it. In this particular case, we initialize it with an argument that defines the rendering API we want to use. OpenGL ES 1.

If we look at the second block of code up here, we'll see the second step we have to take is to set that context to become current. Once that context is current, we are basically ready to issue OpenGL ES commands. And that's our third block of code up here. It issues a single OpenGL ES command to get the renderer string identifying which renderer we're using.

So this is very simple. It's actually just two steps. You can issue OpenGL ES commands. But you're probably asking yourself an important question. If I were to draw right now, where would my rendering go? This is an important point when you're doing 3D graphics programming. So for that, we turn to frame buffer objects.

Now, on OpenGL ES for the iPhone, we use frame buffer objects, or sometimes called FBOs, for all of your rendering. This is probably different from some other environments in which you've used OpenGL ES or OpenGL, because we use it for both on-screen and off-screen rendering, as well as all of your displayable content. And the API is defined by this OES frame buffer object extension that we keep talking about.

So a framebuffer object is simply an OpenGL ES state object that contains references to color, depth, and stencil buffers. And those color, depth, and stencil buffers can themselves be stored in other OpenGL ES objects, such as textures, or a new type of object called a render buffer, which is simply a buffer that stores an image that you can render into.

All right, so let's take a look at what this looks like, sort of the object relationship and how you use this code. So the first step in creating a framebuffer object is to call the gen and bind framebuffer routines. Once again, if you've done any OpenGL programming, this will be very familiar to you. And the API for OES framebuffer object is heavily derived from the ext framebuffer object extension in desktop OpenGL.

So this is very traditional OpenGL programming. You generate a name, you bind it to your context, and we've created a frame buffer object. The second step is to actually allocate the buffers where we're going to store our image data. So once again, we call generate-- gen render buffers, bind render buffers to bind it to our context to create these objects. And then we'll make a third call, render buffer storage. Now render buffer storage's purpose is to actually do the allocation of these buffers for you, to actually allocate the memory that's going to store your pixel values.

Now we've got render buffers allocated, a frame buffer. The last step is actually to attach them to each other. So we call frame buffer render buffer. This is somewhat of a confusing name, but it follows the GL methodology of naming the object that we're attaching to another object. So we're attaching a render buffer to a frame buffer.

Let's look at that code a little more briefly here. And so you can see we have these steps outlined. This is basically what you would need to do to set up a frame buffer object to do OpenGL ES rendering on the iPhone. Now you don't need to write this down, because we actually provide an Xcode template that outlines and handles all of this for you.

But we want you to understand how it's working. So we'll go over it real quickly. You simply generate the names for your frame buffer and render buffer. You bind the frame buffer and render buffer to your context. You allocate some storage for the render buffer, and then you attach the render buffer to the frame buffer.

Okay. So, use Eagle to create your OpenGL ES context, and you use the OBS Frame Buffer Object API to configure your frame buffer. Now you probably have yet another question. How do we actually display this content? Well, for that, we're going to turn to another class, another part of the Eagle API, another part of the Core Animation API, and that is the CA Eagle layer.

Now, the CA Eagle layer is essentially nothing more than a subclass of a Core Animation layer, but one that is specifically designed to display OpenGL ES content. It essentially integrates the OpenGL ES and Core Animation APIs together. It allows you to render into a Core Animation layer. And because you can attach one of these objects to an OpenGL ES render buffer, you can now render directly into the objects that are displayed on screen.

So very briefly, we can see how do we use this. Let me outline the steps real quickly. So the first thing you need to do is subclass a UIView that you're going to use to display your rendered contents. Why do we do this? Well, in Cocoa Touch, every UIView object is backed by a core animation layer.

So we want to back this particular UI view by a specific kind of core animation layer, namely one that you can render into. And the way that you do that is you're going to override a method of the UIView, the layer class method, to return the CA Eagle layer class instead of the CA layer class.

The next thing you'll do, as we outlined, is create and bind an OpenGL ES render buffer and an FBO. You're going to attach your CA Eagle layer to your render buffer.

Let's look at how these objects relate to each other. So you start with the Eagle context itself. And the next thing you're going to do, as we said, is bind a new render buffer.

Once you've done that, you typically would allocate storage, as we were saying before, to do some rendering. And this would generally generate an internal allocation of renderable storage. But we're actually going to take a slightly different tack here, because we don't want to just render to this. We want to display it. So we're going to make another call in the Eagle API.

And this API is the similarly named render buffer storage from drawable. And what this routine does, it actually allocates renderable and displayable storage simultaneously, and then shares that storage by attaching the CA Eagle layer to the render buffer object. Now the CA Eagle layer, as we keep mentioning, is nothing more than a subclass of your core animation layer.

And then the last step is you want to present your render buffer, called Eagle Context Present Render Buffer, to actually display this on screen. So wherever this core animation layer would be displayed on screen, that's where your content will go. Now we have some recommendations a little later in the presentation about the best and most efficient way to do that, but that's basically how these objects relate to each other. So let's take a look next at how you would actually get your rendering into one of these displayable render buffers.

So we have this configuration we've just set up. Next thing you would do is you would allocate your frame buffer object. You would bind that frame buffer object to your context. So now all your rendering will go to your frame buffer object. And the last step is to attach your render buffer, the one you've attached to your CA Eagle layer, to the frame buffer. Now you've created a pathway by which your Eagle Context can issue rendering commands into the CA Eagle layer, and they can be displayed. And if you want, you can also optionally attach a depth buffer.

And the last thing I want to say about Eagle before we move on is that we fully support sharing of objects between OpenGL ES contexts. And there's information about how to do this in the iPhone OS Programmer's Guide. We mention it here for completeness. All right, so we've talked about a bunch of concepts here. Let's just take a quick look at the Xcode template that will tie this all together for you. So I'm going to switch to the demo machine here.

Now, first thing I'm going to do is launch Xcode here. And I'm going to create a new project. We provide right here the OpenGL ES application template for you to use. And it's a great starting point, because it does all of this complicated work for setting up your framebuffer object, initializing your context. It does this all for you. Now, if you simply choose this project, we'll create it right here.

and you'll see that we basically instantiated a bunch of files for you. Now, I haven't added anything to this project. I'm simply going to look quickly at some of the implementation. Now, the first thing you'll notice here is that this template will instantiate in a subclass of UIView, just like we recommended.

in this initialization routine here will allocate an Eagle context. and it will initialize it with our rendering API, and it will set our context to be current.

The next thing that we can notice about this routine, this method, is that it sets up the frame buffer for us. So it does the steps I just outlined, generate your render buffer and frame buffers, call render buffer storage from drawable, passing in this CA Eagle layer we just allocated, and attaches the color render buffer attached to the CA Eagle layer to our frame buffer.

And the last key point about the template is it has basically a placeholder for you to put your OpenGL ES rendering code. And you can see right here that basically it just has the same kind of OpenGL code you're probably familiar with, setting up some state, clearing the screen, and issuing a draw call.

All right, we'll come back to this template in a moment, but I want to talk a little bit next about some of the implementation details of the iPhone OS OpenGL ES so that you can use them when designing your own application. So we can switch back to the presentation machine. There we go.

The first thing we want to say is that the iPhone OS and iPhone is using the Imagination Technologies PowerVR MBX Lite GPU. Now, this is a terrific GPU for embedded devices. It supports hardware-accelerated OpenGL ES 1.1 fully, full vertex and fragment processing acceleration. And it's designed for embedded systems. It's designed for the types of devices just like the iPhone.

It implements a couple of techniques in that design that can use memory and to use the system in the most efficient manner. The first one is called Tile-Based Deferred Mode Rendering, and the second one is Hidden Surface Removal. I'll talk a little bit about each of those because they will affect how you write some of your application code. These are basically guidelines that will affect the best way that you should use to get the most performance and efficiency out of the system, not the correctness of what you render. But they're still important. to keep in mind.

So what is Tile-based Deferred Mode Rendering? Well, a traditional renderer is going to render each polygon in order, in the order that you send it to OpenGL. A Tile-based Deferred Mode Renderer, on the other hand, actually takes a different tack. In order to minimize its access to memory and use memory the most efficient manner, it's going to store entire frames worth of data at a time. It's going to then break that data into tiles. Generally, tiles are organized by screen position. And it will create a list of rendering commands for each of those tiles.

Once it's done that, it will process each tile. And once it has this whole list of the scene data, it can actually figure out exactly which pixels need to be processed for any tile. It only needs to process the visible ones, which saves it a bunch of work. And then it writes out the final pixel values.

Now why does this matter? Well, as we said, it's incredibly efficient. It allows you to get the really great performance that you saw in some of the demos so far. And it just basically can save battery life and processing time for you. But there's a caveat that certain types of operations you might be used to using on other implementations can be more expensive. And basically, those are the set of operations that depend on previous rendering having been completed.

Again, this batches up a whole tile, a whole frame's worth of data into tiles every time before it renders. So anything you do that says, hey, I'm done rendering, can cause us to go through this process, looking through all of the information that is in the scene, and figuring out the best way to render it.

We'll talk a little bit more about that when we get to our best practices section. There's a few other implementation details we wanted to highlight. We support 16 and 32-bit color buffers. We support an optional 24-bit depth buffer. And we don't support stencil buffers in the iPhone OS OpenGL ES implementation. There's two texture units that can be up to 1024 pixels in width and height, and must be a power of two. And there's eight lights and one clip plane supported.

All right. So we've talked a lot about the details of OpenGL ES on the iPhone. We're going to spend a few moments talking about using OpenGL ES on the iPhone simulator next. So what can we say about that? Well, this is an entirely second OpenGL ES implementation in the iPhone OS SDK. This one runs on your host Mac, and it allows you to write your code in the simulator on your Mac, test it out, make sure everything's rendering correctly before you ever deploy it to a device.

This is incredibly handy. Now, the functionality is extremely similar to what's on the device. However, the performance profile is very, very different, or it can be very different. So you want to use the simulator to get real fast turnaround and do rapid experiments, make sure everything's drawing properly, but you want to use the device for all your performance tuning and your mainline development.

So we can take a look next at a demo of the OpenGL ES support in the iPhone simulator. So I'll go back to the demo machine.

And the first thing we want to show is actually the exact same template we just were looking at a moment ago. Now, if we can see what this is about to draw, we have -- it's going to draw a triangle strip here. It's got some rotation. So let's take a look here and target first the iPhone OS simulator. This is the same project. I didn't actually change any of this code. It just came from the template, so you can try this in your own version of the SDK. And all I'm going to do is click Build and Go.

Simulator launches, and we have OpenGL ES rendering in the simulator. And this is, as I said, it's very convenient to use the simulator for sort of quick little experiments. So if I wanted to stop this application and actually say, oh, I need to change my mind. I actually would like the background color to be a little different, and I want the rotation of this square to be a little different, I can actually save that, change it, and with a few keystrokes, I can see the results of what I changed. So this is very, very handy.

Now, actually, if we can switch back to the slides for one moment, I want to show one other bit of information. So a few years ago at WWDC, we showed a demo running on Mac OS X that was highlighting the use of framebuffer objects. And that demo was called FBO Bunnies.

Now, this was back when FBO bunnies were new. And we thought, well, we're using frame buffer objects very heavily on the iPhone. This would be a good candidate demo to run on the iPhone itself. So we decided, great, we're going to take FBO bunnies, we're going to change it from OpenGL to OpenGL ES, and we thought, great, this demo needs a new name. So we decided we would give it a new name.

is now FBO Bunnies. So switch back to the demo machine here, and we'll show that demo running. Okay, so we'll quit out of our project that was the template, and we'll switch into our second demo. And this is the bunnies demo. Now, first off, let's take a look at this thing running on the Mac. Now, we can do this by, again, just a few project settings.

will have this run just so you can see what it looked like when it was running on the Mac. Now, this is running in a Mac OS window using the native OpenGL implementation on the Mac using OpenGL ES here. This demo basically draws this bunny into a texture, texture maps it, puts it on the side of a cube, and then, as bunnies often want to do, multiplies.

And then it does a depth of field effect on the bunnies. All this is done with the Frame Buffer Object API and some render to texture techniques. Let's take a look at that same demo running in the simulator. Now, actually, before we do that, let's see what we actually had to do to get this running.

So at the top of this file, we have this target OS additional we've set to target the device, the iPhone versus the Mac. And if we look at what we actually had to change, you'll see it's a very, very small amount of changes. The first change here is we have to include a slightly different set of headers.

OpenGL on the Mac, OpenGL ES on the phone. And we also include a slightly different set of names for the FrameBufferObject API. As we mentioned earlier, this is the OES FrameBufferObject API. But basically, it's the same. You can see all we really did was change a few values on the suffix.

And lastly, there's a change here from single precision to double precision. But that's it. We don't actually use this conditional anywhere else in the file. And we have one other very subtle change in that the iPhone OS uses Eagle, as we've described, to manage its windows, while the Mac uses Glut, or can use Glut.

And this is just a simple little demo, so we decided to use it. So the Mac sets up its code here using the Glut APIs. And the OpenGL ES The iPhone version uses the Eagle template that we set up earlier. So let's see this running first now on the iPhone simulator.

Here's the demo running on the simulator. So this is actually very cool. Like, we haven't hardly changed the code at all. We made a few little syntactic changes to some lines of code. But the content is all exactly the same. It draws. It looks the same. And it performs pretty well.

This is running in the simulator. So let's take a look next at this exact same code running on the device. This time, we actually won't need to change the code at all. We'll simply target the device in the Xcode window. And I need to switch to the iPhone display screen here. All right. So we'll get this running.

There we go. In theory. Ah, there we go. Great. So here's the same demo. We were just running it on the Mac a moment ago. Our demo from a few years ago, WWC, showing the state of the art technology on your Mac. And now, the power of that rendering is in your iPhone.

So let's stop that, and we'll come back to that demo one more time in a few moments. Well, let's take a look at our last section here. So we want to spend some time-- back to the slides, please-- spend some time talking about our recommendations and best practices for developing your own application. So we've divided these into a few sections. The first sections are just some general guidelines and how to best coexist and how to use the CAEagle layer efficiently. And then we talk a fair amount about optimizing your application to minimize memory bandwidth, storage, and CPU overhead.

Another key point. A lot of developers who do embedded device programming are used to not having a full floating point support on their processor, so they may have some kind of emulated support for floating point. The iPhone CPU fully supports floating point natively, so this is just wasted work. So if you have that code in something you're trying to port to the iPhone, you should just get rid of it, because it won't help you out here, and it'll just eat up your CPU.

In the OpenGL ES case, there's a lot of features in OpenGL ES that give you real high quality rendering, but are more expensive than some other features. Such things as blending and lighting can be expensive. So you don't want to leave those enabled if you don't need to. Now there's sort of a trade-off to be made, because our next recommendation is actually to avoid OpenGL ES state changes as much as possible.

Now, these are somewhat in conflict, because if there are certain expensive features you have enabled, you may want to turn them off to disable them, but you don't want to be toggling them back and forth. So in general, what we can say is that OpenGL ES state changes in general can be expensive, and you should take steps to try and minimize them. You can do this by keeping a shadow copy of your OpenGL ES state, and this can help you avoid redundant state changes, and it can also avoid the need to query the OpenGL for its state.

Now, the next two points are really influenced by the tile-based deferred mode rendering architecture. And as we mentioned them briefly earlier, but you want to minimize the number of rendering destination changes within a given frame. And you want to carefully schedule any of the operations in OpenGL ES that can cause you to force a frame to finish. And you basically want to do these at the beginning or perhaps the end of a frame, but not lots of times in the middle of your frame.

Now there's a few more techniques here in this sort of general category. You want to use alpha blending instead of alpha testing. This might be a little counterintuitive on other implementations. Alpha blending might be really expensive, and alpha testing might be, you know, saving you the cost of doing that expensive work. But in this implementation, alpha tests can actually defeat the hidden surface removal techniques in the GPU, and alpha blending is actually very inexpensive.

You generally want to draw your opaque objects first and your blended objects last. And this can help increase the effectiveness of the hidden surface removal parts of the GPU. And you want to try and minimize certain state changes that can be expensive. And some of these include things like scissor, viewport, and dither state.

Generally, these are not things that you might change often anyway. But we've seen a fair amount of applications coming to the iPhone where they're not used to these things being expensive. And so we wanted to call that out. We also recommend that you don't allocate a depth buffer unnecessarily. So if you're going through some effort to sort your state, you can actually save some memory bandwidth by not even bothering to allocate a depth buffer. It simply can waste memory storage and also bandwidth.

And lastly, in this category, we want to say that you should try and minimize the number of draw calls that you're making. So you can do this by, again, sorting your state to put similar states together before drawing. And you can also do this through little tricks like joining your triangle strips together with some degenerate triangles.

The next section we want to talk about here is how to best coexist with the iPhone OS user interface. Now, as we mentioned, we're rendering into Core Animation layers. And Core Animation is very efficient at compositing content from different sources. But in general, when you're playing a game, you mostly want to focus in on the content you're generating. So if you want the most efficient display path possible, you need to use screen size CA Eagle layers.

In general, you also want to avoid making any kind of transformations on your CA Eagle layers in the Core Animation API. If you need to do simple transformations, such as running in landscape mode, you can do that with the OpenGL API itself. Now, if you do it with Core Animation, it will all work. It's just you might want to save that for special effects or cases where you're not trying to run at your highest possible frame rate.

There's a few other areas of concern and tips we wanted to mention with respect to coexisting with the user interface. So as we mentioned, for best performance, generally you want to be the only thing rendering. And so typically you want to use the CA Eagle layer's opaque property and set that to yes, so that your content is essentially treated as opaque by core animation. And this generally means avoid blending your content with the UI. Once again, this will work just fine, but we recommend you don't do it if you're in a performance sensitive region of your code.

Now, if you decide you do want to composite OpenGL ES and your UI, we still recommend that you leave the opaque setting of the CA Eagle layer to yes. So it means you still want to leave it opaque. But you want to put your OpenGL ES content below is a great example of how OpenGL ES can drive iPhone games and other mobile 3D applications.

Jeremy Sandmel is a great example of how OpenGL ES can drive iPhone games and other mobile 3D applications. Jeremy Sandmel is a great example of how OpenGL ES can drive iPhone games and other mobile 3D applications. Jeremy Sandmel is a great example of how OpenGL ES can drive iPhone games and other mobile 3D applications. Jeremy Sandmel is a great example of how OpenGL in OpenGL to make this happen.

And last, this is, once again, an embedded device. Very powerful. It can do really great things. But you're sharing resources with the rest of the system here. So generally, you want to avoid simultaneous rendering with OpenGL ES and UIKit or Core Animation if you have performance sensitive code.

Now, we'll say a few things quickly about optimizing for memory and vertex and texture next. So first for memory, you're using a shared memory system. And what that means is that the memory that you're using for graphics is not necessarily available to the rest of the system when you're using it.

So you generally want to use no more than 24 megs of memory for your textures and your surfaces. And we actually enforce this limit. Now, we say generally you don't want to use more than this, but actually, you know, you're better off the less you're using. So you should try and stay as far below that as possible.

And we give a few techniques in the next few slides that show exactly how you can do that. But some examples, you can compress your texture data using the PVRT texture compression format. And you can also deallocate other memory that you're not using in the system, such as the textures you've just given OpenGL.

Similarly, when dealing with vertex data, you want to minimize the amount of vertex data you're sending. Now, you can do this through a variety of techniques, such as using indexed triangle strips, or you can actually fake the system out, fake the user out, by not sending the geometry in the first place, either culling it yourself or using techniques like dot 3 lighting or texture to just simply make it look more complex than it actually is.

Generally, you want to use the smallest amount of data you can. And this is sort of a general rule for all of the OpenGL ES programming. For instance, you can get by with 4 byte colors, or maybe two texture coordinates instead of four, or shorts for your texture coordinates instead of floats.

This is good. You want to generally pack your data as tightly as possible. And you want to generally avoid fixed point vertex data. The reason we mention this is that, once again, if you're coming from other embedded platforms, maybe you're using that for efficiency purposes. Or performance. On this platform, floating point data is just fine. If you're going to use 32-bit data, you might as well use floating point and get the performance. I'm sorry, get the precision.

Now, on the subject of lighting, there's a few other key points here. Generally, the lighting is fully accelerated by the GPU, but certain types of lights are more expensive than others. So in your lighting model, you want to use the simplest model you can. And this might mean using fewer lights or simpler types of lights. So you can use directional lights instead of spotlights. And you can even pre-compute your lighting if you find that you can store the values in a color or texture.

All right. Lastly, in the subject of textures, we want to say that you should try to minimize your texture storage requirements. This is, again, this general rule of minimizing the amount of memory you're consuming, because it not only saves you storage, but it saves you bandwidth. All of that data has to be read by the GPU. So if you can get by with 16-bit color for your textures, you should try it, and also for your rendering surfaces.

And if you can use the PVRTC compression format for your textures, this can provide a huge win, because it can drastically reduce the amount of texture storage and memory bandwidth that you need. We provide a tool in the SDK, the path listed here on the slide, to actually allow you to create textures in this format, convert from RGBA values to this PVRTC texture compression format.

And when you're using this format, it's generally a good idea to check out what the compression format looks like. We provide this tool. It will actually give you a preview of that, because certain types of images will compress better than others. So you can take a look at that.

Two last points. When using MIT mapping, we generally recommend you use-- first of all, we generally recommend you use MIT mapping because that can also improve efficiency. But also, you should try and use the linear MIT map nearest setting to improve efficiency the greatest. And lastly, again, as an influence of the tile-based deferred mode renderer, you want to try and minimize the number of times you're modifying sub-regions of textures.

All right, so the last thing we want to say about our best practices is use the tools. We have really great tools in the iPhone OS SDK. And one of them is instruments. We've seen some demos so far. I'm going to show you how to use instruments next with OpenGL ES. So we can switch to the demo machine here.

Okay, so let's take a look back at our bunnies demo. And what we will do is we will set this to the release setting, just because we're going to be looking at performance. And what we want to do first is run this demo on the device. So I'm going to rebuild and run the demo on the device, the FBO bunnies demo. And if you can switch for a moment to the iPhone display.

That's running on the device here. And so this will give you a feel for just from looking at it, you know, what the performance is right now. And if we switch for a moment back to the demo machine here, I'll show you what we can do with instruments.

So I'm going to leave this running on the device, but I'm going to switch in Open Instruments. And Instruments provides a default template for OpenGL ES performance app monitoring. So if you can click on that, and we choose that, We will get the instruments panel with the OpenGL ES template. And in this panel, we can simply choose to attach to our running process, which I'll do right here.

Once we've done that, I'm going to start. And it begins recording statistics from the device. So the device statistics I'm recording right here are frames per second, the basic measurement of performance. Just to get a feel for it right now, we can see we're capturing a few different measurements here. But you can see it's kind of noodling around between maybe 22 to a high of 27 on here. So we'll call it 25 frames per second.

All right. Now let's take a look and stop this for one moment, and I'm going to go back and check out another setting that we have in the application itself. So if I look at the application, we decided when we're looking at converting this to run in OpenGL ES and to run on the iPhone instead of a Mac, there's a few tips and techniques that we've recommended here today we wanted to take advantage of.

So we've listed them here. I won't go through them one at a time, but the first one is -- well, you decided to use short arrays instead of floats. So this converted our floating point data to short data and gave that to the GPU, and this was a considerable performance win. We also looked at batching up our rendering.

So once again, taking similar states, putting them together through a variety of techniques, and minimizing the number of draw calls we actually had to do. And lastly, we looked at using 32-bit color before we convert to 16-bit color. to see what effect that gets. So I'm actually going to turn that on. I'm going to stop it from running.

I'm going to rebuild and recompile it.

And you can see now, performance actually, although it's noodling around, is noodling around a higher number here. And if we let it sort of settle in, we see it kind of settles in at about a little above 30 frames a second.

So we've taken a few of these techniques, such as using smaller vertex size data, using 32-bit versus 16-bit colors-- we chose 16-bit-- and batching up our state changes. And we got-- well, this is pretty good. We got about a 20% performance boost. Now I want to look at a few other statistics we have here.

There's quite a few here, and they have some confusing names. We wanted to call out a few of them that are really important. So we've obviously got frames per second. Now we want to see-- another one that's important is GART resident object size. And that is a measure of how much memory we're using. And the last one we want to click on here is command buffer implicit submit count.

Now, Instruments plots all these for us up here and tracks their statistics down here, and I'll just tell you what these are as we go through them. So the first one is CommandBufferImplicitSubmitCount. That is a measurement of how many times the implementation needed to submit the scene on your behalf because of this tile-based deferred-mode rendering architecture. We batch up a certain amount of data, then we submit it to the GPU. And generally, you want that number to be zero or as low as possible. If you see that number climbing, you're generally going to be seeing some performance issues that you want to avoid.

And you can avoid them by taking advantage of all the techniques that we've mentioned today, minimizing the amount of data that you require, minimizing the amount of draw calls. So that's a number you want to watch out for. The other number here is the GartResidentObjectSize. Now, again, a bit of a confusing name, but it's basically telling you how much memory the GPU is needing to use.

Right now, as we look at it, it's using right about 3 megs of memory. I didn't show this before, but in the other version of this demo, before we turned on our faster options, it was using right about 4 megs of memory. So we saved a quarter of our memory by turning these options on. And again, that can help with performance.

Okay, cool. So I'm going to stop this demo now and show you one last demo that kind of puts a lot of this stuff all together. So we've shown you some spinning squares and some cute and fuzzy bunnies. We'll show you something a little more interesting. will show you how to use OpenGL ES to drive iPhone games and other mobile 3D applications.

This demo is a demo from the iPhone-- I'm sorry, from the Imagination Technologies PowerVR MBX SDK. And what this demo shows is sort of some typical game-like content you can create using OpenGL ES. The demos from the SDK, the Imagination Technologies SDK, show off a lot of these tips and techniques, and we'll see what this demo looks like here.

So we have our robots coming out, much scarier than fuzzy bunnies. And the robots will come out, and they'll say hi to each other. And this is exactly the kind of content which looks really great on the iPhone. You know, it's really focused in on sort of game-like rendering scenes. And here the robots, they begin fighting, as robots often do.

is sort of the hidden Steadicam guy following behind him. Now this part makes me sad. He really knows how to kick a guy when he's down. All right, great. So that is robots. Many thanks to Imagination Technologies for providing a demo. OK, so now I'll switch back to the slides here.

And I just want to give a quick review. So there's some key statistics you want to measure with instruments. There's raw performance you can get with frames per second. There's the graphics memory usage with the GART resident object size, and the implicit scene flushing, command buffer implicit submit count. And again, we want to highlight all of these for you so you can use them in tuning your applications to run an OpenGL ES on the iPhone.

Let's take a look at what we've learned today. So we started off by discussing OpenGL ES, what it is, how you use it, how it's a subset of desktop OpenGL 1.5, and what features it contains and what features have been removed. And we learned that it's incredibly powerful. All of the key features for doing hardware accelerated vertex and fragment processing are all still there. And you can use it to create robots and fuzzy bunnies and all of the other cool demos we've seen so far.

We also spent some time talking about using OpenGL ES on the iPhone itself. And for that, we looked at some key new technologies, such as the Eagle API for creating your contexts, Frame Buffer Objects API for managing your frame buffer, and the CA Eagle Layer subclass of Core Animation for displaying your content and integrating with the iPhone OS.

We also took a look at the iPhone OS support in the iPhone-- OpenGL ES support in the iPhone simulator. And in there, we found that we have a fully featured OpenGL ES implementation. And we found that we have a fully featured OpenGL ES implementation in the simulator, which you can use to do your initial development and algorithmic testing, make sure all your code runs and looks just great.

But you should do your performance tuning on the device, using tools such as instruments and other tools, such as SHARP. And finally, we spent a good deal of time talking about the best practices and recommendations. Most of these focused in on two key areas, specifically understanding how to best use the tile-based deferred mode renderer, and how to minimize memory bandwidth, CPU overhead, driver overhead, and memory storage.

We have some more information. You can take a look at the iPhone OS SDK, where we go through all of this in much more detail. And we also have all of the sample code and tools that you can use to do your performance techniques, performance optimization. You should also take a look at the OpenGL ES website on chronos.org. We have a ton of information about OpenGL ES, documentation, and a lot of good recommendations for how to use the API.

And also, Imagination Technologies, makers of the PowerVR MBX Lite GPU and the iPhone, have a great website about the MBX itself. And you can find out all about its architecture there and how to drive it most efficiently. Alan Schaefer, our graphics technology evangelist, can answer all of your questions about OpenGL ES on the iPhone.

And there's a few additional resources here at WWDC. We have a lab tomorrow morning. We encourage you all to come. I mean, based on what we've seen so far, we've just been blown away about... With the SDK that people have had for just a short period of time, it's been really, really cool.

We've seen some amazing demos so far. So come to the OpenGL ES lab. We can answer your questions there. And there's some related sessions. There's some sessions on core animation, general sessions on tuning OpenGL on the desktop, but also apply generally to OpenGL ES on the iPhone. And just general performance tuning sessions. As we saw earlier this afternoon, there's a great session on games for using OpenGL ES in games. And you can check that out as well. All right.

With that, I thank you very much for your attention. I hope you found this very useful. And I'm going to turn it over to Alan Schaefer next and ask some members from the Embedded Graphics team to come up to do some Q&A. Thank you very much. Thank you.