My name is Gokhan Avkarogullari. I'm with the iPhone GP software group at Apple. And my colleague Richard Schreyer and I will be talking about OpenGL ES and the iPhone implementation of OpenGL, what's new on it and all the new things we added on iOS 4. So last year at the WWDC we introduced OpenGLOpenGL ES 2.0 and iPhone 3GS.

Since then, we introduced third-generation iPhone, third-generation iPod Touch, iPad, and we're going to very soon release iPhone 4 and iOS 4. With all this new hardware and software, there are a lot of new features added to the system. Today we're going to give you an overview of the new extensions that we added.

We're going to talk about the retina display, the higher definition display, the higher resolution, and what it means from OpenGL's perspective, which is a pixel-based API. We're going to talk about the impact of high resolution displays on OpenGL based applications. And finally, we're going to talk about multitasking, and how you can make the user experience the greatest with the changes to your OpenGL API calls. We're going to talk about new extensions first.

Multisample Framebuffer is an extension being implemented to help resolve aliasing issues in the rendering operations. So let's look at what aliasing is. Here is an example screenshot from Touch Fighter application. If you draw your attention to the edges of the wings of the fighter plane, you can see that there's a staircase pattern, a jaggedness to it, on all edges on the wings of the plane. If you do a close-up, you can see in the zoomed up version, there's a staircase pattern. This is actually a significant reverse on the live application, because the pattern changes from frame to frame, and they look like busy lines.

So multisampling helps us deal with this problem by smoothing out the edges by rounding it to a higher resolution buffer and then generating results from that higher resolution buffer. So how can we do it? This is your regular way of creating framebuffer objects without multisampling. You would have a framebuffer object that you would use to display the results of your rendering operation, and it would have, normally, a color attachment, a depth attachment, sometimes a stencil attachment.

Which, if you want to do multisampling in your application, you will need two of these framebuffer objects. One to display the results of your rendering operation, and the other one to generate the -- to do the rendering into where you can get higher resolution images. On the right is the one that you will use for displaying the results of your rendering operation.

It only has a single attachment in this case, a color attachment, and it's not filled in yet. On the left is the multisample framebuffer object where the rendering operations initially take place. You can see that it has a depth attachment and that takes place on this frame-buffer object, that's why the other one doesn't have a depth attachment. And you can see that the buffers in the multisample framebuffer object are fill.

Basically it has a result of your rendering commands. So with multisampling, once that is done, and the size of the buffers are different on the right and left. So we're assuming here in this example, there's 4X multisampling, and therefore the buffer is attached to the multisample framebuffer object is four times the size of the buffer attached to the display framebuffer object.

So at the end of your rendering operation, you do resolve, which means that you take the average of the pixel -- of the color values of the samples for each pixel and generate a single color value for the pixel, and write it out to the framebuffer object that you're going to use to display your images on the screen.

So let's look at how you can do it in the code. Here's a single sampled framebuffer creation. You have a color buffer, you generate it, bind it to your FBO, and then you get a storage packing storage for that from the CALayer, and now finally you attach it to the color attachment of your framebuffer object. Similar operations takes place for the depth buffer.

You generate it, bind it, and get storage for it through the render buffer storage API call, and now finally attach it to the depth attachment of your framebuffer object in the single sample case. Now, as I told you before, you have to create two framebuffer objects for the multisampling operation. So let's look at how the framebuffer objects are created for the multisample framebuffer object. The difference is mainly in how you allocate storage for your buffers.

In a multisample framebuffer case, the color buffer storage comes from the render buffer storage multisample Apple API. Just like in the depth render buffer case. The difference between this API and the previous one is you specify to OpenGL how many samples there will be in your buffer. So in this case, in this example, it could be four. Four samples to use. And the buffers that are created will take into account that you are planning to use four samples per pixel, and they allocate buffers based on that information.

Let's look at how you would normally do a render in a single sample case. You would bind your framebuffer object, set your report, issue your draw calls, and finally bind your color buffer that you want to display on the screen, and then present it on the display. With the multisampling case, you have two framebuffer objects, and you do the rendering operations to one and display operations on the other one.

So to do the rendering operation on the multisample framebuffer object, you need to bind multisample framebuffer object first, and set your viewport to your regular draw operations, just like you did before. But you need to get the data from the multisample framebuffer object to the single sample framebuffer object.

Now, for that, we need to set the multisample framebuffer object as a target of a resolve operation, as the retarget of the resolve operation, and the display framebuffer object as the draw target of that frame resolve operation, and finally you should resolve the multisample framebuffer Apple API call to get the contents of the multisample framebuffer object to the screen size on the display frame buffer object. And just before, you would attach, you would bind the color buffer so you can display it on the screen.

So as you can see, the changes to your application to get multisampling behavior enabled is very simple. You need to change your initialization code to generate a multisample framebuffer object, and you need to change your rendering just a little so that you can do a final resolve operation at the end of the rendering loop.

You might be thinking, "What kind of performance implications that would have on my application?" And the performance implications are different, depending on what kind of GP you have on the product. All iPhone OS devices starting with iPhone 3GS have PowerVR SGX GP on them. And PowerVR SGX GP has native hardware restoration support for multisampling operation It understands the difference between what a sample is and what a pixel is, and therefore it grounds your shaders to generate color value per pixel, not per sample.

But it keeps depth values for each sample, and it also does depth testing for each sample, not per pixel. So in the four-sample case, it would do four depth tests and would generate four depth values. A single color value. And depending on if the depth test passes or fails, it might have the same depth value and same color value for all four pixels, or it might have different depth and different color values for all samples.

It might have different depth and color values for all samples, depending on if the pixel is on the edge of a polygon or the inside of a polygon. And after all these values are computed for each sample and your rendering operation is done, it takes the averages of the color values and that one would be your final color value after the resolve operation.

PowerVR MBX Lite is the GP that we use on all our products before iPhone 3GS. Unfortunately, it doesn't have native support, native hardware restoration support, for multisampling. Therefore, we implemented multisampling in that case, a super sampling. Which means that we used high resolution buffers, and since the MBX Lite doesn't know the difference between the sample and pixel, it generates the same, it does processing for each pixel in these high resolution buffer. So it generates color values for each sample, it generates depth samples, does the depth testing, for each sample, and finally that all these values result in a single value written into your result framebuffer object.

That means that there is more performance impact on the PowerVR MBX Lite GPs than there is on the PowerVR SGX GPs. So you might be thinking to yourself, "I could already have done that, render the texture. I could have generated an FPL that does have larger color attachments as textures and read them back and then sample from them and have the Multisample behavior on my application already. So what is the difference?" The difference is that you're giving us the information that your intention is to do multisampling.

So we can actually award the readback and then average operation at the end of your rendering textures multisampling. When we generate the values, the high resolution values, we know that you're going to generate, you're going to use those values to create a single sampled version, so we write out both the high resolution and the lower resolution one.

And I'm going to talk about another extension that helps performance with the multisampling next, that's the discard. With the discard, the difference between using multisampling and render-to-texture becomes even larger. But first, let's give a small demo, what kind of quality impact does multisampling have on your application. So, in this case, as you can see, there is a plane in the front, the edges of the wings are really busy, you can see the patterns changing from frame to frame. And even if you look at the ones actually, the farthest away, it's even more visible over there.

And I turn on multisampling, it becomes significantly better than what it was before. To have no effect whatsoever you have to have infinite resolution, but this one is significantly better than what we had before. And if I go back and forth, it will be more visible to you, the change from non-multisampled case to the multisampled case.

As I said, multisampling is very easy to get into your application. And you should try, experiment, see what kind of quality difference it makes and what kind of performance impact it has on your application. We're going to talk about a second application, that is a Discard Framebuffer extension. Discard Framebuffer, that helps the performance, fill rate performance with multisampling. And even for it helps with the fill rate performance in the non-multisampled case.

Usually once a frame is rendered, the depth and stencil values that are associated with that frame are not needed for rendering of the next frame. In a multisampled case, not only the depth and stencil but the color attachment of the multisampled framebuffer object is not needed to render the next frame.

So by using this extension, you can tell OpenGL that you don't need it, and we can discard those values without writing out to those buffers. Which means that a significant amount of memory bandwidth can be preserved and not used for this operation, the writing-out operation. If your application is fillrate-bound, is bound by the amount of memory activity, then this extension will help significantly to get better performance from your application.

Here's how it conceptually works. Here's our framebuffer object, it's a single sampled example. Our framebuffer has color and depth attachment. Right now not filled in. And the GP has done, has finished rendering and has generated the color and depth values. Without using the Discard extension the next step would be the GP writing out those values to the color and depth attachments.

So what we don't really need for most of the cases is the depth values for the rendering of the next time, the next frame. So if you use Discard, you can avoid writing out the depth value, and it will never leave the GPU. And this way, the write-out section, the amount of memory used for writing out those values will be available to the application, to read more data into your rendering operations. Let's look at the code example.

This is how we normally render in a single sample case when you're not using Discard. You would have your framebuffer bound, and you would send your report, issue rendering commands, and finally present your color buffer on the screen. But with Discard all you need to do is define what attachments you're going to discard, and through the DiscardFrameBufferEXT call, specify which of those attachments to discard. And my memory's possible OpenGL will avoid writing out to those buffers.

You can imagine that in a multisample case, you've created buffers, they are four times larger than the original single sample case, so there's significantly more memory activity for writing out color and depth values of the multisample buffer. So Discard makes a significant difference in terms of fillrate performance of your application, especially in the multisampled case.

But even in the non-multisampled case, you will still be avoiding writing to the depth and stencil buffers and it will help your fillrate in your application. That was the Discard FramebufferFramebuffer. We have another extension that might help you, that might help on the performance side of your application, and that's Vertex Array Objects.

If you have been to the morning session about OpenGL, the objects in general, GL objects, were discussed in detail, and Vertex Array Objects was one of them. What Vertex Array Objects do is they encapsulate all the data for vertex arrays into a single object. Things like where the offset for your pointer is, your vertex array for your position or normal or for text coordinates, what the size of each element is within those arrays, what the stride is, which arrays are enabled, if there's an index buffer or not.

They're all encapsulated into a single object. So when you switch between different Vertex Array Objects, once you bind all the different information it becomes immediately available to the GL to take advantage of. So how does this help you? First of all, it provides a great convenience. Once you log in your assets for an object, you can log the vertex away and encapsulate the entire state into a single object at the load time. And if you're not dynamically updating your object ever after, it means that you're not going to issue these commands ever again, except for binding it and drawing from the Vertex Array Object. It also allows us to do optimizations.

Since this state is validated only once, at the load time, we don't have to rebuild it. We know, unless you update it, we know that nothing has changed, nothing has to be revalidated. So it saves you CP time on state validation. On top of that, the Vertex Array Object gives us very good information about the vertex layout, the layout of your data for your vertices for the arrays. So we can make use of that to find out how to reorder them if necessary to get better performance out of the drawing operations. Let's look at the code example.

This is when you are using a Vertex Buffer Object, how you would render it. You would bind your video, and you would set your pointers for your position, for your normal, for your texture coordinates, and you would basically enable the non-default enables, such as the normal rate, enable the texture coordinate enables, and you will basically do-- you will do the draw operation by calling DrawArrays or drawElements calls. And finally, you have to set the state back so it doesn't negatively impact the next rendering operation. This has to be done at every draw call. You have to specify these things when you're using VBOs at every draw call.

With VAOs, you only need to specify which VAO you're going to use, because all that information is already encapsulated in the VAO object. So you know about them, you don't have to re-specify them. And we use that and then draw our arrays based on that VAO. This is possible because you've done the work to define the state only once, at one-time setup time.

You basically bind your VAO, you specify where the pointers and strides and all these things are, and you specify which states are enabled, and they are basically written out, captured in one place at one time, and can be re-used over and over again for the subsequent drawing calls.

That was Vertex Array Object. As you can see, it's very easy to take advantage of, and it gives you performance boost, it will hopefully make your code less error-prone because there will be fewer lines of code, and it will make your code more readable, because there are also fewer lines of changes in your code.

There are six more extensions I'd like to talk about. I'll give information on each one and how you can use them in your applications. The first one I'd like to talk about is the APPLE_texture_max_level extension. This extension allows the application to specify the maximum (coarsest) mipmap level that may be selected for texturing operation. So it helps us to control the filtering across atlas boundaries. As texture atlases contain textures for multiple different objects, as mipmap levels get smaller, there is some atlas boundary filtering operation that takes place that generates visual artifacts So this extension is implemented to solve that problem.

You can enable this extension by doing a text parameter call with the GL_TEXTURE_MAX_LEVEL_APPLE call, and you can specify up to which mipmap level you want to use for the texturing operation. Let's visualize this. This is a texture atlas from the Quest game. You can see that there are textures in one, single texture object, textures for walls, for stairs, for statues, and all of them are here in one single texture atlas.

If you look at the mipmap levels and I have from 256 by 256 to 1 by 1 in this picture, this is the visualization of the mipmap levels in terms of pixels. But let's look at the mipmap levels in terms of the coordinates. If you were to use 0 to 1 coordinates for the entire texture atlas, the 0 to 1 coordinate on this mipmap levels will look like this.

At the very top level, it will be exactly correct and perfect. It will have all the necessary pixels in it. This is a 256 by 256. But at the lowest level, 1 by 1 pixel, 0 to 1 coordinate scan is only one pixel, and therefore it has only one color.

So if you can imagine that you have the stairs, the walls and the statues, close up, they will use the first one, the 256 by 256 one, but farther away, they will use this one, or something small, something closer in mipmap level to this one, and they will all look the same color.

So texture_max_level extension avoids this problem by allowing you to specify the mipmap levels you care about and only use those. So in this example, you can say the max level is 3, and the texturing hardware will only use the first, second and third level of mipmap levels for texturing from the particular texture object.

Let's look at another extension that modifies the texturing behavior, that is Apple_shader_texture level of detail extension. It gives you explicit control over setting the level of detail for your texturing operations. So if you'd like to have an object look sharper, then you can specify a mipmap level that is finer than the hardware would choose. You can specify a lower level mipmap with higher precision.

Or if you want to have better fluid performance at the expense of having fuzzy detection applied to your objects, you can choose a lower mipmap level, a coarser mipmap level, through this extension. So all you need to do is in your shader, enable the extension and control the mipmap level through the texture_lod.

There are further APIs in this extension for controlling the gradients and such. I'd like you to go through and read the extension to find out what more you can do with this extension. And this is a shader-based extension, it's only available on PowerVR SGX based devices that have PowerVR SGX GPUs on them. Okay.

So we also added the depth texture extension to our system, so that you can capture the depth information into the texture and use it for things like shadow mapping or depth of field effect. In shadow mapping, you would render your scene from the perspective of light, and then you will capture the depth information from the perspective of light, into a texture. You will do it for every light. And then once you're rendering from the perspective of the camera, you can basically calculate if that particular pixel sees any of the lights, and if it sees all of the light, it will be eliminated by all of them.

If it sees some of them, it will be eliminated by some of them. If it sees none of them, it will be entirely in a shadow. So you can use that texture extension to do shadow mapping. But since you're rendering the scene multiple times, there's a lot of fillrate this consumes, so you need to be careful about performance implications of using this technique. Another example of using that textures is depth of field effect.

I will show a small demo of that, so I will talk to that in the demo. One thing I need to remind you that when you are using a texture as a depth attachment to your framebuffer objects to capture the depth information of the scene, you can only use the NEAREST method.

The reason for that is that doing filtering across different depth values, just you know, it's incorrect values. Something close to the river and something far away in the river, when they're filtered, the depth values filter generates something in between, though there's no object in between these two depth values. Okay. So how can you get that texture extension into your application.

You just generate your texture as usual, and when you create your framebuffer object, you attach this texture to the depth attachment of your framebuffer object. And in subsequent rendering operations, the depth information will be captured in this texture, and then you can use it for texturing later to do whatever post-processing effects or whatever effects you want to do. And let's look at the demo, how we can generate a depth of field effect with the depth_texture extension.

So here again, we are using three planes, and I'd like to point out that this is how you would normally render it without the depth of field effect. Everything is sharp, in focus, the plane in the back, the stars, the plane in the front. So here's the depth, visualization of depth information. This is the depth texture captured by rendering to texture and then displaying the texture.

So things that are black are closer to the screen, things that are white are farther away .And we re-render the same thing in a blurred version, at a low resolution and the blurring introduced. So human eye, when it focus onto something, there's a range of objects that are in that range that are sharp, but the things that are closer or things that are farther away from that focus point on the range are blurrier. So this texture, this will capture that blurrier part, and the original scene will capture the sharper part.

You can see that they are quite different. So the operation between the two is basically generating a mixture of these two. So here, I'm visualizing which texture I'm going to use for my finally rendering depth of field effect. If the focus is by the near plane, and this entire range, it will be black, so it's entirely from the darker, from the sharper image.

And as I get the range closer, smaller, it will basically start using the values from the blurred image. So that when I have the range at 0, it means that everything is blurred, nothing is in focus, and everything will be used from the blurred image. So let's look at it visually.

So if I have full range, I end up with the original scene. Original rendering without the depth of field effect. Then if I set the range to shorter, you can see that the stars have become blurrier, and the two planes in the back are blurry. Now the entire thing is blurry.

I'll move the range out and it gets things into focus and it becomes sharper. Or I can use a short range and move my focal point away and the things in the front will become blurrier and the stars and the third plane in the back will become sharper.

As you can see, the things -- the planes in the front are blurrier. But if I move my range to capture the entire -- near and far planes, again everything is sharper, because we have enough range everywhere to have focal point covering everything. So that's one of the examples of how you can use depth_texture extension.

Another shadowing technique that's very popular is stencil shadow volume. This extension, the stencil_wrap extension, helps us to improve performance for that. With this extension, the value of the stencil buffer will wrap around, then the array goes in and out of the shadow volume. Now, stencil shadow volumes is a large topic, and it's a very nice way of creating real-time shadows. We're spending a significant amount of time in the next session over here in the shaders OpenGLS shading and advanced rendering session, on how to generate them, how to use them. There's really cool demos on visualization of this technique and implementation of it.

We added two new data types for texturing operations, float textures, you can specify your textures to contain 16 bit or 32 bit floats. So again, this requires hardware support that is only available on devices that have PowerVR SGX GPUs. The float values that you can store in your textures can be used for visualizing high dynamic range images, for you can use it for tone mapping and display high dynamic range images in all its glory on iOS 4 based devices.

It also can be used for general-purpose GP operations, for the GPU math. You can load your data in the float textures and do FFTs and other signal processing or other applications, whatever math you want to use with this extension. And the way you specify a texture to be float values is basically telling to OpenGL that its format is GL half float OES or GL float OES.

This is the last extension I'm going to talk about today. It's the APPLE_rgb_422 extension. And this extension enables getting video textures onto GPU. Specifically, interleaved 422 YCbCr type of video. In this extension, we do not specify the color format of the video, so it could be based on, it could be captured from a 601, standard definition video format, or it could be coming from HD709 based color format, or it could be JPEG full-range video.

And, so therefore, since we do not specify it, we give you the freedom and flexibility to implement your color space convergent to your-- in your shader. With this extension, when you specify YCbYCr, you copy the values over from Y to the G channel, copy the values from CR to R channel, and Cb values to the B channel, and once you do your color space conversion, you end up with the RGB values that are coming from originally a video texture. Then again this extension relies on hardware support, so it's only available on the devices that have PowerVR SGX.

So, let's look at how you can use this extension. This is how you would do texturing operations, something from a texture in non-RGB 422 case, you will specify your type of image and then you will create a sample and texture and sample from that. With the 422 extension, you need to specify the format of your texture as RGB_422_APPLE, and you need to specify the type of it as unsigned short, either 8 and 8 rev or forward or reverse ordering of Cb and Ys, and finally your shader, you need to convert from YCbCr values to the RGB values, and then you can do whatever effect you want to do, do black and white or just attach your texture to another object that you want to do -- you want to use it on. So these are these other six extensions. The texture_max_level, the shaded texture level of detail, depth texturing, stencil wrap, flow texturing and RGB 422, and with that, I'd like to invite Richard to talk about retina display, the impact of high-resolution displays on performance, and multitasking.

Thank you.

So thank you. So, Gokhan has just given us a description of all the new features you'll find within OpenGL in iOS 4, so I'm going to continue the what's new topic, but really going to focus on what's new in the rest of the platform around you that impacts you as an OpenGL application developer. First and foremost among these is the new Retina Display you'll find on iPhone 4. So, you've undoubtedly seen the demo. The Retina Display gives, is a 640x960 pixel display. That's in effect four times larger than we've seen on any previous iPhone.

One of the really big points I want to drive home about the Retina display is that we're not cramming a whole bunch of content into the upper left-hand corner. All of the various views and other widgets remain physically exactly the same size on the display. The status bar, the URL bar, are all exactly the same size. What's changed is the amount of detail you find within any specific view. This is equivalently true to the UIKit content as it is to OpenGL content.

So how do you actually adopt -- really make the best use of the Retina display? For OpenGL applications, it requires a little bit of adoption. It's not something you get out of the box. The steps are pretty simple. Right off the bat, we want to render more pixels.

We need to allocate a larger image. The second step is, now that you're rendering to a different size image, we've found that a large number of applications have, for their own convenience, hard-coded various pixel dimensions in their applications. That's something that we'll need to flush out. And finally, this is where it really gets interesting, taking advantage of the new display to load great new artwork. So, step 1. Generating high resolution content is actually done on a view by view basis, and this is controlled with a new UIKit API called Content Scale Factor.

So, you can figure your view with the same bounds that you would always have, in a 320 by 480 coordinate space. What changes is that you set the content scale factor to, say, 1 or 2, and that will in turn affect the number of pixels that are allocated to back the image behind that view.

For UIKit content, this is generally set on your behalf to whatever is appropriate for the current device. Right out of the box, all of your buttons and your text fields are going to be as sharp as they possibly can be. But that is not true for OpenGL views.

For OpenGL views, the default value for content scale factor remains at 1, and you have to explicitly opt in by setting that otherwise. Usually, the straightforward thing to do is to query the scale of the screen that you're running on, and then set that to the content scale factor .On an iPhone 3GS, this will be 1, and nothing changes.

On an iPhone 4, this will be 2, and you're effectively doubling the width and height of your render buffer. At the time you call render buffer storage, core animation is going to snapshot both the bounds of your view and the scale factor, and it will do that to arrive at the actual width of the image you'll be rendering into.

Knowing what that width and height is is usually pretty convenient to have, so you can derive that by doing your own bounds times scale, or , even easier and more foolproof, is to just go ahead and ask OpenGL what the allocated width and height are. So, I just want to -- this is actually a pretty good idea to just ask OpenGL and take these two values and stash them away somewhere on the side. They're really useful to have.

That brings us to step 2, and that is, fixing any place where you have any hard-coded dimensions that may no longer be valid. If you've -- if your application is already universal, it runs on both iPhone and iPad, you've probably already done this, and you can move on. If you haven't done that, you may find that you have a few of these cases, and I want to point out a couple of the most common cases you'll find in your application.

First is that while core animation has chosen the size of your color buffer, the depth buffer is something that you allocate, and the sizes of those two resources has to match. And so, and this is a case where we'll want to use that saved pixel width and pixel height, and pass it right on through to render buffer storage.

If you don't do this, you'll find yourself with an incomplete framebuffer and no drawing will happen. Another common case is GL Viewport. Viewport is a function which chooses which subregion of the view you're rendering into at any given point in time, and every single application has to set it at least once. You'll find it somewhere in your source code.

Most applications really don't ever use anything other than a full screen viewport, so this is another case where you'll just want to pass pixel width and pixel height right on through. Step three is actually where it gets really interesting. At this point, your application is a basic correctly adopter of the Retina display. You've now got much greater detail on your polygon edges, but there's still more room to improve things and really take advantage of this display. And so, you know, this is the right place to, for example, load higher resolution textures and other artwork.

Again, if your application is universal, you may already have a library of assets that are perfectly relevant, that you can use right away on this. Usually, the easiest way to do this is take your existing bitmap textures and just add a new base level, and leave all the existing artwork in place. This can really significantly improve the visual quality of your application.

Just one word of caution here, is that you can do this on any iPhone OS device, but it's going to be a waste on the devices that don't have large displays, and so you actually really want to be selective about which devices you choose to load the largest level of detail on. Otherwise you're just burning memory. One other word of warning is using UIImage to load textures.

UIImage has a size property which refers to dimensions, the dimensions of that image, but those dimensions are measured in device-independent points, not pixels. So if you have a higher resolution image that's 256 by 256 pixels, the size might only be, the size in points might only be 128 by 128, so you can't just take those values and pipe them into GL Tech Image 2D.

So, this is another one of those cases where you'll have to do your own size by scale, or you can just drop down a level to CGImageGetWidth, GetHeight, which will give you the image dimensions straight out in pixels. If you get caught up by this, you'll probably see some really really strange effects.

That's really about all there is to say about making the most of the Retina display. If -- tomorrow there's going to be a session that talks about the UIKit changes in detail, which is where you'll hear all about how UIKits measurements in points, where OpenGL is a pixel-based API.

So UIKit can do almost everything for you with no application changes, where you do need some changes for OpenGL. But really ,when you get right down to it, the one line of code change that really matters is setting content scale factor. There's one more really interesting topic about the Retina display, is that you're drawing four times as many pixels.

That can have some pretty significant performance implications. This is equivalently true if your application runs on iPad, or even if your application uses an external display. TVs can be quite large as well. So I want to talk a little bit about this too. So really, the first thing to do here is to roll up your sleeves and start working through he standard set of fillrate optimizations and investigations. You have to think about how many pixels is your application drawing, in this case, X and Y got a lot bigger.

You also have a lot of control over how expensive each pixel is. Properties of mipmaps can significantly improve GPU efficiency. You are in direct control over the complexity of your fragment shaders, operations like Alpha Test and Discard, also the costs of those add up pretty quickly with screen size as well.

I'm really going to stop here, and really not get into the details of performance optimization, because that's a gigantic subject, and we're going spend a whole session on that this afternoon, in OpenGL ES Tuning and Optimization. That being said, in our experience, there are a lot of interesting applications that do have room for performance applications, and do end up being satisfied with the performance they get on these devices, even when running on higher resolutions, both iPhone 4 and iPad.

But that's not universally true. There are -- some developers are really aggressive. They're already using everything these devices have to offer. And for these particularly complex applications, you may find that you've used up all the -- you've optimized everything there is to optimize. And so there's one more big tool in our toolbox, and we're actually going to go back to how many pixels are you actually drawing.

So you don't necessarily need to render at the size that matches the display. For example, on iPhone 4, has a 640 by 960 screen. If you could instead render 720 by 480, that's still a significant step up in quality when compared to a 3GS, and on the other hand, you're only filling half as many pixels as you would be had you gone all the way up to match the display. You're going to find very few other opportunities out there to find a 2x performance jump in a single line of code.

So if this becomes an option that you want to pursue, how do you do this? You could just throw some black bars around the sides, but not really. What you really want to do is we want to scale that, take that lower resolution image and actually scale it to fill the whole display. Okay? How do you do that? Well, you actually don't need to do that at all. This is something that Core Animation will do for you.

In fact, this is something that Core Animation will do for you really well and really efficiently. Much more so than you could do yourself. In this case, you know, actually, move on to the next slide. That is, in the end, a really nice tradeoff between performance and visual quality. So actually, how do you really make use of this? Well, the answer is that you literally have to do nothing. This is how your applications already work right out of the box. The API that controls this is, again, Content Scale Factor.

As I said, for compatibility reasons, today your applications render into a view with a Content Scale Factor of 1, which means that Core Animation is already taking your content and scaling it to fit the display, very efficiently. So right out of the gate, your application already performs as well as it always has, and looks as good as it always has. That's a pretty decent place to start, if you can change your application such that you can run at the native resolution of these devices, well, that's pretty good too.

But we see that there's going to be a fairly large class of applications that do have performance headroom to, say, step it up a little bit, but you can't take a four times jump in the number of pixels. And so there's some really interesting middle grounds to think about here.

One of these is to stick with a scale factor of 1, but adopt anti-aliasing. We just saw Gokhan discuss our edition of the Apple Framebuffer Multisampling Extension. This can also do a really good job of smoothing polygon edges. In our experience, many applications that adopt multisampling end up looking almost as good as if you were running at native resolution, but the performance impact can be much, much less severe than increasing the number of pixels four times. So this is a very compelling tradeoff to think about.

Another interesting option is that you don't necessarily have to pick integer content scale factor. You can pick something between 1 and 2. In the example I started from, 720 by 480, to get that effect, you can set a content scale factor of 1.5, and that's actually all you really have to do. I want to half change the subject now. I want to talk about iPad.

iPad has an even larger display, and so the motives for wanting to do application optimization are just as true, and for some applications, the motives for wanting to render to a smaller render buffer are just as true. There's just one unfortunate catch, and that is that our very convenient new API is new to iOS 4. It's just not there for you to use in iPhone OS 3.2. Fortunately, there is another way, and that is using the UIView Transform property, which has been there since iPhone SDK first shipped.

So, I'm going to put up a snippet of sample code here. So think back to the beginning of the presentation, when I said that the size of your render buffer was your bounds times your scale. On iPad, the scale is implicitly 1. There's no API, so we'll call it implicitly 1.

In which case, if we want to render an 800 by 600 image on iPad, you can set the bounds to 800 by 600, and then you can set on the Transform property, you can set a scaling transform that will take that and scale it up to fill the display.

Performance-wise, these two methods are -- actually, both performance and quality-wise, these two methods are pretty much equivalent. The advantage of this method is that you can start using it on iPhone OS 3.2, whereas ContentScaleFactor for iPhone iOS 4 and later is just more convenient. That's what I have to say about large display performance.

We're going to talk a lot about this kind of performance investigation in detail later this afternoon in the Tuning and Optimization session, and if you just run out of steam there, then you've got some really fine-grained control over what resolution you actually render at, which can significantly reduce the number of pixels you have to fill.

That brings us to the last topic of the day, and that's multitasking. So, I want to start this by providing an example. Say your product is a game, and the user is playing, and they receive a text message. They leave your game to go write a response. Sure, dinner sounds great. They come right back to your game. They probably spent 10 seconds outside of it. So what do they see when they return? They see that they're going to get to wait.

That's not a very good user experience. In fact, they end up waiting for longer than they spent outside of your application in the first place, that's really not a good user experience. And so, this is what we're talking about when we talk about fast app switching. If you went to the session, you heard that there are a bunch of other scenarios. There's voice over IP, location tracking, let's see, location tracking, there's finish tasks, there's audio tasks. All of those are about various modes of doing work while in the background.

Fast App Switching is different. The Fast App Switching scenario is an application that does absolutely nothing in the background. It's completely silent, completely idle. It's there simply to lie in wait, so that it can leap back into action the instant that the user relaunches that app. And in fact, for OpenGL, that is the only mode. GPU access is exclusively for the foreground application to ensure responsiveness. So while you can use some of these other scenarios, you can create a finish task to do CPU processing in the background, you do not have access to the GP one in the background.

There's one really important point I want to make, is that if you think back to that progress bar, a lot of what that application was probably doing was loading things like OpenGL textures. That tends to be pretty time consuming. So one thing you don't have to do is de-allocate all of your OpenGL resources. You can leave all of your textures and all of your buffer objects and everything else in place. You just have to go hands off and not touch them for awhile, but they can stay there.

This means that when a user does bring your application back to the foreground, all o those really expensive to load resources are already there, ready to go right away. Keeping all that stuff in the background does have some implications on memory usage, and that leads to a really interesting tradeoff that you should think about carefully.

It is generally a really good idea to reduce your application's memory usage when you run in the background. For example, the system memory as a whole is a shared resource. If the application in the foreground needs more memory, the system will go find applications in the background to terminate to make room for it. That list is ordered by who's using the most memory.

Guess who's going to be on top? So, you have a really compelling, even perfectly selfish reason, to want to reduce your memory use as much as possible. Because that means that your process is probably going to be more likely to be there to come back to in the first place. On the other hand, if you're making the resume operation slow by spending a whole bunch of time loading resources, we've kind of defeated the purpose. There's really a balancing act to be made here.

So the way you should think about it is look at your application on a resource-by-resource basis, and think about what you really need to pick up right where the user left off. And also think about how expensive is this to re-create. If you've got your standard textures, those are probably pretty expensive to re-create. You want to keep those around. On the other hand, there are some that are really cheap to re-create. Think about your color and depth buffers. There's no drawing in the background, so they're really just sitting there, not doing anything.

Re-creating them is not like a texture, where you have to go to the file system and load data and decompress it and so on. Reallocating your color and depth buffers is just conjuring up empty memory. It's really fast. Also think about cases where you actually have idle resources that aren't actually needed for the current scene.

You know, if you've got a bunch of GL textures that are around because you needed them in the past, and you're keeping them around pre-emptively because you might need them in the future, this is a really good time to clear out all the idle textures in that cache and leave all the active ones in place. A little bit more about the mechanics of it. How do you actually enter the background and come back? Your application is going to receive a data enter background notification.

When this happens, we have to stop our usage of the GPU. Specifically, your access to the GPU ends as soon as this notification returns. So you have to be done before you return from this function. The second is that you want to save application state. In this case, if you're writing a painting application with OpenGL, that might involve using read pixels to pull back what the user has painted and save it to the file system. And this is really important, because your application may be terminated in the background to make memory, to free up memory. And then finally, here's our example of releasing memory for this application.

You know, we say we're going to go release our framebuffers, because we can re-create them really fast, without slowing the user down. On the other side, when your application wants to enter the foreground, you'll receive an applicationWillEnterForeground notification. We'll spend a tiny fraction of a second allocating our framebuffer, and then we're ready to go. This is exactly where we want to be. So that's great. Except that there's one other case here that you really have to think about very carefully. And this is-- and this is, you might receive applicationDidFinishLaunching instead.

If your process was terminated while in the background, then you have lost all of your GL resources, as well as everything else. And you now have to reload them, and that's going to take time. The more interesting part here is restoring that saved state, restoring that state that we saved when entering the background. Because when the user enters -- the user doesn't know when an application is terminated in the background to free up memory. It's a completely invisible implementation detail to them.

So to you, it's effectively unpredictable which one of these paths your application will take, to reenter the foreground. And so, if in one of these cases you put them right back in their game, and everything's golden, whereas in the other case, you make them page through a parade of logos and select a menu and click Load Game, the user is effectively going to see random behavior here. They won't know which case to expect when they press on your icon, and that's fairly disconcerting. And so it's really critical here that regardless of which path your application takes, you put them back in exactly the same place, say, reloading their game.

Ideally, the user just can't tell the difference between these two cases. Practically speaking, there will be a performance difference. The whole point of Fast App Switching is to keep those resources around, and if they're not around, you're going to have to spend time to load them. But for the best-behaved applications, the application's performance will be the only difference in behavior the user can see.

That's Fast App Switching. You want to free up as much memory as possible, but not to the extent that you're going to slow down Resume. And then you also need to think about doing a really good job of saving and restoring your GL state. Which could include actually the contents of your OpenGL resources, if you're modifying those on the fly.

So that actually brings us to the end of today's presentation. To just give you a quick recap of where we've been, we've talked about some new OpenGL extensions to improve the visual quality of your application: multisample, float texture and depth texture. We have new features to improve the performance of your application: Vertex Array Objects and Discard Framebuffer. Discard and Multisample go together particularly well. We talked about how to adopt the Retina display. Ideally, that's one line of code.

We talked about resolution selection for large displays. This is really your big hammer to solve fillrate issues if you have no other option. And finally we talked about multitasking, where the key is always think about the phrase Fast App Switching. We have a number of related sessions. Coming up later this afternoon is OpenGL ES Tuning and Optimization, where we'll go through the process of how to actually look at fillrate performance in your application, as well as introduce a new developer tool that can really help you understand what your application is really doing. Shading and Advanced Rendering is more of an applied session, where we're going to go through some really classic graphics algorithms and then talk about how we practically applied them to the graphics in Quest.

OpenGL Essential Design Practices happened earlier today, and this is a really great talk which goes into the subject of general OpenGL design practices, where you want to use this kind of object, learning about modern API changes. This kind of stuff is equally applicable to both desktop and embedded OpenGL. And then finally, there's a couple sessions that talk about multitasking and the Retina display, as they apply to the whole platform, not just OpenGL.

You can contact Alan Schaffer directly, he's our Game and Graphics Technologies Evangelist, and we also have a great collection of written documentation in the OpenGL ES programming guide for iPhone. So, with that, I hope this talk was useful to you today, and I hope to see you at the labs. Thank you.