Good morning, everybody. And welcome to WWDC 2009. My name is Allan Schaffer, I'm Apple's Technology Graphics Evangelist, and I'm going to be talking to you this morning about game development for the iPhone. So welcome to the center of the universe.

You know, last year I got up here and I had the privilege of making the prediction that we were all witnessing the birth of a new mobile games platform. We talked last year about the iPhone 3G and the iPod touch as being such fantastic platforms for handheld games. Last year went through a lot of technologies that are available on the iPhone, iPhone -- and iPod touch.

Things like touch input, to be able to directly manipulate a character within your game, and the kind of game play experiences that that enables. Also talked about the accelerometer. So just the idea of being able to use the orientation of the device as a game controller itself. Also, the iPhone is always connected.

And so the ability to have multiplayer game experiences was created for you then. Last year we also went through a lot of the graphics and media technologies. We started with a rich graphic stack use OpenGL ES on the iPhone. Went into the capabilities for audio playback and recording, and also for video playback. So you guys took all of that stuff and it's just been an amazing year.

I think you saw something like this yesterday and downstairs. These are a few of the icons of games that are on the App Store today, and I just have a question for you guys, the audience, raise your hand if you have an app published on the App Store.

I'm curious to see. Wow, okay, so that's very cool. All right, and you know games have been huge. The gaming category on the App Store is the largest category on that store. And you guys deserve a huge round of applause for what you've accomplished in the previous year. So thank you guys.

So this year of course we're talking about iPhone OS 3.0, and the technologies that have been added there for game development. On the left, the game kit API, giving you the ability to have network gaming over Bluetooth, and head-to-head communication between two different instances of your application on a device iPod library access. So letting you take the user's iPod library, and the play list, and the songs that they have synced onto their device, and have those play as the background sound track within your game. OpenGL ES 2.0.

So now for game developers, the shader pipeline that you need to develop rich and powerful graphics on the new iPhone 3G S is enabled for you through that API. And something that's very interesting, and I'm sure a lot of game developers have been thinking about is how they can take in that purchase and use that to unlock levels in their game or download additional content. And needless to say also, the iPhone 3G S, being able to take this new platform with higher performance and incredible graphics compatibilities and load and use that for game play.

So to show off some of these capabilities and just the things that we've been talking about, I want to bring back up on stage Graeme Devine, who you saw yesterday during the Graphics and Media State of the Union. He's going to tell you a little bit more about shock. Graeme, take it away.

Good morning. Yesterday we showed you a little bit about shock.

Today I'm going tell you a little bit about how we actually made it. So five days ago we were asked to make a demo for WWDC. And you know, that's not much time to make a demo. But we were given a bunch of engineers, and a phone. A lot of people hadn't seen the phone before.

It's the very first time they've seen the iPhone 3G S. And the one thing that struck us was it has a CPU, which is quite powerful, but it has a GPU, which is extremely powerful. So what would be interesting to us was to write a sandbox application, something that was really simple, and yet could show off the power of the GPU which is where we wanted to spend the majority of our time working. So we came up with the idea of this highly scoped, this really simple game that was just really a sandbox game. And we had this game up and running in one day.

Or maybe even on the first hour. And it was really easy to get up and running. And then we started to add shaders to the background, and shaders to the demonstration. So the first stage was really very, very simple. It was three lines of GLSL code. And a couple things struck me about that.

Last time I wrote a shader was in assembly code. And this is all in C code. And it's way easier than it ever was before to write a shader. This is a fragment shader up and running in just a few lines of code, doing stuff per pixel. So the smoke simulation that you see running is actually six or seven shaders, and it took us about three days to write.

It inserts density and velocity with the ball, and you can see that the bat; see if I move the bat up and down (you know, I hate to lose the game) as we effect the smoke simulation in the background. But the key thing is the simulation is completely solved on the GPU. The CPU isn't doing any of the work. Leaving your CPU free to do things like AI, networking, voice chat, or play pong.

One thing that struck us too, was once we had the simple game up and running and the shader running, we could change the pallets. Lots of ways to be able to change pallets. One of the easiest ways is to be able to just insert a few more lines of GLSL code. But because we're working on a team with artists, we actually made a texture that would let the artist adjust the pallet for us, and that generally gets much better results, because you get things like this fire, actually, then, that looks pretty darn cool.

So if you're listening to the sound, we added sound into the game as well. And you can hear if we stop for a second-or if I stop for a second-that the sound is actually working 3D, and bouncing from one side of the room to the other. I think this is the loudest game of tennis ever. The sound was actually written by an Apple engineer in the CoreOS group. We've a great team working on this project.

Then lastly, we made one more shader. Which is a bump map shader. This is nine lines of GLSL. And you might think, well, I know how to write nine lines of code that just casts a light around a ball. But actually it's taking a normal map and it's attenuating the light based upon the normal, in the normal map, and doing the correct falloff for that light, and working out how the actual light would work against that normal map, and presenting that-so you can see around the ball here, that it's actually falling off around the actual textures in the background.

And all of that is nine lines of GLSL. Now I don't know about you, but last time I wrote a bump map shader, it was a few more lines than nine. The power of the GPU is absolutely incredible, and we were able to make this application in a very short amount of time.

Using -- it was really a good little crash course in a small disciplined team showing off the power of the GPU. But it was a lot of fun to work on, and we're still using this today at Apple to be able to play around with more shaders. And so we wanted to just give you a quick little background on some of the work that went into making this.

Thank you very much.

So thank you, Graeme. And it's a little cruel of me, I think, to make you have to play the game while you were talking about it as well. That had to have been kind of tough. So for folks -- today, this is a two-part session.

Here in part one I'm going to spend some time talking about graphics and audio, and then after lunch we'll come back to talk about game kit, the accelerometer, and store kit. So let's just dive straight in with graphics, and we're going to focus on OpenGL ES. So OpenGL ES is the high performance 3D graphics API that you use for many of the games that you see on the App Store today. It's designed for hardware accelerated 3D rendering.

And OpenGL ES is the mobile variant of the OpenGL standard from the desktop. It's defined by an industry consortium called the Khronos Group, of which Apple is a member. And really, OpenGL ES represents a distillation of the API that you find on the desktop, brought down, -simplified, and made more efficient and appropriate for the capabilities of mobile devices, which is as you can see are very, very capable. And really, what OpenGL ES is, is the front leading edge of the capabilities that you find on OpenGL on the desktop.

There's a lot of the older functionality that have been built up over the years in the desktop API that simply didn't need to be brought over on to a mobile device. There's more efficient ways of submitting geometry now and doing other thingso only that functionality has been brought into OpenGL ES.

So there's a subtle point that we need to make regarding OpenGL ES versions that are now supported on iPhone OS devices. We have some devices that support OpenGL ES version 1.1, and other devices such as the iPhone 3G S that support both, OpenGL ES 1.1, and 2.0, which is a more capable graphics pipeline.

So a lot of times when you hear us talking about OpenGL, you're hear us discussing this graphics pipeline. The parts that are in purple up here are intended to be essentially the elements that are under direct control by you. So the OpenGL ES API, the definition of your 3D geometry and textures, and the creation of frame buffer surface.

And the parts in blue are intended to represent the elements of the pipeline that are implemented in the GPU hardware itself. And so ES 1.1 implements a fixed function pipeline. And so that means that each one of those blue stages is essentially burned into the silicon. And there are elements that you can manipulate for the different stages.

For example, you can change texture modes, or adjust, enable stenciling, change the blending modes, and so on, enable lighting up on the vertex processing stage. But fundamentally, that API is fixed, and its functionality is fixed. So the ES 1.1 API is targeted to hardware that supports that pipeline.

Now all of the rendering that is done in OpenGL ES kind of -- or all of the APIs calls kind of boil down into these three categories where you're either creating objects, things like textures or buffers to render into, you're setting graphics state, that right-hand column, for changing basically knobs and dials along the graphics pipeline to control how the next element to come through the pipeline will be rendered.

And then you're submitting 3D geometry. You know, things like your vertices, normals, colors, and texture coordinates. So now let's draw the contrast between that and the OpenGL ES 2.0 pipeline that you find on the iPhone 3G S. So this is a programmable pipeline. And many of the stages in the hardware are now programmable by you.

You write shader, so you write these little C-like programs using the GL shading language. That is compiled by the system and loaded directly into the hardware. And then the geometry and the different graphics, 3D data, that is submitted to that hardware then runs through the pipeline, but using the program that you wrote to actually handle the processing.

So the objects now, you have another type of object, which are shaders. Vertex shaders and fragment shaders that can be loaded into different parts of the pipeline. A lot of things about setting graphics state now actually just becomes about writing that shader and loading it into the GPU itself. And of course there's a new API now or a different API for submitting 3D geometry using uniforms and attributes, and then sampler is kind of a search case for texture data coming into the vertex unit.

So we're actually not going to really teach you guys OpenGL at this conference, and certainly not in the 20 minutes that I'm spending on it here. So to learn the basics of OpenGL the places to start are the OpenGL ES 2.0 Programming Guide. This will go into the shader pipeline quite a bit. They call it the Gold Book-it's kind of gold and purple, and that's what you're looking for. If you want to get more information about using the GL shading language, there's the so-called Orange Book, OpenGL Shading Language, now in a second edition.

Apple has published a lot of information about our implementation of OpenGL on the iPhone, so that's The OpenGL Es Programming Guide for iPhone. Then for those of you who are maybe thinking about also using OpenGL on the desktop, this is a good place to start. The so-called Red Book is the OpenGL programming guide really for a lot of us who have been using OpenGL for a long time-this is probably where we got started. And then there's a tremendous amount of resources up on the web.

And so OpenGL.org is a great place to go. Now I wanted to pay special attention. This is where you can find the spec. And I would say that those books I showed you can kind of take a beginning up to being, you know, an intermediate to slightly advanced user. But to really become an advanced user of OpenGL you have to read the spec, and really learn about how the internals of that API really work, rather than just how to use the API from above. So check that out.

Now those of you who are knowledgeable about OpenGL ES already and just want to be -- maybe you're not really this familiar with iPhone programming and you want to know how can I drop some OpenGL ES code onto the iPhone and kind of get it to work with all of these other frameworks, like Core Animation and UIKit, and so on. We provide a template in Xcode that lets you just drop some code into the DrawView routine there, and you can really get up and running really, really quickly. So check that out.

But all right, so for this. For today where I want to actually spend the most time is to talk about our implementation of OpenGL ES on the iPhone. And the first place to start is to remind you about these platforms. So here we have a list of the different devices on the left, and the different variants of API in the other two columns.

Notice that OpenGL ES 1.1 is supported across the product line. And so this is how applications that are in the App Store today all written using OpenGL ES are going to be able to run on the iPhone 3G S, and of course are already running there. Now for the iPhone 3G S we also support OpenGL ES 2.0.

And so that's the API that's going to enable the shader pipeline to be used- that's available on that hardware. So it's important to notice, also, that that means that the iPhone 3G S really has kind of two different paths that it can take for applications. Both versions of OpenGL can co-exist on the same device, and we'll be showing you a few best practices about that in just a minute.

So you need to now decide which version of OpenGL are you going to use for your application, and the decision should fundamentally come from the requirements of what you need to be rendering. So for a lot of the very hardcore game developers, they've been asking us, we need that shader pipeline and we need a device that can implement that in hardware.

So for those folks, obviously, it would be a choice of OpenGL ES 2.0. But also, well, you may also want to have a product with the widest reach, and that can be addressed, can load on all of the products in the product line. So for that choice you would choose OpenGL ES 1.1.

And so how you reconcile these two things. Well, this is the way. So what you should probably do is if your application needs the shader pipeline of OpenGL ES 2.0, you are capable of implementing it to use that pipeline, but then also implement code in your application to provide an ES 1.1 fall-back path for all of the other devices.

All right? So that's something we would really recommend that people look into, to get the best reach between platform capability on one hand, and capability of the hardware on the other. So to do this in code is -- this is configured by the EAGL context object. And so when you initialize the EAGL content you actually define which API you want to be using.

You make a choice between ES 2 or ES 1. And it's important to note you can make both of these calls in one application. If you request the ES 2 context on, say, an iPod touch, you don't have hardware there that would be capable of supporting that, so that call will fail. But then, it will return nil.

But you can then go and try to initialize the API with ES 1, and that will succeed. Now I want to switch gears a little bit and talk about some of the extensions to OpenGL that are of interest on this platform, and maybe just to back up a little bit.

So every version of OpenGL defines some core functionality for that version. And then in addition to that, there are a number of extensions specific to that version which have been either added by the vendors themselves, some have been added by Apple, some have been sort of standardized by the Khronos Group and so on.

And there's core functionality and then there's this sort of cloud of extensions around each version. And as the versions progress, the trend is for a lot of the more popular extensions sometimes to make their way into the core of the next version of OpenGL. And so that's something that indeed happened with the frame buffer object extension.

So on the iPhone, we use the frame buffer object extension extensively. It is the destination for all of the rendering that's done in OpenGL on the iPhone. And so every time that you set up your surfaces in OpenGL ES you're creating a frame buffer object. And then all of the rendering will end up there. So this was an extension in OpenGL ES 1.1 in our implementation there, and it's now become a core feature of OpenGL ES 2.0.

The second extension here is PVRTC. So this is Powered VR Texture Compression. And it's because it's important to be using, because memory is such a precious commodity on mobile hardware. So this is a way that lets you take a compressed representation of texture images and load those compressed representations into memory and keep them compressed when they're there. And then the hardware is able to decode that in realtime as those textures are being referenced. And the third one here is a really interesting one.

It's an extension that we've added called the limited non-power-of-2 extension. So a lot of times you might want to define a texture that is something other than a power of two in dimensions, know, something where it has different dimensions, and in particular one that would be interesting on the iPhone hardware is 480 by 320. Because that corresponds to the size of the display. So the support for non-power-of-2 textures is core in OpenGL ES 2.0.

So in OpenGL ES 2.0, you just get it. And the way that the textures are referenced there is through pixel coordinates. It's through pixel coordinates you can use bitmapping- there's a lot of functionality that's associated with that core feature set. We decided that since the hardware on the iPhone 3G S can actually support these non-power-of-2 dimension textures, that we would port that functionality back to the OpenGL ES 1.1 implementation on the iPhone 3G S. And so it's a limited availability extension, and it's available there.

And so it's called the texture 2D limited npot-. So now let me spend a little more time on PVRTC, because this is something we really want to drive home with developers. This allows you to take an original image, say a 32-bit PNG or2 TIF image, and compress its depth down to 2 or 4 bits per pixel. There's various modes that you can set as you're doing the compressions, sort of one for linear weighting for how the compression is down, and one more tuned for the human visual perception. But the main benefit of doing this is the memory savings.

Taking an image from 32 bits down to 4, I mean, that's an incredible amount of memory savings. And usually a game using OpenGL ES is going to have a lot of textures. So there's also a performance benefit from doing so. Reducing the amount of memory that has to be moved around as objects are being rendered can reduce the memory bandwidth pressure as you're drawing your scene, and give you performance enhancement because of that. So we provide a tool with the SDK, and this is the path to it here. Now I wanted to do a quick comparison of different PVRTC results. So the image on the left here is the uncompressed original PNG image.

And this is an explosion sequence that we took from the Touch Fighter game that we showed everybody at last year's WWDC. You see, it's a texture atlas defining sort of an animation sequence for that explosion. Now on the left, original 32-bit image. In the middle there is the compressed down to 4 bits per pixel using PVRTC. I know that we're projecting this, and it might be a little hard to detect. There are only some slight differences in that image.

Now if you go down to 2 bits per pixel, in that one you can start to notice some artifacting, and it depends on how detailed you need your imagery to remain. So PVRTC is great for sort of natural, real-world looking textures. It's less well suited for things like line art or vector type graphics. But for this sort of thing it's really great.

And of course the memory savings, 8X memory savings, 16X memory savings. So just a couple of best practices to bear in mind as we're talking about textures on the iPhone OS hardware. So I'll drive this home again. Minimize your texture storage requirements. You always want to be treating memory on the device as a precious commodity. And I saw how many of you raised your hands about loading your apps, getting apps published on the App Store. You know how precious that memory is.

So the thing to do is, you know, the first stop is to try and use PVRTC. Probably try the 4 bit depth for that. And sort of a pro tip that some people will do, you know, we'll hear from developers, well, you know, the compression quality of PVRTC isn't quite what I want, so I'm sticking with my original image. What you could actually do is sample from a larger original image than what you would be using in your game.

So say you had a 512 by 512 32-bit depth image. Well, what would happen if you increased the -- if you had art work that originally existed at 1 K by 1 K, 32-bit depth. You could depress that into a 4 built PVRTC. And if you work out the math it will actually save you half of the memory compared to that 512 x 512 32-bit image.

It's a great way -- and the quality is very comparable. So it's a great way to use PVRTC. The second thing, obviously, if PVRTC isn't going to work for you then maybe you want to switch over to use a 16-bit depth, just raw depth on your images instead of 32. And we provide a number of different formats for you to use, 4444, 5551 Alpha, and 565 with no Alpha.

And just a reminder, OpenGL ES makes a copy of your texture data when you define it. It keeps its own separate copy. So you can deallocate a copy that you've perhaps loaded from a file if you're not going to be needing it any more. And then one more thing is we -- OpenGL programmers are always talking minimizing stay changes.

And so that previous example, where I had all the different explosions burned into one texture image, that was a texture atlas. And you're able to achieve some efficiency games because of that. Essentially, you can load that one texture and reference all 16 of the explosions within it, rather than having to load 16 different textures and switch between them with state changes at runtime. All right, so I've talked about all of the platforms, and there's actually one more platform that I want to address.

So let's go back to our chart. We have the iPhone 3G S, 3G, iPhone, iPod touch 2nd Gen, and the iPod touch. And there is one more that game developers really need to spend a lot of time focusing on, and it's the Simulator, okay? So the Simulator supports now in the 3.0 SDK supports both an ES 1.1 implementation and an ES 2.0 implementation.

Which -- that is fantastic. For developers to be able to have all -- either one of those choices available as they do their programming on their Mac with the Simulator is going to really speed up your workflow. And just think about the other tools. If you're familiar with working with shaders, you know, you can use things like Quartz Composer to make live edits on a GLSL shader on the Mac, and then very easily take that shader with minor modifications and bring it over into OpenGL ES. So it's a great environment for you to work in, to be able to use the Simulator.

Now there's some fine print, of course. So when you're using the Simulator for game development you need to always keep in mind that the functionality is there. The ES 2.0 implementation and the ES 1.1 implementation are there. But the performance is going to be very, very different from what you see on the device. And it's because the performance is depending on what kind of Mac you're running on and a number of different things like that.

So use the Simulator basically just for your coding and development time. It's to give you faster turn around on minor changes in your builds, but not so much where you would necessarily look at the Simulator to figure out if you're doing the right thing for performance. When you're testing for performance, and you should be always testing performance as well, you need to be doing that on a device.

So be sure to be hopping back and forth between the two of them. Now a couple of fine print about the Simulator. Number one, it doesn't enforce all of the same OpenGL ES error generation that you might actually find when you're on the hardware itself. There's limits in the hardware that are not necessarily enforced on the Simulator.

Let me give you an example. On the original iPhone, iPod touch, iPhone 3G, and iPod touch second generation you have a 24 megabyte limit for the textures and surfaces that you can load into OpenGL. Now that limit no longer exists on the iPhone 3G S. There's no architectural limitation, you just are limited by however much memory you have available. So the Simulator, though, does not enforce that limit no matter how you're running the application. It doesn't know which hardware you would be running that on.

And so just bear that in mind. Likewise, it's a different architecture. You know, on the power on all of the iPhone devices use a tile-based deferred renderer within the GPU. And that's the architecture of the GPU. And it's just-it would be a different representation in the Simulator. So use the Simulator to test for code correctness in your code paths, use the device to test for performance. But all right, so folks, that's going to end now the part of the section on graphics, and I would like to move on to audio.

So we're going to start with audio just to -- sort of a high level view. The Audio Toolbox framework that's available on the iPhone provides a broad set of APIs for audio playback and audio recording for a variety of different application types. But for today, I want to focus on specifically the APIs that you might use for games, and I have a specific scenario in mind that you might be using the OpenAL API, which is a 3D spatial audio API to play your game sounds, and then you might be taking music out of the IpodLibrary to play the background sound track. And so I'd like to talk to you about how you would accomplish that thing together today.

So let's start with OpenAL. So OpenAL is an API for 3D audio mixing, and we see it in use in a lot of the sound effects in -- we see it in use in a lot of the applications in games that are up on the App Store today, just like we see OpenGL in such heavy use as well.

And OpenGL is modeling audio in a 3D space as its being heard by a single listener who is moving around in that 3D space, and may be changing his location, changing his orientation to hear things, and hear things in different ears as he moves around, and so on.

Now OpenAL mimics a lot of the conventions that you may be familiar with from the OpenGL side of the world. So things like the coronet space can be made in common between these two different APIs. A lot of the actual programming style is similar as well, where you're setting up a context, and then all of the things that you're doing with the API after that point are relative to that context. Now on the iPhone, our implementation is OpenAL 1.1 without the audio capture functionality. So it's simply a playback API. And our implementation there is built on top of the remote I/O audio unit.

So that provides you with very low latency playback of your sounds in OpenAL. And OpenAL is an open standard, you can actually find out a lot more about it on the OpenAL.org web site. But now I'm going to go into a bit more detail and walk through some of the fundamentals of OpenAL.

So there's three fundamental concepts with OpenAL. You have sources, and sources are a 3D point in space, or excuse me, a point in the 3D space, that is emitting audio. So in a game perhaps that would be an opponent or a game piece, or some character crawling across your tower defense environment, emitting audio as it goes around. You have buffers, which are the data containers for the audio. And then the listener is you, corresponding to, for example, if you're using OpenGL, corresponding to the I point that you set in OpenGL.

And it's the position in the 3D space from where you're hearing those sources being played. So I'm going to unwind this now, go back through these things. So in OpenAL the listener defines the context for the audio playback that's going to occur. There's one listener moving around, being positioned in the virtual space. And its position orientation is defined -- excuse me -- the orientation of the listener is defined by two vectors. It's a vector of which direction they're facing, and then also a vector that defines which way is up.

So that way if you change the up direction to something else then the ears, the stereo playback might change. So here you see a little bit of code that is how you might set the listener orientation. We are defining an up vector here with a positive Z going up, and, excuse me, -- the looking-at vector is positive Z. So this is like we're looking straight out.

And the up vector is up, with Y up. Okay, the next fundamental objects are sources And each source is a sound in the game -- or is a sound source in the game. And as I say, maybe these are opponents, or maybe these are different game elements that are flying around in the 3D space, and em=itting some audio.

And so there's a number of different attributes that you can actually set to control how sources are rendered. You can change their attenuation, to have control of the fall-off as the source moves away from you. You can adjust the pitch of the playback of that audio from that source.

You can set up the source to loop the contents of its buffers, so it just keeps on playing it over and over again. All right, and so on. So here's again just some sample code. The first line we're attaching up to a buffer; I'll cover that next. We're configuring some attributes on the source. So here's we're setting is, yes, we want looping.

We're setting the position of the source in space to some XYZ value, and we're defining a reference distance for that attenuation fall-off curve. And then when we call AL source play, that source is going to start to be rendered within our 3D sound environ=ment. And the final fundamental topic here are buffers. And these are containers of the audio data itself.

And there are two different ways of defining buffers that -- and the two methods of doing this have differences in terms of how they implement their memory behavior. So AL buffer data and AL buffer data static are the two different ways that you might load data into a buffer. With AL buffer data, OpenAL is going to copy the data you provide into its own internal buffers. And so it will have its own handle to, you know, that audio data, and you're able to release your own copy to conserve memory.

But better may be to use the OpenAL buffer data static extension that we've added into our implementation. So with this one, OpenAL simply references the data, the memory that you have provided to it. And it's very important, then, for you to realize that you can't go messing around with that data if it's potentially going to still be used by OpenAL.

So you must not dispose of the data memory while it's in use, and in particular there are some intricate behaviors that you need to be aware of if you are, if your application is using multiple threads. And so we're going be covering some of the best practices for using the AL buffer data static method routine in the audio playback or the audio playback session at 5 o'clock, later today. Let's now move on to just walking through some code for setting up OpenAL. And before I start this I want to make a mention, both this session and the one after lunch I'm going to be having a lot of sequences like this where I walk through code.

You guys don't have to write all of this down. All of the code I'm taking is either from published examples that are already up on the Dev Center or examples that have been published for WWDC, or that we'll have published within another day or two, there's one of those. And so don't worry about it. Just watch the sequence here and learn the flow, and you can go back and read the details of the code so you don't have to be typing furiously.

Okay, so the first step here is creating the listener object in OpenAL. So we open the device and on the iPhone, we'll use the default system output as the output for your OpenAL rendering. So this will depend on whether you have, for example, the head phones plugged in or not, and so on. Now Step 2 here is to create an OpenAL context, meaning the mixer, or implicitly what this means is to create the listener that's going to be used for rendering. And so this will be the object that all of the sounds are rendered sort of relative to.

And so we create our context here, passing in the device. And we make that context current. And now this is going be the active object that all of the sources that we define after this will be local to this context. I want to note, it's actually as I mentioned before, it's interesting how this correlates over with OpenGL ES.

You go through a lot of the same, you know, a lot of the same API steps, where you're creating a context and then making it current to control a lot of your rendering behavior there as well. All right, so going now to the next step. We have our listener, the next step is to create sources and to create a buffer. So the first -- up here, we're going to create an OpenAL buffer object that we're going use to store the audio data.

And then we fill that buffer using the AL buffer data static method that I'd shown you to the previous slide. Setting the format, sending in the data, the size, and the rate. Now remember, since we're using the buffer data static extension, OpenAL is referencing your copy of that memory, so you cannot delete it if it's potentially still going to be used.

Number 6. Okay, now we create a source object, and we're going to hook that source up to the buffer. So now we have all of our pieces in place. And we can now go and play whatever that sound was. So we set some attributes on the source, here we're setting the looping position and reference distance like I showed you on that previous slide, and set the source to begin playing. And now the listener is going to start hearing that source. And then every frame or every time through our game loop now, we can be changing different attributes on the source and the listener.

So here we're changing the position of the source, we're changing the position of the listener in XYZ space, and changing the orientation of the listener as well. All right, so that was a quick run-through of the sort of capability that you are -- through some of the fundamental capabilities that you'll use if you're using OpenAL. And I certainly recommend if you're developing a game that needs any kind of spatialized audio that you'll look into this API. So moving on in our scenario now, they have our sort of game sounds set up through OpenAL.

And we want to now go and use the iPod library access to load in background music that's going to be the sound track to our game. So iPod library access provides you with, well, you know, access to the iPod library. You're able to instruct the library to playback the user's music directly from within your application. And it also provides a built-in user interface for choosing songs, or letting them choose songs to play.

Also, there are programmatic interfaces to this, so you can programmatically search the library for content or playlist, and construct a queue of what you searched -- of those search results -- and queue those up to be played as well. And as I said, the scenario here is that we're mixing that playback with sort of the foreground audio sounds coming from OpenAL.

So let me just kind of walk you through the different parts of this API. The first is that you have the iPod library on the device itself. And it contains a whole lot of media items. There's a separate media item object for every song or every audio book, or every audio Podcast that's synced over to that device.

So there could be a lot of media items. Now media items can also be organized in the library in collections. You might think of collections being all the songs by one artist, or all the songs on one album, or all the songs in a playlist that have been defined by the user.

You're able to get at all of that information as well. Then pull music out of the library, to select what you're going to be playing from the library there's two objects. Either the Media Picker, which is some built in user interface to be able to just -- for the user to just select from albums and artists and play lists and so on, or as I said, a music query, which is going to let you programmatically decide what to playback.

And then the final object here on the right is the MusicPlayerController. And so this is the object that you'll actually use to playback those media items. So let's look inside a media item. Here we have one song. So it's media type is "music," the song title is "Hanging Around," by the Venetian Blinds -- made it up -- and the album is "Open Up." Okay, and here also you can get at the album art work for that particular album, all right? There's actually a whole lot of other metadata that you can get at for each song or each media item that's in the library. This is a list of all of the things that you can get on a media item here.

It's kind of what you would expect. Over on the left-hand side, title, artist, album, genre, and so on. Then there's some more dynamic information about that media item over here, the items on the right. To get at the data there's a method on the MPMediaItem object called value for property. You just pass in the appropriate string, it's actually a longer string than what I'm showing you here, but you can get at that data. So those are items, and now we also have collections of items.

And so here, for example, might be the collection of all of the songs that I have loaded from the Venetian Blinds. And so we have these five songs on loaded in our library. Or a collection could be a play list that the user has put together of a sequence of songs. All right, and to get at the data for a collection, it's sort of a lot of the same process here. So a collection is an ordered list of items.

You know, your play list, they play in a certain order, albums play in a certain order, and so on. So you can get the array of MPMediaItems directly out of the collection just with the items method. Every collection also has a representative item. So maybe this is the one item that you would use to get the album art work if you were looking at a list of different albums.

And in the case of the media play list subclass, which is a subclass of the collection, there's a number of additional properties that are defined, that you're able to get at, that are specific to play lists. Like their name, a persistent ID, attributes of that play list, and seed items for genus play lists. It'll tell you the media item that was used to generate that genius play list. So okay, so that's tells you what's in the library. Let's now talk to you about getting the stuff out of the library.

So first we'll show you the Media Picker. So this is it. It's a very similar user experience to what you're probably accustomed to from using the iPod application that's shipped with the device. You're able to choose -- it presents different play lists, artists, songs, albums, and so on. And you can just navigate around in this in the way that you're already familiar with. And so to present this to the user, we're just going to call the allocate for this class, it's called the MPMediaPickerController.

We'll set up a delegate, so that delegate is going to receive the list of MPMediaItems that they selected. And then we present.- This object is a View Controller, so we just present it modally. If we selected it to animate, then it will animate in from the bottom, and the user is then able to then go and pick some media items. Then after they made their choices, one of these two delegate methods is going to fire. Either Media Picker did pick media items, and you'll be presented with a collection of what they chose, or Media Picker did cancel, if they cancelled out.

Because we've provided them with a way to get out of this interface once they've gone in. So that's it. I mean, if you just want to present that to your user, you just pop up that UI, they chose the music that they want to play, and boo, you're off, and you can skip this next section and just wait for me to talk about the player controller. But let's say that you do actually want to programmatically get at the music that's in the library to make choices yourself.

So you use MediaQuery for that. And so there's sort of a sequence of events that you go through when you're using a MediaQuery. So typically, you can just allocate one of these and then start adding essentially predicates to it. But it's perhaps a little bit simpler if you actually start from one of the built-in queries. And we have built-in queries for albums, artists, songs, and so on. So down here at the bottom, two examples. So the first query is just doing a songs query and returning the NS array of all of the media items that are loaded in the user's library.

The second one is loading -- returning an NS array of collections which represent all of the play list that they have loaded in their library. Okay, but -- so you started from one of those built-in queries, and now you can actually go and add -- and basically filter down further. You can add media predicates to construct compound queries. So you're setting up -- excuse me -- so your predicate will be based on a particular property that you want to search for. Maybe like album title or song title or play list name.

And then you'll provide the value that you're looking for and the search will be done. So let's look at some of those properties that you can -- there they are, they're in the middle. So the properties you can use to search for to construct a -- a predicate are listed here in the middle. So persistent ID, media type, artist, and so on and so forth.

And you notice that actually this is a shorter list than all of the properties that are available for a media item. Remember, there are things like -- you know, what track number is this on the album. You probably wouldn't want to actually do a search for all song number 4's on albums. It just wouldn't make sense.

These are the ones that make sense to be searching for. And likewise, a media play list also provides these properties that you can construct a predicate with. And now at runtime the top line there is that you can actually query to see which properties you're allowed to construct predicates with, and it's a class method called "can filter by property." All right, so now let's put all of this together. So the first line here, we're going to query the library, we're going to start with a songs query.

We're going to look for all -- within that group, we're going to construct a search predicate of all of the artists whose names match our favorite band now, the Venetian Blinds, and we add that predicate to the query to refine those results. So now the final NS array here is going to be an array of MPMediaItems containing the results of that query.

All right, so that is how you can construct a MediaQuery. So now let me go to the final part of this. The MusicPlayerController object. And this actually is really interesting. This comes in two flavors. You can see indicating here, the application music player and the iPod music player.

And the behaviors of these two objects are different-you need to make a choice as to which one you're going to use. For games, I recommend that you use the application music player, and here's why. So the application music player is going to playback the media items sort of within your application or within the bounds of your application. And so when you quit your app the playback will stop.

Okay? That's probably what you want, if you're a game. Now the other variant is to use the iPod music player. So with that one, you're essentially handing a queue of items over to the iPod functionality and saying "Play these." And if your app quits in the meantime while they're playing, those songs are going to actually still keep playing. Just like they do if you exit out of the iPod app while you're using your phone today. And that behavior is really not appropriate for games.

So as I say, I recommend that you use the application music player. So now let's put it together. To construct the application music player, first you're going to set it up with a queue of items for it to play. These items are the MPMediaItems or a collection of MPMediaItems. Now remember, it's a queue. Because we're just going play one song after another as we go through of the items list.

We have control over the playback mode, so the repeat mode and the shuffle mode here can be set. And there's also some subtlety here as to whether you're using the application music player or the iPod music player. If you're using the iPod music player then these modes you set will actually be sticky, and if you're using the application player they'll be localized to your app.

All right, now when we're using this object controlling playback, play, pause, stop. No problem there. You also have the ability to move through different items in the queue, so if you want in your game you can have something so the user can very easily skip to the next item, say, in the play list. So skip to next, go back to the previous, skip to the beginning of the list, and so on. So now, final example here of putting all of this together. The top line we construct a query using that code that I just showed you a few minutes ago.

So we have some compound query that we have set up looking for a list of items. And it's going to return to us the songs that we want to play. We create our MP MusicPlayerController here, we're selecting the application music player variant of that. We're going to set our repeat mode to repeat the whole play list, so it will just play endlessly. And finally here, we past to the music player a queue, and we can actually just pass the query directly to the player, and it will actually perform the query. And so now the player will have a queue of items to play.

Now, just pause. This is one scenario. The other scenario was that maybe you had popped up that built-in user interface and have the user pick their music that way, so you weren't using a query. If that was the case, then the code would look something like this. In that delegate method of Media Picker did pick media items, we follow the same sequence here, but the place where we pass those media items into the queue is done by calling set queue with item collection.

And that way now those results can get in as well. But whichever one of those two paths you came from, now you can play the music. All right, so now we have our foreground audio being played through OpenAL. We have our background music sound track coming from the iPod Library. And we need to mix them together.

And yesterday at the graphics and media State of the Union we talked a little bit about this API called the AVAudioSession. And this is the API that really makes it very easy for you to control how that mixing behavior happens. There's two particular categories that you can set in the AVAudioSession that might be appropriate depending on the audio behavior you need in your game. So the first category that I want to talk about here is called the ambient category.

This is appropriate for games that have their own perhaps foreground audio coming from OpenAL but want to mix with background music coming off of the iPod. This category is not appropriate for games that need to play their own background music. So for example, if you have MP3s in your game that you're going to be playing and that's your background music sound track, you should not use this category, you should use the one I'm going to be talking about next.

But back on ambient now, this also follows some of the recommended behavior that we have for games running on the platform. Number one, their audio should silence if the user turns the ringer switch to silent. And number two, the audio should be silenced when the screen locks. Now that other category that I was talking about is called ambient solo. So this is a category that's appropriate for games that essentially want to take over and play their own background music.

And play their own MP3s. Around this category will interrupt any playback happening from the iPod, and allow your application to access the audio decoding resources that are available on the device. So this one also follows the recommended behavior for games, obeying the ringer switch, and silencing when the screen locks.

All right, so folks, here is a quick look at how you can set up this audio session object. So first we retrieve an instance to the audio session singleton here, we're going to request the ambient category. So this is appropriate now, because remember our scenario: We want to be playing the foreground music from OpenAL and have background music sound track coming off of the iPod player. So we want to mix.

And the ambient category allows us to mix those two. We set up our session as active, and now we can go on to continue to set up OpenAL and set up access to the iPod library. So there's actually a lot more functionality that you can get at using the AVAudioSession API.

There's a number of other categories that you might be interested in, but really probably aren't appropriate for games in many instances. So it's only if you have a different kind of app that you might want to go look into some of those. But it also provides functionality to handle interruptions and notify you of state changes that happen in the audio system on the device, and enables you to set your preferred hardware settings for different things.

And also there's a lower level C based framework called the Audio Session Services API, that really provides you with a lot of additional flexibility and some more advanced behaviors. Just one for instance of what you'll find there. It's actually possible for you to detect when your application starts up whether there is already audio being played from the iPod. And you might use that knowledge to make a decision about whether or not to play your own background sound track for music. And that decision should drive which one of those audio session categories you choose for your game.