Hi, everybody. Hopefully, you're having a good time so far. I'm Jim Batson. And for those of you watching at home, I'm the one with the full head of hair. OK. Today we're going to be talking to you about AirPlay and external displays. And first thing we're going to cover is how to create a great AirPlay experience for your application. And then we're going to be covering Some specific things about video because there are some nuances there. And finally, we'll be talking about taking advantage of multiple displays so that you can create an immersive environment for your application. So starting with AirPlay.

We released AirPlay for iOS last fall. And since then, we've been working a whole lot, been adding a whole bunch of stuff. Now, AirPlay is a family of technologies and user experiences that basically have one big goal in mind. That's the ability to share the experience

With AirPlay Audio, you have the ability to share your audio stream from your phone or iPad or iPod Touch straight to an Apple TV.

Now, if you-- and this works for all audio on your system. But if you are an application that has music, You can make your app really shine by sending additional information that will be displayed on the AirPlay receiver. Now, AirPlay audio is not just for Apple TV. There's a whole host of devices that are coming out from third-party companies that have AirPlay compatible devices, AirPlay compatible receivers, and dockable speakers and other accessories.

So there's going to be more and more places your AirPlay audio can play. Now on to video. With AirPlay Video, the video content itself is streamed directly to the Apple TV. So that means there's no degradation in quality and you get the best viewing experience. And you get to take advantage of the whole screen.

But one of the great features of AirPlay Video is that while you're watching the video, You can also do multitasking. And so you can do other things while people in the room are watching. So they're checking out your gnarly moves and you still can surf the web and buy stuff for your dad because, you know, it's Father's Day and less than two weeks away.

Now, before we go to AirPlay Screen and AirPlay Mirroring, let's talk a little bit about the iPad 2. When the iPad 2 was released, we introduced a feature called Mirroring for that. And that's accomplished by plugging in the Apple Digital AV adapter into your iPad and hooking it up to a TV that supports HDMI. When you do that plug-in, the screen will automatically be mirrored.

And with iOS 5, with AirPlay mirroring, you've got the same capability but without the wire. So the live audio and the live graphics are sent to an Apple TV wirelessly, and you get the same behavior as you did with a cable, namely that when you first connect, you're mirrored. And this works great for collaboration. You can imagine working on slides with some other people on the big screen while you're manipulating them on the pad. But since it's based on the same technology as the cable case, you've got the support for second display.

Which really works great for things like Keynote, where it has its independent content, something different from the iPad screen. But it's not just great for presentations. It's also great for things like games and other applications. You've got the opportunity here to create applications that go beyond just using it on a single display.

So you have opportunities to kind of create a new user experience. So you should look into that. So how do you as a developer take advantage of AirPlay? Well, there is no AirPlay framework. AirPlay is implemented throughout the system, integrated in all the frameworks and technologies you're already using.

So let's go take a closer look at what you might do. We're going to take a look at increasing the AirPlay experience. We're going to cover a little bit about media routing. about the AirPlay UI. Showing extra information for the user. Accepting remote controls from the AirPlay device and adding multitasking to your apps. Let's start with media routing.

Now, media routing on the system solves the basic problem that when you're playing media, you want the audio and the video to go where the user wants. Now, with physical devices, that's pretty easy. When you plug in some headphones, It's a physical gesture. So you know the user wants the audio to go to the headphones.

When they plug in an HDMI adapter, they want the audio and video to go out through the HDMI adapter to the television. But what if there's a TV available, an Apple TV? There's no physical connection that can be made. So instead, the user finds the AirPlay route picker, uses it and selects the Apple TV as a destination, and then the audio and video will go to the Apple TV now. So while we're on the topic of media routing, let's talk about audio routing behaviors.

Now, when your application makes sounds and plays audio, typically the audio you make is about your application experience. There's ambient sounds or media playback. But there's another class of sounds which are about direct user feedback, things like key clicks or alerts. Now, with iOS 5, when you're using AirPlay, the application audio is going to go to the Apple TV.

But the system sounds are now going to come from the device. So that's where the user is touching. That's where you want key clicks to happen. So it's really important to use system sound APIs for playing system sounds only. You don't want your audio that's intended to go with your content to be playing locally when it should be playing on the remote device.

So some of you have been using system sound APIs because they're easy to use and they're convenient for these small, short sounds. There's a lot of choices, but let me just point you to one of them, AV Audio Player. With AV Audio Player, it's pretty simple. You just create an AV Audio Player using a file URL pointing to your audio content. And then with a few controls, a few playback commands, you can play that sound inside your app. So check that out. Now I'm back to the route picker. I'm going to talk a little details about how the user interacts.

So there's the system AirPlay Picker, which is available through the multitasking user interface. It's always available. The user just double taps the menu button, swipes right, and reveals these media controls. Now, if there's an AirPlay compatible device available, You get an AirPlay button that will appear. And then when the user clicks on that, they get a list.

And they can choose the Apple TV. Now, if the Apple TV is the destination device and if the Apple TV is running iOS 5, mirroring is an option. So that'll appear and they can just slide that. You saw that yesterday to invoke AirPlay mirroring. So keep in mind that the system route AirPlay Picker is always available.

Even though your app's in the front most, the user can always get there and change the route. Just be aware of that. Now, for media-based applications, it's really convenient to have an AirPlay picker inside your application UI. You don't want the user to have to change their focus from your application to direct the content to wherever they want to play.

So how do you go about adding an AirPlay picker to your UI? It's really easy. Just use MPVolumeView. You just create a volume view and just add it as a subview to one of your views. Then you can place it around. So the default is that you get the volume slider as well as the route picker.

But if you want the picker only, it's really easy. You just set show as volume slider to no. And then you just get the picker element. Now keep in mind, if there's no device available, the picker is not visible. But now you've got the ability to place that wherever you want in your UI.

So once you're routing media content from the device to the AirPlay receiver, you want to enhance that experience on that receiver. And you can do that because if you don't do any extra coding yourself, there's nothing we can do. There's just no information for us to display there.

But with iOS 5, you as an application have the ability to send metadata over that will be displayed on the Apple TV. So you use MPNow Playing Info Center, and it's part of the Media Player Framework. And it works with any playback framework you're using to play your content.

And it's really simple to use. You just call setNowPlayingInfo with a dictionary of the metadata that you want associated with a song. What kind of metadata? Well, besides the standard strings of like song name, artist, album title, things like that, you have the ability to send artwork over.

And that kind of really enriches the experience. Now, besides the static information, you can also have dynamic information that updates the progress bar visible on the TV. You've got playback rate, duration, and elapsed time. Now, it's important to give an accurate playback rate. The reason is that the progress bar is updated on the device. By interpolating those values so you don't have to keep hitting it with, now is the elapsed time, now is the elapsed time. So you don't want to do that because that'll just unnecessarily send lots of data.

Now that you've gotten extra information going to the device being displayed on the Apple TV or on an AirPlay compatible receiver, you want to be able to handle events coming from the remote device. That allows your user to control the media playback using the remote control from your Apple TV. And it's not just for Apple TV. These same APIs work with a host of other devices.

The example here is that the headphones-- you've got the remote control clicker-- So if you support these APIs, those will control the media playback as well as the lock screen. And then there's a host of accessories that also support remote controls. So what do you need to do to actually support remote controls? Well, the first thing you have to do is the application has to ask to receive them, these remote control events.

First thing you do is you have to become a first responder. And you do that by first overriding the can become first responder method and returning yes. So the system knows you can be in that situation. In your view, did the appear method, you ask to begin receiving those remote control events. And then you become the first responder. Now this just makes you eligible to receive these remote control events.

The system will decide when to send them to you. Typically, that's when the user has pressed play inside your UI and media starts playing. Now, it's important to also let the system know, don't send them to me anymore. So you do that by calling and receiving remote control events and resign first responder.

A very short example would be, let's say I wrote a magazine application where I can play incidental videos from my site, but when they're just reading the magazine elsewhere, you'd like the user to be able to listen to their music from their music application, and you want them to control it. So it's important to have the right context and only be getting these remote control events when it's appropriate for your app.

So once you start getting these remote control events, what do you do with them? Well, when you get the events, you look at the type and subtype. And there's a bunch of different events that come to you, including toggle, play, pause, next track, previous track. It's up to your application to interpret those control events appropriately for your app.

Now, be careful and do something sensible. Since these are control events for the user, you don't want them to be confusing for your app. Otherwise, they're just going to think your app behaves strangely. Now, once you've dealt with controlling your media playback app from the remote events, you want to add multitasking to round out the experience. Now, with iOS 4, you've been able to play audio in the background. And of course, playing audio in the background from your app works great with AirPlay Audio.

With AirPlay Video, you want to be able to play the video in the background and let the user multitask and do something else as well. So what do you have to do? We do the same thing that you do from playing audio in the background. You use Xcode, open your Info.plist.

Add a required background modes key and then add the app plays audio item to that key. That flags your app as being able to do that. That's not quite the whole story. You need to also use AV Audio Session. AV Audio Session is a critical piece. AV Audio Session lets you tell the system what kind of audio you're playing.

And for the case of media playback, you want to use the playback category. This gets all the right behaviors for the audio in your application. Now, there's a whole host of things you can do with IV Audio Session, and it's a good idea to really understand all those intricate details. So I encourage you to check out the Audio Session Management for iOS session tomorrow.

So now that you've added multitasking to your application, you've combined all these three to kind of round out the whole experience. Now let's look at the lock screen. So first of all, your phone's locked and your audio's still playing. That's because you've implemented a background media playback. The lock screen also has that great album art. That's because you use Now Playing Info. And the user can control playback using either the remote on the headset or the controls in lock screen.

So I encourage you, if you're playing any kind of media playback, to support all these things together and create a nice full rounded experience for your application. Next, we're going to talk about-- Video and getting some more out of the AirPlay video experience. We'll talk about some of the different APIs available, determining when best to use AirPlay video, Showing the user where the video is playing and a few tips on providing the best full playback experience for the screen. Now, before we jump into which frameworks, let's take a quick sideline about the kind of content.

Media content, especially video content, can come from many places. AirPlay video works well with local content, content either from within your app or from the asset library. But it's also used for internet content, things like progressive download and HDTV live streaming. Now, HDTV live streaming is used for video on demand as well as live events. And with iOS 5, encrypted streams work with AirPlay Video. They work with AirPlay Video if you've taken our recommendations from the past to implement encrypted streams the way we suggest.

So what are those recommendations? It's all about getting the right key, getting the keys to the system. So either use custom URL protocols using NSURL protocol to provide the keys, or you can use HTTPS and session cookies to provide keys to the system. Now, later this afternoon, there's a session on HDTV live streaming, filling in all the changes that are happening, and some more details here as well. I encourage you to go to that session to find out all the latest.

Now about the APIs. You've heard a lot about AV Foundation, a powerful set of APIs and a framework that lets you do lots of things. You get a lot of control, but it has no UI elements. So you can totally write your app on top of AV Foundation. It's great. If you want something a little bit higher level, you have another choice, the media player framework. The media player framework embodies the expected behavior that we want the apps to have in terms of media playback. So that's a good place to look.

So when's the right time to allow AirPlay Video to work for your application and for your content? If your application is a media playback app, it makes a lot of sense because you want the full playback experience. Like I said before, the content is streamed directly to the device. No degradation in playback quality. Works great.

But it's not always the right choice. Imagine the user had just finished watching their favorite TV show. Display it on the big screen. And then they go, play your video game, and you have an intro movie. You don't want your intro movie to show up on the TV taking focus away from where they're going to be playing the game.

The other case, other than incidental videos, is audio playback. You don't want the AirPlay video mode to be used when you're playing audio-only content. MP Movie Player Controller takes care of this for you, but when you're using AV Foundation and AV Player, you need to take care of that yourself.

Keep in mind that in iOS 5, AirPlay Video is now allowed by default. When we first released AirPlay Video in iOS 4.3, It was disabled for media player framework and for web pages. The application or the web page designer had to explicitly enable it. With iOS 5, not only have we brought AirPlay video to AV Foundation and UI WebView, but it's now the default to be allowed. So if you're delivering content or applications to multiple iOS releases, go ahead and be explicit. Just use these APIs to set the value that needs to be.

So how do you do that for applications? It's pretty easy. There's a couple of properties, mostly named about the same, allows AirPlay. And you just set that to yes or no. And UI WebView has media playback allows AirPlay. This basically allows... web pages that enable AirPlay to play inside the UI web view.

It's a little bit more about the web pages. There's two ways to enable AirPlay video playback on a web page, using the video tag and the embed tag. For the video tag, it's xWebKitAirPlay. You set that to allow or deny. And for the embed tag, you use AirPlay, and you set that to allow or deny.

So when you're using AirPlay video and it is actually playing on the TV, Make sure that your app doesn't look broken. You don't want the user to be sitting there and be confused what's going on. Instead, you want to give them an indication where to look to see their content.

So what do you have to do with MPMoviePlayerController to get that behavior? Nothing. It does it for you. But for AV player, you need to do some work yourself. Like I said before, there's no built-in UI. There's no UI elements inside AV Foundation. Instead, you do this yourself. You check the property AirPlay Video Active, and if AirPlay Video is active, you display your helpful message. And it's key value observable, so you can listen for state changes. Now let's talk a little bit about the full screen experience.

Here's an example of an application that uses inline video. While you're playing the video, you've got this additional content that goes along with the video and enhances the experience for the user. Whether you're hooked up to a TV through a wire or through an AirPlay screen, it makes sense to mirror the whole display because you want to watch the video and see the other content. Now, if the user clicks on the full screen button, you want to transition into a mode where you take advantage of the full screen of the TV, either through AirPlay Video, or using UI Screen APIs, which we'll talk more about later.

So to achieve this with MP Movie Player Controller, it takes care of most of it for you, but it needs one piece of information. It needs to know that you mean to be in full screen. So you just set full screen, and then the player will do the right thing.

It's not okay just to stretch the view to fit, because then you're not going to get the right behavior. With AVPlayer, you've got more work to do. The application is responsible for taking advantage of that external display. So typically, your application has a concept that it wants to be in full screen or not.

When you're in AirPlay screen mode, everything's being mirrored. So you need to tell us that it's okay to switch to AirPlay video. So you set the use this AirPlay video while AirPlay screen active property to yes. And this is harmless if AirPlay screen is not active. So go ahead and set it whenever your app is in a full screen mode. And then we'll just do the right thing then. Now in the wired case, you need to do more work with the UI screen APIs to get the video on the second display. So let's talk more about using multiple displays.

Remember what I said before. AirPlay Screen, we use the same basic techniques at the lowest level to implement the mirroring and second display as we did for wired displays through the Apple Digital AV adapter. That means the same APIs, from your perspective, work. And it's great for all apps. Not just media playback apps. And to tell you more about that, I'd like to get Josh on the stage, and he'll walk you through all sorts of great stuff.

All right. Thanks, Jim. So now we're going to talk a little bit more about how you can take full advantage of these external displays as a unique second display. Everything that Jim's been talking about so far is nice tweaks that you can make to your existing applications and the things you already have in place to get great experiences with the existing AirPlay video and AirPlay audio technologies. So now we'll look at using those second displays as independent content.

So we're going to cover a few things today. First, we'll talk about mirroring your content to the external display, which, as Jim said, is the default behavior. If you do no other work, this is what you'll get. But then from there, we'll talk about how you can use that second display to display additional content that is not the same as what's showing up on the internal panel. Once we've covered that, there's two other sort of side notes we're going to go into.

The first is overscan. So if you've been around for, you know, the last 30 years, you've probably seen overscan at some point and hoped that it had gone away by now. But it's here, so we'll talk about it. And finally, we're going to talk about screen modes, which we'll get into later on, how you can take advantage of different modes available on the display.

So first off, what are we talking about with this second display? Well, we're not talking about this. You may be used to Mac OS X. Second display is an extended desktop where you share content across the displays and you can bridge them with a single window. That's not what we're talking about on iOS.

On iOS, the second display is a unique display. It is controlled entirely by the foreground app. So while your application is frontmost, you have full control over all content that shows up on that secondary display. No one else is going to run and take over your screen. You also have these two different modes, the mirroring and the second display, where you can display separate content.

So first off, this mirroring. If you do nothing else, as I said, this will happen automatically as soon as you either plug in the hard cable or your user picks AirPlay from the route picker and turns on mirroring. And anything you have displayed on your internal panel will automatically get mirrored out to that second display exactly as you see it internally.

Now, of course, as you can see here, this isn't necessarily ideal. You're not using that entire screen space that's available to you. You've got this beautiful 16 by 9 display connected, but we've got these big bars down the side because our main content is just taking up that small amount in the middle.

So the nice thing, of course, is that you don't have to do any additional work to get this. There is one thing that you really do need to keep in mind if you plan on using mirroring, though, and that is to set your status bar orientation appropriately. Now, hopefully you already know this and are doing it for other reasons. For instance, if you use keyboards in your application, the orientation that the keyboard comes up in will follow the status bar orientation. So if you have the wrong status bar orientation, it may come in upside down. System alerts will appear in the status bar orientation.

So there's many reasons that you want to make sure this is right. But for the purposes of AirPlay Screen, your mirroring will be upside down if you have your status bar orientation right or possibly rotated sideways. So if your user rotates the iPad and you don't update the status bar orientation, you're going to get that.

Probably not what you wanted. But it's really easy to fix. Just make sure you set the status bar orientation to match whatever orientation you're rendering your content in, and it will just rotate right around and be correct right side up. no matter what orientation the user is holding their iPad in.

So if you need to detect that mirroring is happening, for instance, somewhere in your application you decide you need to do something slightly different in the case where mirroring is occurring, it's very easy with one line of code to detect that this is happening. Once you get your UI screen object, which we'll look at in a little bit, you can just send it the mirrored screen message. And if mirroring is occurring, the object you get back from mirrored screen will be UI screen main screen, because mirroring always is mirroring that main screen. So one line of code, easy to see if it's happening.

This is supported, of course, on iPad 2, and it's supported with a variety of cables, including the Apple Digital AV adapter, the Apple VGA adapter, and, of course, now Apple TV. So there's a number of different technologies that you can have connected with and still be getting this mirroring experience. They're all the same from the purposes of your application.

All right, so that's mirroring. Not a whole lot there. Pretty easy to take advantage of. But what if you want to really take full advantage of that second display and fill that full 16 by 9 bounds with that beautiful content that you have in your app? Well, we want to do something more like this and really take advantage of that screen. This is pretty easy, too.

The one thing that I want to strongly encourage you to do is avoid this. If you're using second display and displaying separate content out to the panel, don't leave your main iPad or iPhone screen blank. The worst thing you can do for your user is have the iPad, this beautiful device with a great screen in their hands with nothing on it. So make sure you keep something, some UI, on the main screen as well.

So if you want to take advantage of this, UIScreen is the API that you use to do all your drawing to the external screen. And the great thing about this is that you already know how to do it. Any content that you know how to draw on the internal panel, you can draw on the external screen using all of the same APIs plus UIScreen.

So there's two main things you'll have to do in order to make use of UIScreen in this way. The first is to detect the presence of an external display so that you know that it's time to try and display content out there. And the second then is to configure that screen and place content on it.

So there's actually two places where you'll want to detect the presence of a screen. The first is on application launch. And as with most other things that you would do on application launch, you do this in the application did finish launching method. You just can check on your application launch if there's already a screen attached. And you do that just by counting the number of screens. The first screen in the UIScreen screens array will always be the main screen, the internal panel. So if there's more than one there, the rest of them are external displays.

For the purposes of what I'm looking at here, we're going to simplify it and just get the last screen out of that array. You may want to look through the array and see if there's one in particular that you were interested in. But I'm just going to grab the last screen. And I'll write in just a minute this prepare screen method. This isn't a system call.

It's one that we're going to write in just a minute. But we're going to use that same method to detect both application launch screens and screens that come later. So the second case is detecting display connections. connections after your application is already running. And this is also really easy using an NS notification, UI screen did connect notification.

So probably also, when your application did finish launching, you would just register for this notification and then you'll be called back any time the user plugs in a cable or connects to an AirPlay screen through the route picker. And again, once we've registered for this notification, we can call that same prepare screen method, and this time we'll get the UI screen object out of the notification because UIKit has passed the screen object that was connected to you as the object of the notification. So in both cases, we get the screen we want to display on and call this prepare screen method that we're about to write. Now that method is pretty easy, except you get to see it all at once because my build didn't work.

There's basically three things that we want to do here. The first is we want to get the screen's bounds. We want to fill that entire display with content. So we want to get the screen bounds, which is the outer size of that entire display. Then we'll create a new UI window that fills that entire screen bounds to make sure we fill the entire area on that display.

The only other thing we have to do then is move that screen to the display to tell UIKit that we want to be drawing to the external screen. So we'll call window set screen and pass in the screen that we decided we want to display on. Of course, there is one last thing, which is setting the window's hidden property to no, because by default, windows, when they're created, start out hidden. So we've moved that window to the screen. We've caused it to become visible by setting hidden no. And now you can use any of the technologies you already know how to use on your internal panel.

To draw content to the external panel. So you can use OpenGL through CAEAGL layer or any of the various UIKit classes to get content out to that screen exactly the same way you would get it onto your internal screen. So we've now written our app. Everything's looking great.

We've got this beautiful game that we wrote and we decide that we're going to add some overlays on top of our content. We want to add maybe some health information for our player and a couple of messages along the bottom. So we just add those around the edges of our game and everything's looking great. We've also added some of our So this is something that you will end up seeing and have to deal with in some way.

But we've greatly improved this situation. In iOS 5. So we'll get to that in just a second. Of course, what we want to do is inset that content so that it doesn't get clipped. Now we could do that by just shrinking everything down. But of course, then we have these black bars around the edges and I'll just highlight them in red to make it more clear that they're there. So our content no longer fills the entire screen. Ideally, we would like to actually stretch that background content out to fill all the available pixels, but still avoid clipping the bits that are over top.

So UIKit has actually been able to do that. So we've added these new modes to help you adapt to this UI screen overscanned situation much easier than was ever possible before. So the new property is UI screen overscan compensation. And there are three possible modes that you can set it to.

The default is UI screen overscan compensation scale. And then there's two more, inset bounds and inset application frame. So let's look at all three of these and see what they're going to do. Now the default, as I said, is scale. And this is really great for generic UIKit content.

Generally anything that isn't particularly performance sensitive. And what you'll get here, and the reasons that you might want to use this, are you get a full frame buffer of whatever size screen you're using. So for the purposes of all these examples we're about to walk through, we'll assume that you're connected to 720p displays. Now the same things we're talking about here would apply if you're connected to a 1080p display with an iPad 2 or even a 4:3 aspect ratio. Like 1024 by 768. So let's just assume 720p here stands for whatever mode you're in.

So the nice thing about the scale mode is that you still have a 720p frame buffer to fill. So from the perspective of your application, things are exactly the same whether you're overscanning or not. And you don't have to change any of your code. Then UIKit and CoreAnimation will make sure that when you draw your content, it just doesn't get clipped by scaling it down and keeping it centered. So there's no additional work on your part.

Of course the downside of this is that your content is being scaled. And this scaling of your content will introduce a performance penalty which will lower the quality of your content and reduce your frame rate that you can render at. So it's not ideal if you can spend a little bit of additional time working on your external display code. We can use one of these other modes and greatly improve the situation. So let's look at inset bounds. This is sort of the next step. It's sort of the next step up in terms of quality with minimal additional effort.

This is really good for high performance UIKit content. If you've got something that you really are concerned about the performance of and you want to get the best possible quality, you can switch to inset bounds. Now what this is going to do for you is give you a smaller screen bounds to draw into.

So if you're again at that 720p mode, you actually will not have the full 1280 by 720 pixels to fill. So this is going to allow you to fill fewer pixels. The result of that is that we won't have to scale anything down in order to avoid cropping your content. So there will be no scaling which will preserve that quality and give you the best quality rendering that you could get.

This will also give you that full performance because you'll avoid the additional scaling pass that the system would otherwise have to go through to scale your content down at every frame. Now the downside of this is that you have to do a little bit of extra work. And because you have this less than 720p frame buffer, you can't really scale your content down.

So you have to make sure that when you're drawing your content, you're able to draw it at slightly less standard resolutions. In the case of 720p, you'll probably have about 5 to 10% less area than a full 720p frame to fill. So you have to be able to draw at these sort of resolution independent modes. So let's go back to the first slide.

Now, if you can really spend a little more time and go all the way, the best you can get is inset application frame. Now, this is great for videos which are already compensating for overscan or for game content where you can just make sure that you don't draw anything that can't be clipped out to the edges. So the nice thing here is that you still get that full 720p frame buffer, so you can actually fill all of the pixels on the screen, and some of them around the edges will be clipped, but you have access to every pixel and can fill them all.

You'll get the full screen bounds so that you know that you can fill everything. And you'll get a smaller application frame. Now for those of you that are not familiar with the UIScreen API yet, there are two rects on UIScreen that are interesting. The first is the bounds, which is that full area of the screen. And the second is the application frame, which is usually used to indicate the area that you can safely draw your content in without it getting clipped.

On the main screen, the application frame is inset to avoid the status bar. On the external screen, when you have inset application frame chosen, the application frame will be inset to avoid clipping around the edges. This is, for those of you familiar with the term, the content safe area. And you can just, we'll tell you where it is so you can draw your extra content in there and know that it will not get clipped.

The nice thing here is that there's no scaling again because you're filling the full bounds, but you're allowing your content to get clipped and just insetting anything important into the application frame. So you'll get full performance and you'll get full quality. The downside, of course, is that you have to have a little bit more ability to position different parts of your content independent of one another because you may have to have those overlays positioned inset from the outer bounds.

So there was kind of a lot there. Let's just do a brief recap of all of those. In the case where you're connected to a display that is not overscanning, so we've detected that you're connected to this display and the display has told UIKit and the system that it is displaying every pixel and not overscanning it, no matter which mode you have selected, scale, inset bounds, or inset application frame, you will still have the full 720p buffer and all of them will get displayed on screen. So which mode you pick only has an effect when we detect that the connected display is actually overscanning.

Once we've detected that, in the scale mode then, of course, you'll fill the full 720p buffer and it will be scaled down. In the inset bounds mode, you'll fill fewer than 720p. And in the inset application frame case, you'll fill 720p but draw important things in that application frame.

So now we've got all of our content, and we've tested on one display, and everything's working great. And maybe we haven't tried on a whole lot of different connections yet and a whole lot of different displays. Well, there's a number of different ways that your users may be connecting their devices to their TVs. For instance, there's a variety of different cables they might connect with. There's a variety of iOS devices they may be using in the first place. And there are many different TVs they may be connected to.

And when you have all these different combinations of things, there's a variety of different screen modes that you may have available to you. Now we're going to get into a large number of things here, and we're going to talk about a whole lot of different modes you might see. But just at a high level, keep in mind that if you test a 4 by 3 mode and a 16 by 9 mode, you've got yourself pretty well covered.

So let's say that we've got this combination right here. We've got an iPhone 4 with a digital AV adapter, and we've got one TV that we're testing with. When we're using that, maybe with doing nothing else, we're getting 720p output as our default mode. But maybe our user is actually using an iPad 2. So instead of that, they're getting 1080p.

They may also be using a different cable or connected to a different screen, so you might instead see 1024x768, 800x600, 640x480, or possibly even if they're using some of the older analog cables, NTSC and PAL. Now, as I mentioned, there's a lot of stuff here, but don't worry too much. They're all going to be either 4x3 or 16x9, so if you've tested with both, you're pretty well covered.

Even the NTSC and PAL resolutions, which, as you maybe or maybe don't know, are not square pixel, are actually fixed in UIKit to be square pixel display resolutions. So when you get an NTSC or PAL display connected, as long as you're using a UI window and putting it on that UI screen, you'll either have a 640x480 bounds or an 854 or something like that by 480 bounds. They'll either be 4x3 or 16x9.

So if you test those two, you're in pretty good shape. Now, with any one combination of those devices, iPhone or iPad or iPod Touch, particular cable and particular display, you will have one preferred mode out of whatever large available list of modes there are. Now, there's no single preferred mode for every combination, but in any given combination, there will be one preferred mode.

So let's say that in the combination we've got, 720p is our preferred mode. Now, preferred mode is... It's actually an actual API thing. It's not just something I'm saying as we prefer this mode. If you look on the UI screen API, there's actually a property called preferred mode.

And that may be 720p. It may be 1080p. It may be 1024x768. Again, it depends on what combination of device, cable, and screen you're on. But there will always be one that's preferred for that given combination. Now, the best advice that I can give you about this is, to the best of your ability, use the preferred mode.

There's a variety of reasons for that. The preferred mode, first of all, is chosen automatically. So if for no other reason, you can be lazy and don't have to do anything and you're gonna get it. But there's actually real good reasons too. So there's no hardware mode switch if you stick with the preferred mode. When you tell a connected display to change modes, this can be a lengthy operation, possibly even taking multiple seconds. And there's often not real nice for your users flashes of perhaps white or different colors on the screen while the mode switch is happening.

So sticking with the preferred mode will avoid that hardware mode switch, avoid the screen flashes, and give you a seamless transition between mirroring and second display. Mirroring, the system mirroring mode, will always be using the preferred mode. So if when you switch from mirroring to second display, you stick with the preferred mode, you can be sure that there will be no display mode change. change. This will also give you faster launch times or faster application transitions from mirroring to second display.

A couple other things that are nice about this is that it gives you the best aspect ratio. So if you're on a 16 by 9 display, your preferred mode will be a 16 by 9 mode. And it gives you the best quality. All of these LCD TVs that are out there, they have a native resolution. And very often, the native resolution, when available, is the preferred mode. So if you're using the preferred mode, you're probably also avoiding an extra pass through the TV scaler.

So you're preserving the best possible quality for your users as well. All right. So if I haven't hammered that in enough, there may actually be a case where you might want to change modes. So why would you do that? Well, The best example I can give of why you might want to do this is if you're writing a high-performance game and you find that you simply can't render enough pixels fast enough to fill a large mode.

So maybe you just can't get your performance, your frame rate up high enough at 1080p to stick with 1080p and you really just want to always choose 720p instead when it's available. So there's three ways that you can address this problem. The first is ignore that and do more work and optimize your drawing code until you get to an acceptable frame rate. And so of course, I strongly recommend that.

But, you know, the benefit there is that you'll still get to render at full resolution, so you're going to get the best quality. This will, of course, also give you that optimal quality and avoiding the mode switch. The downside is that, well, it's more work. You have to do all this extra performance tuning, which hopefully you're doing anyway, so maybe you've already done it all and you just still can't get there. And you're still possibly ending up with lower performance on these high-resolution displays.

So, if that doesn't work, next option is to use the preferred mode but render at a lower resolution. Now, what I mean by that is, say you're drawing OpenGL content. Create your CA OpenGL layer at 720p size, even if you're using a 1080p mode, and then apply a scale transform to the layer to scale that smaller render buffer up to fill the full 1080p. This allows you to render fewer numbers of pixels and just adds that final scaling pass to scale the rendered pixels up to fill the entire frame.

This way, you'll stick with the benefits of the preferred mode. You won't have that hardware mode switch, and you'll still be avoiding the TV scalers. You'll be using the built-in high-quality scalers on the iOS device. But the disadvantage is that you're going to be using more memory than is necessary. You'll be allocating 1080p frame buffers, even though you're really only trying to display 720p worth of content.

So if that's not acceptable, your final option then, of course, is to actually change the mode. And there is a list of available modes on the UI screen object. You can look through them, find one you want, and set it as the current mode. The benefit will be that you can optimize your performance for a known good mode.

So if you can't support 1080p, you can optimize for 720p and really get a great experience. The downside, of course, then, is that you're going to end up with those hardware mode switches, and you're going to require a pass through the TV scaler because changing the mode will require that the TV scale your content down, or up, rather, before it's displayed.

So with that in mind, though, the last and most important thing about modes is just don't ask the user what mode to use. This isn't Mac OS X with the system preference pane launching and looking at 20 different available modes. Please, please, please don't put a big mode selection thing up there for your users to choose.

Test your app, see what you can support, optimize for a particular known good mode, and then pick what's best for your app. And make sure that you've really tested it and know that it's a great experience. Because leaving the experience up to your end user is really just going to result in bad reviews anyway, so might as well pick the right one for them.

So in summary, just three important things that I'd really like you to take away. The first is always set that status bar orientation if you're just mirroring, because otherwise you're going to end up sometimes with the content upside down on the TV. Second, adjust for overscan. I know that we'd like to think that overscan didn't exist anymore.

I know I would, but there it is. So think about it and use these new APIs to adjust for it. And finally, use that preferred mode if at all possible. So with that in mind, we're going to have Jacques come up and give us a demo of the AirPlay screen APIs. Jacques Pepin: Thanks, Jacques. Jacques Pepin: Okay.

Hi, everyone. My name is Jacques-Philippe Gasselin de Richebourg, and I'm from FireMint. I'm here to talk to you today about external display support in Real Racing 2 HD. I'm going to first take you back a little bit to our first racing game here, Real Racing. It was the first installment in our Pro Circuit Racing franchise. We got an Apple Design Award for this last year, and this was based on a fixed-function rendering pipeline.

We've since updated this quite substantially. We have Real Racing 2 HD, which I hope you've all played. This was specifically updated for the iPad 2, and we really try to use the advanced shader capabilities of the iPad 2 to do some fantastic effects. We have per-pixel lighting, anti-aliasing, and reflections on the cars, for example.

Also, the extra power of the iPad 2 allowed us to put about 10 to 20 times more geometry in than we had in the first installment. There's also a very exciting new feature of the iPad 2, of course, that you should all know about by now, which is we have external display support. I'd like to walk over here and show you a bit of a demo of this.

Okay, so we're starting the game up here in connected mode. So we're actually displaying the two screens at once. So I'm actually showing a bit of a user interface right now as the user has shown the video. It was asking me to tap to continue, which I did. And it's also important, I think Josh mentioned this earlier, that you are always presenting user interfaces on the iPad if you need the user to interact with the iPad.

So you can see the game showing there on the external display. I just want to show you some of the effects we were able to do here. You can see a per pixel lighting effect on the road, a per pixel lighting effect on the car as well, and of course reflection there of the scene that you're in.

I'm going to take you into a race here and show you some of the other great features of external display support. You could actually detach the cable right now and play this in standalone mode as well, and the game will switch. And you can plug it back in again. All right.

So... The two screens allow you to increase the level of immersion here. And I'm going to go inside the car here just to show you what I mean. Now you see here, I'm turning the iPad 2. You can see the arms and the steering wheel turn. This is actually increasing my level of immersion because, well, if I was facing that way normally, my eyeballs would be on the screen, right? So I'm not actually distracted by the iPad, and I can really feel like I'm driving.

Now we also display extra information about the race on the iPad 2, just in case you are, you know, maybe you're winning or something, and you have some time to look at where the opponents are. I'm going to bring that up as soon as I come around the corner here.

Okay, I'm going to show it to you now, what I see on the iPad. So you see this extra information here, this race indicator, so where I am. That's the orange dot, and we have the gray dots. That's the opponents. You can also see the position, where I am, the laps, the speed, and, of course, the lap time. So I'm going to bring it back in here. - Right here.

And, yeah, I guess I just want to leave it there. I'll just tell you that, of course, the API for this is very easy once you've got the wired connections going because the wireless mode, as you saw yesterday, works with exactly the same thing. So you just want to make sure that you are supporting the notifications.

That you're doing what Josh told you to do? OK, so some tips and tricks. So follow Josh's advice. Make sure that you register for the notifications and that you gracefully handle the display, connect, and disconnect events. Now AirPlay, as you see the new feature here, means that the user is going to connect and disconnect when your app isn't active. You want to make sure that you support that gracefully too.

With the two displays, you're going to have more screen real estate. So you want to make sure that you are drawing appropriate content on each screen. As you can see in our game, we were showing the action content up ahead where we were focusing on the user having the attention. And then we gave you some extra information on the actual iPad. So we gave the user more immersion and more information because we had more screen real estate.

Now an important part with a game like ours is we're actually using the iPad 2 as a steering wheel. So you may get some orientation notifications back and you need to make sure that you're handling them properly. The external display has to obviously always be facing the right way and you don't want to get the embarrassing situation of that turning as you turn the device.

Josh mentioned there are many different devices that you are going to connect to. You want to make sure that your game is adequately prepared for this. I said before that the API is really easy, and the hard part is making sure that your game is actually up to it. Test on a variety of displays. They're going to have different properties. They're going to have different aspect ratios, different resolutions, and you want to make sure that you're scaling your content to it.

There's also a secondary consideration there, which is the impact on performance or perhaps on the effects you had prepared in your game. You may have prepared an effect that's perfect for a certain screen resolution, let's say the iPad's internal display, and that effect might not work so well at 1080p perhaps. So you might want to make sure that you have a switchable effect in there that looks great on each and every configuration that the user might connect to. Okay, thank you very much.

Thanks, Jacques. Now, we've given you a lot of information about AirPlay today and also Second Display, but really encourage you to go install the SDK. Both your iPad and also your Apple TV. And explore AirPlay. Take a look at how AirPlay audio behaves, AirPlay video behaves, and AirPlay screen.

Make sure your app works well in this environment because you want to work well in this ecosystem. Additionally, we've given you a whole bunch of new stuff to help you make your apps even better, especially in iOS 5. Take advantage, like the overscan compensation that Josh mentioned, the MP Now Playing Center info center to really fill out your media playback apps.

And finally, really look at taking advantage of the second display. The ability to add a second screen, especially a large one, can really change the experience. For instance, people that know me know I'm not a game guy. Played Real Racing, you know, just on the iPad, and that was fine. That's nice. But then you hook it up to a TV, a big TV, and it just totally changes the experience.

It's really immersive, and it's great. And it's actually really fun to play, even for me. So think about that. Take the opportunity to look at your apps. There might be some unique things you can do in that space. The ability to use a second display to expand the experience of your app for your users.

So for more information, various people to contact, Alan Shafer and Eric Verschen. Customer-related sessions. The first two, Audio Development for Games and Exploring AV Foundation, have already occurred. If you didn't check them out here, check them out when you get back. Coming up later today, HTTP live streaming update.

There's also the Advanced HTML5 Media Controllers in Safari. Talks about the web pages and the various tags and how to use that. Audio Session Management on iOS. So you can find out all about audio sessions and get the right audio behavior from your apps. And if you're writing games or doing any kind of OpenGL, check out Best Practices for OpenGL ES apps for iOS. You're going to get a whole bunch of great tips there. OK, that's it. Thanks. Thanks for coming. Hope you enjoyed it.