[Applause]

Welcome to session 211, Multitasking Essentials for Media-Based Apps on iPad and iOS 9. My name is Stefan Hafeneger and today Jonathan and I will talk about adopting Picture in Picture and mastering shared resources. If you have already or plan to integrate a media-based app on iPad and iOS 9 this is the right session for you. This is the second in a series of three sessions about multitasking on iPad this year.

By now you should be familiar with all of the iPad multitasking features we have introduced this week. I want to remind you about the concept about primary versus secondary app. The primary app is always the one on the left unless the device is set up right to left which means it's the opposite. In the first session we showed you how to get started with multitasking on iPad in iOS 9. In this session, I will show you how to adopt Picture in Picture.

When you are playing video on an application, in iOS 8 when you are playing video, and you switch to an application the video stops playing. With Picture in Picture in iOS 9 the video automatically animates into a floating window allowing you to use other apps on the iPad while the video keeps playing.

I'm going to say Picture in Picture a lot in this section so from now on I will refer to is as PiP. PiP is supported on all of our latest iPads, iPad Air 2, iPad Air, iPad mini 2 and 3. Now, you might be wondering should my app adopt Picture in Picture.

If your application provides an immersive video playing experience, then absolutely. If your application provides a kid's viewing experience like a game, then probably not. If you think your viewer was like to keep videos playing while playing with other applications. Let's take a look at the video player APIs in iOS 8.

The media player framework has been around since the first iOS SDK. Last year we delivered AVKit as a new high-level media framework, replacing parts of the media framework. AVFoundation is available for clients that want to create their own video player with user interfaces. [inaudible] using their technologies. Let's talk about the media player for a moment.

In iOS 9 we are deprecating MP movie player controller and MP movie player view controller. In iOS 8 we are providing AVPlayerViewController as a replacement for both classes. And last year, in the Mastering Modern Media Playback session we showed you how easy it is to adopt AVKit instead. In this session we are focusing on AVKit, AVFoundation and WebKit.

All three media frameworks rely on AVPlayerLayer to display video content. Let me show you how PiP works. We start with an AVPlayerLayer in your application. In order for the video to keep playing when the application transitions to the background we need to reroute the video frames to a different process. For that, the system adds a new layer that can host your video content.

When PiP is started, a PiP window is created in this layer and positioned over an AVPlayerLayer. Now, the video frames can be rerouted. There is a seamless transition and the video will just keep playing. To be clear, we are not touching the AVPlayerLayer on your application. We just stopped delivering video frames. Now, assistant can position and size the PiP window however needed, and the video currently keeps playing even when your application transitions to the background.

So which of our modern video players support PiP? I'm happy to announce all of them. In the first part of the session I will show you how to adopt PiP in your applications using all three media frameworks and we will start with AVKit. AVKit's main API for video playback is AVPlayerViewController. A modern AV controller subclass that provides standard controls and behaviors.

In iOS 9 AV view controller smarts Picture in Picture and it's a button in the lower bar. When the user taps this button the video automatically animates into the PiP window. My colleague Felix will show you now what exactly you have to do in your AVPlayerController application for the PiP button to show up. [Applause]

FELIX HEIDRICH: Hello, everyone, I'm excited to show you the new Picture in Picture support of AVPlayerViewController. For this I have a demo application running in the simulator that shows a collection of videos using a collection view controller. I can tap on one of the videos to reveal detailed information about it.

I can also tap the big play button to bring up AVPlayerViewController and start video playback in full screen, but as you can see, the user interface does not yet contain a PiP button in the lower right corner. For the button to appear, two simple steps are required. Let me show them to you in Xcode.

The first step is to set the apps background modes. For this, I switch to the project settings, select the targets, and go to the capabilities tab. Here under background notes need to be enabled and the audio and air play check box needs to be selected. The second step is to set the app's audio category.

For this I go to the app delegate and first import AVFoundation. Next, an application did finish launching this option. I can set the audio session category by asking AV audio session for its shared instance and then setting the category to AV audio session category playback. With this code in place, let's launch the demo again.

I again select the video, start playback in full screen, and now in the lower right corner, the PiP button is present. When I tap the button, the video starts to play in Picture in Picture. The user interface in the PiP window contains three buttons, the central button toggles video playback and I will talk more about the other two buttons in a bit.

Also notice that there is a playback progress indicator present in the UI. Now, while the video is playing, I can navigate back to the collection of video, of videos and, for example, look at detailed information of a different video. I can also simulate pressing the home button and launch a different app all while the video continues to play.

On the left button in the PiP window should return video playback to the application. When I tap this button, the demo app is brought back to the foreground, but as you just saw, the PiP window is simply dismissed and that's because our PiP machinery does not know if the app is ready to handle video playback. Let me show new Xcode how to restore the user interface to handle video playback just as you're users will expect it. For this I switch to the collection view controller.

New in iOS 9, we are adding a delegate property to AVPlayerViewController and in this code I already made the collection view controller the delegate of AVPlayerViewController. To reinstall the user interface the collection view controller needs to implement a simple delegate message has a very descriptive name and is called player view controller, restore user interface for Picture in Picture stop with completion handler.

In this code, I ask the navigation controller for its top view controller, and then use the top view controller to present the player view controller that the delegate provided. The other thing in the code is to call the completion handler once the player view controller is presented on screen. By doing this our PiP machinery know that it can start animating the video back into place. So let's run the demo again.

I again tap on one of the videos, play it in full screen, switch to Picture in Picture and now when I return video playback to the app, AVPlayerViewController is presented and the video nicely animates back into place. Another thing I want to show you is what happens when I leave the application while the video is playing in full screen.

For example, if I now simulate pressing the home button, in this case, we automatically start Picture in Picture so that the video continues to play. At any time users may tap the right close button to dismiss the PiP window and that's Picture in Picture support of AVPlayerViewController. Thank you. [Applause]

So let's review what you just saw. You have to do two things. First, you have to enable the correct background mode otherwise your application won't be able to keep playing when it transitions to the background. You do this in Xcode [inaudible] capabilities. Audio and air play is the one you need to enable. In the future iOS 9 [inaudible] rename to audio play and Picture in Picture.

Second, you need to set a valid AV audio session category. A good place to do this is did finish launching with options. You ask for the shared audio session, and then set the category to AV audio session category playback. If the application also supports audio recording, you can alternatively use AV audio session play and record.

If the application supports background audio on air play iOS 8 you are already doing both things. If you think the application should not support Picture in Picture in iOS 9 we have a new option on AVPlayerViewController that allows you to disable it. Notice how Picture in Picture is automatically started when the user taps on the message notification.

Why does this happen? When the primary application transitions to the background, an AVPlayerViewerController is presented full screen, the system starts Picture in Picture on the user's behalf. If the video [inaudible] is playing, then Picture in Picture is possible. Keep in mind though that the user can turn off this behavior in settings in general multitasking persistent video overlay.

Felix showed in the demo that you automatically dismiss the AVPlayerViewController and the user starts Picture in Picture. We don't know the structure of the application, or where the user navigated to when it was active, so we need your help to restore the AV viewer controller by implementing this delegate method. In a simple case you could just call presentViewController animated, by using the [inaudible].

A few applications have more sophisticated structure than the demo application, you might have to do a few more things. In either case, don't forget to call the completionHandler when you are done with the restore. And that's AVKit and AVPlayerViewController. Next let me show you how to do Picture in Picture using AVFoundation.

As a client of AVFoundation you have an AVPlayerLayer and most likely player controls. I'm going to show you what you have to do and get from here to here in your applications. In iOS 9 we have a new class called AV Picture in Picture controller. It allows you to implement same PiP viewers I just showed you, but using your own user interface.

Before I create an instance of an AV Picture in Picture controller you should check whether Picture in Picture is supported on the current device. You create an AV Picture in Picture controller by providing an AVPlayerLayer. Think of AV Picture in Picture controller as an objects that allows to present the content of the AVPlayerLayer in the PiP window. Setting a delegate is optional but it is very likely that you will need it.

Next, you have the PiP button to use interface. So that the user can start Picture in Picture. You should only add the button if the device supports Picture in Picture though. There are situations where starting Picture in Picture is currently not possible, even though the device supports it in general. In this case, you should disable your PiP button. For that we have a property on the PiP controller that you can observe and then update the enable set of your PiP button.

The implementation of your PiP button action will look something like this. You first make sure that PiP is not already active. And then you can start Picture in Picture in the pipController. Do not call this method without user interaction. If you use this inappropriately the App Store team will reject submissions.

If the application provides additional content while PiP is active you may want to dismiss the video player viewer controller after PiP is started. In order to do so could implement the did start delegate method and then call dismissViewControllerAnimated on your controller. When you do this though make sure that you don't release your PiP controller. Because if it does happen, the PiP window will disappear. So make sure you structure your application accordingly.

AV Picture in Picture controller has a delegate method for restoring your view controller. In a simple case you could just call present view controller animated. Again, make sure you call the completionHandler when you are done with the restore. Because when you call the completionHandler it is assumed that the AVPlayerLayer used to initialize the PiP controller is back on screen. If this is not the case, we won't animate to video frame spec. Your AVPlayerLayer won't show on video frames while PiP is active so if the application provides a non-modal view player experience like this example you might want to update the user interface.

AV player viewer controller shows [inaudible] like this and hides viewer controls. This behavior is not required but we highly recommend this for consistency. In order to do this you could implement the start delegate method, hide your player controls and show your place holder art work. When PiP stops you do the opposite, you implement the did stop delegate method, hide your placeholder art work and show your player controls again. Very simple.

Finally if you want to allow PiP to automatically start when application transitions to background you have to be sure that the AVPlayerLayer covers the entire UIWindow. If this is the case, and the video is playing, and PiP is currently possible, the system automatically starts Picture in Picture on your user's behalf. And you will see all of this in action, we have brought a great sample project you should check out after the session. So you saw how easy it is to adopt Picture in Picture using AVFoundation as well. Finally, let me show you Picture in Picture with WebKit.

WebKit's main API for video play is WKWebView. This part of the session is for those among you using WebKit to present your [inaudible] web technologies. In iOS 9 WKWebView supports Picture in Picture for HTML5 video if the application is set up the same way as for the other APIs. If your web content uses iOS default in the playback controls, PiP will just work. If you are a website developer interesting in incorporating a PiP button into your controls check out section 501 what's new for web developers in WebKit and Safari.

If you want to support background audio air play but don't want to allow Picture in Picture, we'll have a property in WKWebViewConfiguration in the future iOS 9 seed that allows you to do so. And that's WebKit. So you saw how easy and straight forward it is to adopt Picture in Picture in your applications for all three media frameworks. I can't wait to see PiP support in all of your iPad video player applications. Picture in Picture is a lot like background audio. If the application is not on screen but the user is enjoying your media.

The same rules apply to background audio and air play apply here as well. For instance, do not perform any task unrelated to playback and limit your memory usage to only these tasks when the application is in the background. Properly sharing resources is critical not only for PiP apps, but for all iPad applications. And to discuss this further let me welcome Jonathan on stage to tell you everything you need to know about this topic. Thank you. [Applause]

Thanks Stefan multitasking on an iPad allows you to use multiple apps onscreen at once. But hardware resources are still shared between all of the presented apps. I would like to explore with you some of the iOS media policies and updated best practices you can use to manage audio, video, and camera in your applications with multitasking on iPad.

The good news is that some of these best practices should already be familiar. And many apply to all iOS devices so it's easy to make your app shine everywhere. Now, before I dive into specific media resources, I'd like to take a few moments to discuss some of the roles your app may take on iPad.

In iOS 8 apps are presented full screen. And new in iOS 9 you can have up to three apps on screen at once with the new multitasking modes. These apps may be vying for some of the shared resources so to help manage that, the system categorizes your app into one of three roles.

Full screen apps are considered primary. And they continue to be so when a secondary app is presented in slide-over. The full screen app also remains primary when it's resized to fit next to a pinned app in split view. And a Picture in Picture application can be put into the background while the video remains visible to the user. In this case your app's considered background media. For certain shared resources these roles help define the system's policies and your app's capabilities. So let's dig into some of these resources. Starting with audio.

The audio system on iOS is well-suited for multiple applications. If your app currently using audio, you have configured the AV audio session API to state the nature of the audio in your application. The system uses this to figure out how your app's audio mixes with, docks or interrupts other app audio on the system. The good news is a properly configured AV audio session should just work with multitasking on iPad. To this end, never change your audio configuration based on how your app is presented on screen. It's all about the audio.

With that, I would like to go over some of the audio session best practices most relevant to multitasking on iPad. If you are not following these practices, your app may have worked okay on iOS 8 but showed issues on iOS 9. First off, only activate your audio session when audio is first needed. This is especially true if you have an application whose audio interrupts other apps such as music or video application using audio sessions playback category.

Never solely determine to activate your audio just because your app launched or came into the foreground. Wait until the user interacts with your audio feature before activating your session. For example, a music and video application would wait until the user taps the play button. Next, there is a huge category of apps that don't need to interrupt other audio.

For example, if you have a game or have other ancillary audio like sound effects, you should use the ambient audio category, this category provides appropriate behaviors for this kind of audio including mixing with other applications. This is critical for these kind of applications on, with multitasking on iPad, because your app may remain in the foreground while other apps come and go. And you don't want your game audio interrupted and having the user have to restart your app in order to get its audio back.

Lastly, some apps have secondary audio Soundtracks. An example of this is if you have a game with bold sound effects and a music Soundtrack that plays during the game. Now, if the user is already listening to music in another application, you would like to silence your secondary Soundtrack. So that the user it continue listening to their music while your sound effects still mix in.

We provided a great way in iOS 8 to know when to do this. Simply check AV audio sessions secondary Audio Should Be Silenced Hint and listen to its began and ended notifications. So these are some of the most important best practices for using audio on iPad, and as a bonus following these will insure a great experience on all iOS devices. If you would like to learn more about configuring your audio session, there is a bunch of great tips in last year's Core Audio talk and in our detailed programming guide.

Next up, video. And first off, a lot of video has audio associated with it. So if you have an app that provides movies or TV shows, you will want to properly configure your AV audio session. In that example, you will want to set audio session's category to play back. Next. If you support air playing video in the background or add the new PiP support or add the new PiP feature in your application, you will be considered a background media app when you are playing media in the background.

This means that you will have lower maximum memory limits so it's really important that you discard any unnecessary data such as view controllers, views, images, and other data caches that aren't necessary when your app is off screen. You will want to do this proactively. Don't wait for a memory warning to do this. You will also want to limit yourself to performing just the task necessary to perform playback in order to help share the CPU.

Now, many iOS video apps provide their content via HTTP live streaming. If you have a streaming video application, you will want to do two things. First, make sure that you are providing multiple variants in your stream including a smaller resolution, low bandwidth variant. And second, make sure you are annotating all of your variants with the resolution attribute in the master play list. Now, if you get your video from an external vendor, be sure to check with them to see if they are providing you with these diverse well-annotated variants. And this is important for three reasons.

On iPad and iOS 9 your video may not take up the entire screen. Providing well annotated variants allows iOS to choose the right-sized video for how it's currently presented. And when it's presented in Picture in Picture a smaller resolution low bandwidth variant lowers your memory footprint, and helps avoid terminating apps due to memory pressure. And lastly, following these practices help users avoid unnecessary data plan usage. And they will really appreciate that.

The last topic I would like to talk about today is camera. In iOS 9 only one app can use the camera at a time. In addition, for certain camera features they may only be available when one app is visible on screen. This means that camera availability can change at any time and your app's camera usage could be interrupted because the user is interacting with other app using multitasking.

For this reason, it's important to consider how people use your application and its camera features. If you have a camera centric application, you may want to consider being full screen only. Our camera app is a great example of this. If your users expect an experience like this, you can add the UI requires full screen key to your info P list. This will help let users get quick access to the camera inside your application and use the entire iPad as a view finder.

Now, if using the camera is not the primary feature of your application, you will likely want to adopt the multitasking enhancements on iPad. Our notes app is a great example of this. People are going to love the flexibility of using the new notes features inside of slide-over and split-view modes. However, this mean that's some of the app's camera features may not always be available. So let's take a look at how our camera and notes apps handle camera availability on iPad and how your apps can do the same.

When one app is on the screen, UI image picker can let you see a preview, take a photo, or do a video capture, just like today. And when multiple apps are on screen, you can still see a preview and take a photo, however, video capture will be disabled. So let's dig into this case.

Here we have full screen maps with your app presented in slide-over. The UI image picker is in photo mode so you can see a preview and the user can take a photo if they tap the shutter button. When the user swipes over to the video mode, they will see a message that says your app needs to be in full screen in order to take video. What does this mean for your application? First off, active video captures can be interrupted at any time.

An example of this is if you are recording video full screen and an app slides over a new app in the slide-over feature. Your video will be interrupted and it's important to note that this kind of interruption can happen to any video capture app regardless if you support the multitasking modes or full screen only.

Next, the UI image pickers start video capture method may fail because we are in a multitasking mode that doesn't support it. If you are not currently paying attention to this return value, now is the time to do so. So these are some of the things for UI image picker. Let's transition to AV capture session now and see how the API differs.

When one app is on the screen, you can still have full access to the camera. And because AV capture session is a flexible low level API, you can do more than just these three features listed here. And we have been amazed at some of the creative and powerful features you have added in your applications.

In order to ensure that we can offer you the resources necessary to provide the highest quality camera experience inside your application, an AV capture session using the camera will only be available when one app is visible on screen. This means that whenever slide-over, split view, or Picture in Picture is active, your camera's session will be interrupted. You should handle these interruptions and communicate to your users that the camera is currently unavailable. Let's see how you can do this. The first step is to listen to AV capture sessions interruption reason or excuse me, was interrupted notification.

And new in iOS 9, the notification's user info dictionary states a reason stating why your camera was interrupted. There is multiple reasons you could be interrupted but when it's due to multitasking on iPad you will be told that the video device is unavailable with multiple foreground apps. This is your opportunity to adjust your UI by disabling any capture buttons and also identifying that the camera is currently unavailable.

Now that you have handled these interruptions, what do you do when they end? When an interruption ends, AV capture session automatically resumes the session, there is no need to manually restart it. You can be notified of this by observing the interruption ended notification. And, again, this is your opportunity to restore your camera UI by re-enabling any of the buttons that you previously disabled.

Be sure to check out our updated AV cam sample code for examples on using these new interruption reasons inside your application. So in review, Picture in Picture is a great new feature for video playback applications. You can adopt it easily with any of our modern playback frameworks and remember that if you are using MediaPlayer framework for video playback to transition over to AVKit to get the benefit of PiP and other great features. Also be sure to follow best practices Stefan talked about to ensure a great experience inside your application. As an example, make sure to restore your UI so that the video animates right back into your app.

Multitasking on iPad provides the power to use multiple applications at once. However, this comes with a responsibility of sharing resources. Three things you can do are to properly configure your audio session, provide adaptive video streams and handle the new camera interruption reasons. For more information, get in touch with our evangelist and check out the forum. And also we provided a new document called adopting multitasking enhancements on iPad.

Now, sharing media resources is only part of the story for shared resources. Be sure to check out the optimizing your app for multitasking session happening next up in Presidio. There is also a great Friday talk on delivering performance on iOS. And if you missed it, the getting started session is a great place to learn about adopting adaptive UI in your apps. On behalf of Stefan, Felix and myself. It's been great to talk to you this afternoon. Thanks, and have a great rest of the week! [Applause]