Good morning and welcome to the session, Live Screen Broadcast with ReplayKit. My name is Alexander and I'm very excited to be here and talk to you about ReplayKit concepts and new features that we bring to ReplayKit this year. We've got lots to discuss so let's get started.

ReplayKit is a framework that allows you capture application screen audio and microphone content in real time or record to a video file that the users can later edit or share. And ReplayKit also supports live broadcast. For application generating content like games, ReplayKit provides tools that allow them to stream audio and visual content using [inaudible] and broadcast services.

And for the broadcast applications ReplayKit provides ability to receive that content captured from other applications or entire iOS and encode and stream to the servers directly from the same device. ReplayKit delivers its content in high quality with lowest latency and very low performance overhead and better usage. It also safeguards the user's privacy by asking for approval before any recording or broadcast begins, and also showing a prominent indicator whenever content is being captured.

In this session today I'm going to talk about the live broadcast functionality of ReplayKit and we're going to start with overview and talk about [inaudible] broadcast and iOS system broadcast, and then we are going to talk about the new API called System Broadcast Picker. Now, this is an API that allows you to initiate the broadcast directly from your broadcast application. After that we're going to discuss how you can actually implement the broadcast extensions and how that would work for you with the new API that [inaudible]. Also, we're going to talk about the protecting of the content of your applications from being captured.

So let's start with overview of the live broadcast. As I said, ReplayKit allows you to broadcast your app screen, audio and audio content to search a third-party broadcast service directly from your iOS or [inaudible], and on iOS you can also provide voice or video commentary using microphone or camera. And all this content is absolutely secure and only accessible to the servers that you're using.

So when you're streaming your game play to Mobcrush or YouTube, share your screen in WebEx video call. Use TeamViewer to work as customer support, or maybe stream your drawing app on a Facebook. All of that is powered with ReplayKit technology. Originally, with the ReplayKit live broadcast the user would be in the app and would use the app to start and stop the broadcast.

An app would do that by directly communicating to a ReplayKit API and ReplayKit itself would involve the broadcast extensions and begin providing it with visuals and audio content from that application, and the extension would encode the media and stream to the servers and this is what we call In-App Broadcast.

And what was new with ReplayKit 2 is that instead we have entire system in a pack being broadcasted. And how this works is the user would initiate that broadcast from within the Control Center. So this is where they start and stop the broadcast. And that begins a system wide ReplayKit 2 broadcast session and, thus, audio and video samples, again, go to the extension and upload it to the servers. And conceptually this is the difference between the two.

And just to talk about the In-App Broadcast in the beginning. In the In-App Broadcast this is your app or your game that is providing the content that is captured live as you play in a game and your game is calling the ReplayKit API to start and stop the broadcast. And it also presents as broadcast activity controller so that the user would pick the service.

And in line with that, the broadcast application is providing you the broadcast extension which ReplayKit is working on behalf of that game to sign users into the service and upload the broadcast content live to their servers. That was our original way of using ReplayKit Live Broadcast and we covered this before. And in case your application does need this fine [inaudible] control for starting and stopping the broadcast, for more information please find the session called Go Live with ReplayKit from a couple of years ago.

But really, what today's session is about is what we call iOS system broadcast. Now, this is, again, the broadcast of all your onscreen activity and all your sounds, and instead of being initiated from within your application and starting and stopping when the app is paused, for example, this session is started and stopped within Control Center, and this is systemwide and it runs continuously as the user moves from the home screen to the app or as they move from one app to another. And all of this is built into iOS 11 and above and this is what was introduced last year as ReplayKit 2.

So here's some game and you're playing a game and you want to start the live stream and initiate the broadcast. So you pull down the Control Center and say you touch into that screen recording button and that will bring up the picture for you to record your game play, or if you tap on one of the [inaudible] of broadcast providers, it will stream out to their service.

So once you have done that, you're back to your game and now you're streaming to that service. The broadcast extension receives the media samples captured from the screen and uploads the video stream to its back end and the viewers they have in their web browsers, or however they want to use that stream, are watching your broadcast on their own devices around the world.

This is how you start the iOS system broadcast. And now, what's interesting about this kind of broadcast again, is that, as I said, it continues regardless of you moving from one app to another. So I'm broadcasting from my home screen here and then my home screen is what is being broadcasted.

And then if I'm launching a different app, that app is going to be broadcasted now, and that's really the core thing now. So the broadcast itself is a systemwide experience and you can go back to the Control Center and stop it or you can tap on the status bar at the top and it will bring up the controls to stop the broadcast. And another thing is that if next I ran as a landscape, there's also a way to support that and you can move between landscape and portrait as you go from one app to another.

So this was an iOS system broadcast we enabled last year and the feedback and adoption we've seen since then was amazing. And our live broadcast API today is used in many apps in categories like communication, streaming, house, help desk, education and social. But at the same time we had a lot of feedback that people are having hard time finding this broadcast UI in the Control Center so we wanted to make it easy for you to edit right into your app. And this year our goal was to allow you integrate iOS system broadcast into your application, and we are very excited to announce System Broadcast Picker.

So let me go back to my diagram. So what we added now is sort of a hybrid approach to last two [inaudible] styles. And in this hybrid approach we edit up the ability for your application to also be initiator of iOS system broadcast. So your app now can do the same thing as what Control Center is doing; start the broadcast and then allow it to continue as you move from one app to another and then move to the home screen, and so on. So as the user starts a broadcast from your application, he can later go to the Control Center and stop it from there or vice versa; he could start from the Control Center and then go to your app and tap the button that stops.

So again, the System Broadcast Picker allows users to start the iOS system broadcast without leaving your application, and this is just a simple button in the view that we provide and this is a new feature we are enabling in iOS 12. Here is example of that. Here I have the sample app, Fox 2 and I edit app with button at the top.

So if I hit this Broadcast Picker button, that's going to bring up the same UI as you would have seen if you would go to the Control Center. So I pick out the broadcast provider and start my broadcast. Now I'm back in the app and now I'm just broadcasting to that provider and people around the world can watch my live stream.

So how is this done? Okay. There is a new class called RPSystemBroadcastPickerView and it's just a subclass of UI View and you can add it to your application using the interface builder as a view with a custom class, or you can do it programmatically and all you need to do is just run it in instance of RPSystemBroadcastPickerView and add it to your view hierarchy.

So this is a more simple way to use the broadcast picker, but I know that many of you in this room broadcast service developers. And finally, with this new API your users can start the broadcast directly from your application and there's no more need to include all those tutorials, how you can enable the screen recording and settings and how to find your service and that [inaudible] in the Control Center.

But, however, you would probably like the Picker to show the broadcast extension for just your service, and ReplayKit provides API that lets you do exactly that. And it is just a property of that view and what you need to do is just get the bundle identifier from your broadcast extension and assign to the property of the view called preferredExtension.

Here's a code example of how you could do this. You would probably like to set the property right after correcting the view, and the only difference from that example would be that you need to replace the string called com.your-app.broadcast. extension. It's a bundle idea of your extension. So here's a diagram, and I just want to emphasize the point about the picker view that you create in your app. So that view is just a shortcut that brings up a system UI of the broadcast picker, the same UI that you can access using the Control Center.

And the picker itself belongs to the system and your application does not own any part of the broadcast state and also cannot start or stop the system broadcast programmatically. So this is what you need to do to add the iOS system broadcast to your app. There's a new API and it is fairly simple and you could really be done adopting it before leaving this session.

And next we are going to walk through the process for the broadcast extension developers and discuss how you can actually implement the broadcast extension and how things would work for you if your app provides both the extension and the broadcast picker. So let me come back to that diagram. We are sort of moving to the broadcast extension site now and we want to show you the path assuming the system broadcast.

Let's start with talking about the broadcast application and the broadcast extension and what each of them needs to do during the system broadcast session. So you install the broadcast extension along with the application that contains it. So if you install Facebook, you also install Facebook broadcast extension. And broadcast extension and the application are separate binaries and each runs in its own process and those processes are invoked directly by the user and independently from each other. And typically you would use application to allow user sign in or sign up for your service. And now for the apps that will adopt the broadcast picker API, you can also let user to type in some message before posting this particular broadcast to his Facebook.

And the broadcast extension, that's all the actual work when everything required to start the broadcast is already available. So again, the extension does the work of taking samples edited by ReplayKit, the raw samples of the audio from the app, video of the screen and perhaps audio from the microphone and encoding starts and uploading to your service using whatever technology you prefer to use.

To help you get started developing the new broadcast services we provide the Xcode template and just add the extension to your project using the template and you'll be ready to begin. So now let's look how you can code all of this. So when you create an extension in your project using Xcode template, you'll get a SampleHandler class like this and this is where you add your code to handle different events during the broadcast. And also, you would implement this function that handles incoming audio and video samples.

This is how the life cycle of the broadcast usually looks like. So we have states called setup, initialized, started, processing and stopped. Over on the left there's this green square which corresponds to the state when your broadcast extension is not running yet and at this point your app could, for example, get the login credentials from the user and use a shared keychain to process credentials to the extension. Or again, here you could let the user to type in that message for the Facebook part and pass it to the extension like one way or another.

So once the user initiates a broadcast, using the Control Center or your application that adopted the broadcast picker, ReplayKit launches an extension process and creates an instance of your SampleHandler class. And at that point you can override the default initializer of SampleHandler and do something essential for your broadcast and not directly related to handling the media samples. So, for example, you could get login credentials from this keychain and establish a broadcast session with your servers.

So once the process is launched and the instance of your SampleHandler already exists, the extension is notified that ReplayKit will begin providing it with audio and video samples using the function called broadcastStarted, and in this function you'd probably like to create your media engine or do anything else that you need to start receiving the samples in real time and encoding them and uploading to your servers.

So once a broadcast is running now ReplayKit will provide raw audio and video samples and your extension needs to encode it and upload it to your video service using whatever technology you prefer to use. ReplayKit provides your extension three types of the samples; samples with video captured from the screen, samples with audio tapped from the application and samples with audio captured from the microphone, and your extension will encode all of this and upload to the service.

And all of those are going to the extension and are handled by the function called processSampleBuffer. This function receives a single CMSampleBuffer, as you see, and the type of the buffer. And what it needs to do is to encode and upload the media samples, and in this code example we show you how you can encode the video using the Video Toolbox. So we recommend using the Video Toolbox as it provides hardware accelerated encoding capabilities, and that is especially important for our use case because extensions have much lower memory limits compared to their applications.

So once you're encoding and streaming the video to the broadcast servers the viewers around the world can or could watch your broadcast live, but they need some way to find your broadcast. And say if a person likes playing and watching Angry Birds, there should be someplace on your website or in your app where he can find all the live broadcasts with that game.

And in order to implement this you need to add to your video stream some information about the game that is being broadcasted now, and for that ReplayKit provides API function called broadcastAnnotatedWith ApplicationInfo, and as soon as a player starts his game Angry Birds sends this example, ReplayKit notifies your extension that that has happened and provides a dictionary with details about the app that was started. And as an example, you can use the key defined by ReplayKit called RPApplicationInfoBundle IdentifierKey and get the bundle ID of that game and pass it as a metadata to your stream.

When the user stops the broadcast, one way or another, ReplayKit is using function called broadcastFinished to communicate to your extension that broadcast session is over and there will be no more samples delivered to the SampleHandler. And in this function you would probably like to finish uploading your buffered video and [inaudible] your media engine or anything else that you need to release.

So this is how the broadcast life cycle usually goes. And there's one more point that I would like to mention here. As we discussed, everything required to set up the broadcast should happen in the application and you would get the login credentials using the app or you could get the name for the broadcast in the application, and we have a state for this, this green square on the left side. However, this could be the case that when the broadcast has started, your extension is missing something to proceed as a broadcast. For example, the login process could have failed or something else is needed, and I want to show you how you can handle this case.

ReplayKit provides an API that allows extension to communicate to the user that broadcast cannot be started and provide a particular action required to fix the problem. And the way it works, the extension calls some API function and ReplayKit will terminate the session and present the user the alert that will include the failure reason provided by the extension, and in this example this error message is from Mobcrush and it says that user is not logged in.

So next is the user tabs at bottom go to the application, ReplayKit is launching your application and what you need to do there is handle this case in some way, present the UI by user will sign in or sign up, and the next time the broadcast will just work.

So we have this code sample for the broadcastStarted before, and here I want to show you how do you handle this case when user is not logged in. So there's a function called finishBroadcastWithError, and what you need to do is provide the user info dictionary with a failure reason and ReplayKit is going to use that string and allow that is shown after the session is stopped.

Okay. This is how you develop the broadcast extension, and as you see, the process is pretty straightforward. The application handles the broadcast setup and then communicates that info to the extension. And after that the session starts and you receive and encode the media samples. And also, you can get information about application on the screen so the viewers could find your broadcast.

And so just one more point I would like to discuss today. This is protecting content of your applications. So your app may display some content that you don't want to be captured in screen recording or live broadcasts. That applies to both audio and visual content. There's an API that you can use to tell if your content is being captured and that API is provided UIKit framework and I'm talking about the property of UIScreen called Captured. So you can register to receive notifications whenever the value of this property is changed and stop your audio playback or hide the visual content when recording starts.

There's also a special case that you need to take care of, which is Airplay Screen Mirroring, and I would like to show you how you could do this using this code example for the notification handler. And basically, in the case of a screen mirroring there's always more than one screen and this array called Screens is going to have more than one element, and even though that isCaptured is going to return yes during the screen mirroring session, we have this extra check that will allow your playback to continue.

So this is the Live Screen Broadcast and how it really works. And to summarize, ReplayKit provides high level API that allows your broadcast content of just one app or all your onscreen activity. In iOS 12 we provide you an API or a way to initiate the system broadcast directly from your broadcast application. And what you actually need to code is a broadcast extension that does all the actual work of encoding the video. And also, we just discussed the way to protect the content of your apps.

You can find more information on the website of our talk and we also will be having the ReplayKit labs later today at 3:00 p.m. in the Technology Lab 5, so if you have any questions or comments, please come. We'd love to hear from you. So thank you so much for being here with us today.

We are really excited of bringing you the new approach for live stream broadcast in iOS and looking forward to see test applications adopting it. So please go ahead, create your first broadcast extension using the Xcode template, or open your existing project and add the broadcast picker to your app and enable live stream broadcast from iOS to your platform today. Thank you.