Thank you, hello everybody and welcome to Controlling Game Input for Apple TV. I'm JJ Cwik, Game Technology Software Engineer. Now when we introduce Apple TV back in September we talked about the 10 foot experience. We talked about it being a shared communal experience, one that's ideally suited to longer gameplay sessions.

And the fact of the matter is that people just love playing games in their living room and Apple TV is great for that. It gives you the tools you need to create really compelling game experiences. And I'm really happy to talk to you today about the Siri Remote, MFi game controllers and how to best integrate them into your game.

Now before I start spoiler alert, here's what's new in tvOS 10. As you heard on Monday, games with advanced game mechanics that cannot be supported by the Siri Remote can optionally include the requirement that they require game controllers. Now we've also included support for up to four MFi game controllers.

This is going to be great for multiplayer gaming. And as you also heard on Monday, we have an Apple TV Remote app that can be used to control your Apple TV and can be used as a game input device. Now I'll talk about all these throughout the course of the talk, so let's get started.

Now apps on tvOS typically take their input from the focus engine or they also take the inputs from touches and gestures. But there's also a third option available and that's to read inputs directly off of the Siri Remote and game controllers with the Game Controller framework. Now let me give you a quick overview of the Game Controller framework for those of you who may not be familiar with it. The Game Controller framework was initially created to expose third-party game controllers to your games. These compatible game controllers are what we call MFi controllers. Now the framework has since been updated to also include support for the Siri Remote.

The Game Controller framework is a standardized simple API that allows you to access all controllers and use the same API calls regardless of the controller vendors. It standardizes detection of controllers. It seamlessly handles controllers that connect and disconnect while your game is running. And it allows you to control input or rather to read inputs off of the controllers using polling or an event-driven model.

And it's also available on tvOS, iOS and macOS. And so even though this talk is geared towards tvOS a lot of the code and concepts still apply on iOS and macOS. So now that you have an understanding of the framework I'd like to talk about some specifics. There is a core class in the Game Controller framework called GCController. This represents physical controllers and it's the same class for the Siri Remote and MFi game controllers.

Now the first thing you're probably going to want to do in your app is to get a list of all the currently connected controllers. And to do this you use the controller's class method on GCController. This returns you a list of the currently connected controllers as an array of GCController instances or an empty array if none exist.

It's also common for controllers to connect and disconnect as your app is running and to be notified of this add observers for the GCCcontrollerDidConnect Notification and GCControllerDidDisconnect Notification. A great place to do this is in your application Did Finish Launching with Options. So now we know which controllers are connected and I'd like to shift focus and talk about the Siri Remote.

Now the Siri Remote is surface to your game as a game controller using the GCController class that I just mentioned. And in the Game Controller framework, there's this concept of profiles and profiles grouped together common functionality and the Siri Remote supports two profiles. The first is called GCMicroGamepad. This is what you use to access the touch surface and the buttons on the Siri Remote.

The second profile that the Siri Remote supports is called GCMotion and this is what you use to access the gyroscope and accelerometer on the Siri Remote. Now note that this is different than what you might be used to on iOS where motion comes through the CoreMotion framework. On tvOS motion comes through the Game Controller framework.

So let's look at each of these profiles in turn. First, the GCMicroGamepad. Now the touch surface on the Siri Remote is surface to your game as a virtual DPAD of sorts. And you can query its value using an analog representation as an XY coordinate pair or as a digital input basically as four buttons up, down, left and right.

The GCMicroGamepad profile also has an A button and this corresponds to clicking the remote. This is ideally suited for the primary action in your games, especially when the remote is being held in portrait orientation. And the profile also has an X button, this is useful as a secondary action in your games and it becomes really natural when the remote is tilted on its side and held in landscape orientation to have a thumb on the touch surface and another thumb on the X button. Now note that there's also a Menu button on this remote, I'll be talking about that momentarily. And the remainder of the buttons on the remote are reserved for system use.

Here's a code example to show how simple it is to read inputs off of the buttons. Here we see we have a controller that's already been connected and from that we can read the MicroGamepad profile. That stores the button A and button X properties and from there we just check the isPressed property to determine whether or not those buttons are pressed. Now we also support an event-driven model. So if you want to be notified when the buttons are pressed instead of having to poll you can do that.

These are called the pressedChangeHandler and this block of code that you supply is run whenever the button changes state. Notice of this is called twice for traditional button press once on button down and another time on button up. So those are the buttons. Let's look at the DPAD now. The DPAD as I mentioned can be read as four buttons up, down, left and right. And interestingly, has a second representation as two axes, xAxis and yAxis. So let's look at those.

Here you can see there's a DPAD property on the MicroGamepad and it has xAxis and yAxis properties. From there we read the value property which returns a float normalized between zero or sorry, normalized between negative 1 and positive 1. Now as you'd expect, we also have an event callback for that.

And note in this case our value change handler is used or we're supplying a value change handler on the DPAD itself not on the xAxis and yAxis. This is because we want to be notified whenever either of those two axes changes. Now there's some interesting aspects of the DPAD remote that I'd like to talk to you about now.

And the first one is called DPAD windowing. Now what this is is it really defines where the origin of your x and yAxis is for the DPAD. Now you might be wondering shouldn't this always be at the center and the answer is a lot of times, no not really. A lot of times a player doesn't concern themselves so much with the precise location that they're touching on the touch surface, but rather they just want to put their thumb down and move relative to where they initially touch down.

And that's what DPAD windowing allows you to do as a game developer is to tap into this relative motion. So initial touch downs on the touch surface establish an origin and then further movement from there has DPAD values reported to your game relative to that origin. Let me explain with an example.

So here's our touch surface and I place my thumb down right here. Notice it's not at the midpoint of the touch surface, but with DPAD windowing enabled this is where our origin for our x and yAxis is established. Also note there's an imaginary DPAD window placed outside of that centered about the origin. Now watch what happens as I move my thumb across the touch surface.

The values reported to your game are relative to this origin and in proportion to the size of the window. Now as I continue dragging my thumb across the touch surface it drags the window and the axis along with it. Now we have established a new origin and any further movement across the touch surface has the DPAD values reported to your game relative to this new origin. So that's DPAD windowing.

If you want to receive absolute DPAD values all the time you can opt into this with the reportsAbsoluteDpadValues property, set this to true and we always have the origin at the physical midpoint of the touch surface. Then as I place my thumb down and move it across the touch surface all the values reported are relative to the physical midpoint of the touch surface. So that's DPAD windowing.

Next up rotation. So the DPAD values surface to your game are in portrait orientation and this means X is going to the right and Y is going up. However, this might cause a problem if the player wants to play in landscape orientation. In this case, your game would need to transpose those values so that it's proper to be played in game. But then what happens if the user tilts the remote in the other direction, you would need to transpose those values in the opposite direction.

And how would you even do this, you'd either have to force the player to play in a particular orientation all the time or you'd have to start reading accelerometer data and track the orientation changes yourself. And this isn't something you want to be doing. So we've provided a property for this called allowsRotation. Now by default this property is set to false.

But if you set it to true, then the DPAD values reported to your game will be with respect to whatever orientation the remote is being held in. So when this is true and when the remote's in portrait orientation X is going to the right and Y is going up.

And in landscape left X is going to the right and Y is going up. And in landscape right you've guessed it X is going to the right and Y is going up. So if your game wants to allow players to play in landscape orientation set allows rotation to true so you don't have to worry about orientation changes or rotating DPAD values yourself.

So that's rotation. Next up motion. As I mentioned earlier, the second profile that the Siri Remote supports is called GCMotion. Now as the user moves the remote around the gravity in user acceleration vectors supplied by the GCMotion profile are updated. Note that these values are already filtered before they get to your game. So if you have a tvOS game and you're doing motion filtering we suggest you remove that so that you don't introduce any unnecessary lag into your game.

Also, this is fused motion data and what we mean by that is that the accelerometer and the gyroscope help to reinforce and correct each other. So just like on your iPhone the gravity vector from the accelerometer helps to correct the drift in the gyroscope and the gyroscope data helps to smooth the values coming from the accelerometer. They reinforce each other.

But really, vigorous motion can overwhelm that, so for this reason we recommend you avoid creating scenarios in your game that require the user to vigorously shake the remote. Because this causes data that's difficult correct until the sensors have had a chance to stabilize. Next up I want to talk about the Menu button. Now all controllers supported by tvOS in the Game Controller framework have a Menu button. It's the same Menu button with the same behavior on the Siri Remote and MFi game controllers.

Now in an app on tvOS when the Menu button is pressed the recommended behavior changes depending on the context. Sometimes pressing the Menu button will minimize your app and return you back to the Apple TV Home Screen. That's typically done in your game's main menu. At other times pressing the Menu button will go back one level in your apps menu hierarchy and this is typically done in submenus. Lastly, pressing the Menu button sometimes pauses or resumes your active gameplay.

Now nongame apps written entirely in UIKit get the first two behaviors largely for free without any additional work, but most games due to their custom UI have to do a little bit of extra work here to indicate to the system their intent. Let me explain. So in UIKit when the Menu button is pressed for UIKit apps it pops the child view controller off the stack. This returns you back to your previous level of your menus. Now you can do this successively with each Menu button press popping the child view controllers off the stack and then at some point you have one view controller, your root view controller remaining on the stack.

Now when there's only one view controller left on the stack the next Menu button press minimizes your app and bounces you back to the Apple TV Home Screen. Most games by comparison are architected differently, they typically have custom in-game UI that's not written in UIKit. And they also typically have scene transitions between the different portions of the game that also aren't written in UIKit.

And so for this reason they typically can get away with only having one view controller, the root view controller, for the entire duration of their game. But by the rule that we just saw because there's only one view controller in the stack at all points of your game by default pressing the Menu button will bounce you back to the Apple TV Home Screen. So you're going to need some way to indicate to the system when you want Menu button presses to return you to the Apple TV Home Screen and when you don't. Enter GCEventViewController.

This is a special view controller we created for single or rather for game controller games that have a single-view controller. So if your game is one of these you're going to want to set this as your root view controller. Now this view controller has a property called controllerUserInteractionEnabled and this is the key that you have for controlling when your app returns back to the home screen for Menu button presses. When this value is false this view controller effectively traps the Menu button presses and prevents them from going up the responder chain, effectively keeping you in your app.

Incidentally, this is also where your controller pause handler is called and I'll get to that momentarily. Now when this value is set to true Menu button events are allowed to proceed up the responder chain as normal, which allows your app to be minimized and takes you back to the Apple TV Home Screen.

So your job as a game developer is to manage the state of controllerUserInteractionEnabled as a user goes back and forth between different parts of your game. Let me show you with a block diagram. So here we are at the Apple TV Home Screen and we launch our app.

Some games have a splash screen or some other introductory sequence and many games will eventually land on an in-game main menu of sorts. And for both of these the user's expectation is that when they press the Menu button they will be bounced back to the Apple TV Home Screen immediately. Therefore, set controllerUserInteractionEnabled to true.

Now when the user progresses to other parts of the game, say active gameplay or in submenus, the user's expectations is that Menu button presses will not bounce you back to the Apple TV Home Screen. So in these cases set controllerUserInteractionEnabled to false. And if a user transitions back to the in-game main menu remember to set this back to true again.

So that covers our first behavior when we return back to the Apple TV Home Screen and not. And we still have the two behaviors remaining. We need to worry about going back in submenus and pausing and resuming active gameplay. Basically all the dark boxes on this diagram. And the good news is that those are both handled in the controllerPausedHandler.

The controllerPausedHandler is a block of code that you supply to the controller or for each controller rather and this is code that runs whenever the Menu button is pressed and you're not going to be bounced back to the Apple TV Home Screen. And it's a simple matter within this block of code to check if you're in a submenu go back to the previous level menu in your game. If you're in active gameplay toggle the pause state. And that's it now your game is set up to properly handle Menu button presses regardless of where you are in your game.

So now let's talk about MFi game controllers. Some games can really take advantage of the extended set of controls offered by wireless extended gamepads and these are optional accessories that players may have. Now by extended what we're referring to is the control layout and it's a layout that you're probably already very familiar with. It features a DPAD on the left, it features four face buttons A, B, X, Y in these positions and in this shape, the diamond pattern on the right.

On the front of the controller it features two thumb sticks and on top we have two shoulders and two triggers. Now all of these buttons that I've mentioned so far are pressure-sensitive and I'll cover that momentarily. We also have a Menu button, which I already discussed and we have four player indicator LEDs.

Now just like the Siri Remote supports the GCMicroGamepad extended game controllers support the GCExtendedGamepad profile. And you'll notice on the left here there's a table listing all the properties. Also notice the data types GCControllerDirectionPad and GCControllerButtonInput. These are the same data types you're used to on the Siri Remote.

Now one thing I'd like to point out is that the DPAD and the buttons and every single input on this table actually is pressure-sensitive. So even those buttons and inputs that are typically considered digital only in other controllers that you may have encountered like the DPAD and the face buttons and shoulder buttons they're all pressure-sensitive and so they can be read in an analog and in a digital sense.

So let's look at code that shows that analog digital duality. Here we have the A button and the first line is showing how to access the digital state of the button whether or not it's pressed. So if your game only concerns itself in a particular situation with whether or not a button is pressed this is what you want to use.

However, if you care about how hard a button is pressed, that's when you'd use the value property. And this is great to add more nuance control to your game. Maybe you're creating a sports game and you want to allow players to vary the speed of their passes from a soft gentle pass to a fast bullet pass and you can do that by reading how hard the button was pressed with the value property.

We also offer event callbacks as you'd expect. Now notice there's a pressedChangeHandler, which corresponds to the press property. This is for Boolean state changes or digital state changes. And there's a value changed property which corresponds to the value or sorry, a value change handler which corresponds with the value property. Now the value change handler is called many times in a typical button press as the button progresses down through its travel range and back up again.

And for those of you looking for guidance on whether to use polling or event callbacks, there's no hard and fast rules, but polling tends to be good for reading those inputs that change over a longer period of time. Like maybe the accelerator on a racecar game being tied to the trigger of a button or rather the trigger of the game controller.

And event callbacks tend to work really well when you're concerned about edge transitions of buttons. So maybe you have an adventure game and a player swings their sword and you want to activate the swing sword animation the moment the button is pressed. Event callbacks are great for that.

So by this point you probably have an understanding of the mechanics of connecting and reading inputs from the Siri Remote and game controllers and you're probably starting to think about which control types your game is going to support. Will it support just the Siri Remote or will it support the Siri Remote and MFi game controllers? Or as you heard on Monday, games that have advanced game mechanics that can't be supported by the Siri Remote can require the use of a game controller.

Now how you specify which controller types your game supports is in Xcode. In your target settings there's a capabilities tab and in there you'll find the game controllers capability. Enable this capability and select which types of controllers your game supports. Interacting with this GUI changes the keys and values in your info.plist file accordingly.

And this is important for app review because all games that link the Game Controller framework are going to or rather these keys are going to be looked for in app review by all games that link the Game Controller framework. And where this really matters is in the App Store because your users are going to be notified in the App Store if your game has game controllers as optional or game controllers as required. Additionally, the user may be warned if they tried to download a game controller required game when they haven't paired a game controller to that Apple TV before.

Now note that this is only a safeguard because it's very possible that your game that requires game controllers may still launch and find no game controllers connected. And this is a case that you need to account for and handle. It's very possible that maybe the user merely forgot to turn on their game controller. So in this situation notify the player that no game controllers were found and to connect a game controller.

Otherwise, the user may be wondering why your game is seemingly not controllable. Now there's one other place to concern yourself with notifying the user in this kind of manner and that's on controller disconnections. This is because the controller or rather the controllers available may pass your initial check, but then somewhere down the line while your app is running one of the game controllers could disconnect. And if this is the only game controller that was available then at this point your game can't proceed since it requires game controllers and again a place to notify the user. For more information, see Designing for tvOS, session 802 for more information on requiring game controllers.

So let's talk about something new now, the Apple TV Remote app. As you heard on Monday, this is a new app and it's available in beta form on developer.apple.com. This app allows you to control your Apple TV and it features a large touch surface in the middle of the screen and on-screen buttons that you'll find as well on the Siri Remote. And importantly for our discussion this can be used as a game input device.

And the way that's done on tvOS is it's surfaced by the Game Controller framework to your game in a very familiar manner. It's effectively an emulated Siri Remote. So it shows up as a GCController instance and it supports the GCMicroGamepad and GCMotion profiles just like the Siri Remote. In fact, it's indistinguishable in the Game Controller framework from the Siri Remote. So the code you write for the Siri Remote will work for both.

Now by default these remotes are coalesced, so if you have a Siri Remote and a TV Remote app they will appear as one game controller by default in your controllers array. This means that button inputs from both will be surface to your game as if they came from one controller and the same thing with DPAD inputs. Now for motion we didn't want the motion input form one overwriting and fighting with the motion input from the other, so we take motion from one controller or sorry, from one remote at a time.

And the remote we take it from is the remote that last received intentional user input either swipes on the touch surface or A or X button presses. Now if you want you can opt-in to separate these remotes and have them show up as distinct instances in your controllers array.

To do this, go to your info.plist file and add the Boolean key GCSupportsMultipleMicroGamepads, set this to yes and now the Siri Remote and Apple TV Remote apps will show up as distinct controller instances in your array. Now I want to talk to you about a special mode inside of the Apple TV Remote app called Game Controller mode. This is a mode that users can switch into and out of at will in any game that links the Game Controller framework.

Now this app is in landscape orientation and as such, the DPAD values are rotated by 90 degrees before they even get to your app. So your app doesn't have to do anything to take advantage of this mode. However, just note that if your game manually rotates DPAD values by 90 degrees you're effectively going to be doing a double rotation.

So our recommendation is if you have a game that wants to allow players to play in landscape orientation with the Siri Remote and Game Controller mode on the Apple TV Remote app set allowsRotation to true like we saw previously in the talk and allow us to handle the orientation and DPAD changes for you.

Now one more thing to note, the Apple TV Remote app's Game Controller mode separates the touch surface from the A button. This is different from the Siri Remote where clicking the touch surface it activates the A button. Now mechanically this doesn't provide any problems the user can still simultaneously enter input on both and your game code can still check for simultaneous input on both. But the key point is if you have game code that assumes that all A button presses will also activate the touch surface this won't work in the same way on the Game Controller mode for the Apple TV Remote app.

And lastly, I'd like to talk to you about multiple controllers. We get asked all the time how many controllers does Apple TV support. And the answer is one Siri Remote new for tvOS 10 up to four MFi game controllers and the Apple TV Remote app. Now we typically get asked this in the context of multiplayer gaming and with the addition of up to four MFi game controllers for tvOS 10 this is going to be great for multiplayer gaming.

But something that may not be initially obvious is that this is very important for single-player gaming as well. Let me explain. When your app launches you're not sure if you're going to find only one Siri Remote connected or maybe you're going to find a game controller connected or maybe you'll find one of each or maybe you'll find a Siri Remote, multiple game controllers, the TV Remote app. You get the picture right? So the key point here is on Apple TV it's very natural to receive inputs from multiple controllers even for single-player games. And importantly, you don't know ahead of time which controller the user is intending to use.

So you can't just assume that the controller's array index zero is the one that the player wants to use. So how should we handle this? Well the key realization here is that players can switch controllers at any point. Maybe they launched your app, traverse through your in-game menus and launch into gameplay all with the Siri Remote and then switch to the MFi game controller to actually start controlling the gameplay.

And games that allow players to switch freely and flexibly between different controllers without having to prompt the user with unnecessary confirmation dialogs or without requiring the user to exit gameplay and reenter gameplay with a different controller will create a really great seamless user experience. So how should we do this? Well one way to go about this is to treat input from all controllers as valid input for your single-player game.

So if you have a character that moves onscreen from the DPAD allow all controllers that are connected, allow their DPADs to control the character movement. Similarly with buttons, if your character or rather if your character jumps by pressing the A button allow all the A buttons on all the connected controllers to control the character. And if a controller connects in the middle of your gameplay you can silently start tracking that controller too and allow its inputs to control your character.

Now this is working really well we're able to give the player a system here in your game where they're able to switch back and forth between controllers. And this is effectively a manual coalescing strategy where we're taking the inputs from Siri remotes and game controllers and feeding them all into our single-player experience.

But there's one edge case here that I want to talk about and that's motion. So let me explain with an example. If you have a game that uses motion say to control the attitude of an airplane flying in the air with the Siri Remote you might also offload that controlling the airplane to a thumb stick if the player is playing with a game controller. And that's perfectly fine that works really well.

The key concern here though is you don't want your thumb stick that controls the airplane with the game controller to be overwritten by any motion data that might be coming from the Siri Remote when you're manually coalescing them. So in this case the code in your game that controls the attitude of the airplane should gate on whether it's taking that information from motion or from thumb sticks.

So then back to our example as I'm controlling my plane with my Siri Remote if I pick up my game controller any intentional user input on that game controller whether it's DPAD or thumb sticks or triggers or buttons will start ignoring the motion coming from the Siri Remote and use the right thumb stick to control the attitude of the airplane and vice versa.

If I then pick up my Siri Remote intentional user input, namely swipes across the touch surface or A or X button presses will at that point start listening to motion coming from the Siri Remote and switch to that. In using this we have a control scheme that's very flexible for players and allows them to switch as seamlessly as possible.

Now I'd like to end this section by saying that maybe these techniques that I've shown you here can be directly used by your game or maybe they're a good starting point and you can tweak them and customize them to suit your game just perfectly. But in any case, I really want to encourage you to go beyond the mere mechanics of controllers connecting and disconnecting and reading button values and really thoughtfully consider how you're going to integrate game controllers into your game. Because when you get this right in the mind of the player game controllers just melt away and it really allows them to become immersed by and enjoy the game experiences that you've worked so hard to create. And that's what games are all about right.

So we've talked about a lot today. To summarize, we talked about the Siri Remote how it supports the GCMicroGamepad profile in the GCMotion profile. I talked about DPAD windowing and allowsRotation. I also talked about the Menu button and how on Apple TV the expected behavior varies depending on the context of your app. And how you can use the GCEventViewController and control or pause handlers to control that behavior. I also talked about MFi game controllers that these use the GCExtendedGamepad profile that they introduce pressure sensitivity and that they can now be required as a controller for your game.

I also talked about the TV Remote app and how that's an emulated Siri Remote from the perspective of the Game Controller framework. And I talked about Game Controller mode of the Apple TV Remote app and design considerations so that your game works as intended for the Apple TV Remote app.

And lastly, I finished up by talking about multiple controllers and how it's great for multiplayer gaming, but it's equally as important to consider for single-player gaming. Namely, allowing your players to switch controllers as seamlessly as you can for your game. Here's the URL for this session. You can find the video, documentation and code samples here. This is session 607. Related sessions include Mastering UIKit on tvOS and Designing for tvOS. There's also a tvOS lab for more general questions. Thank you so much for your attention and have a wonderful rest of your show.