Good afternoon. My name is Gregory Hughes and I'm really excited to have the opportunity to speak to you today about a super important and interesting topic. The accessibility of media and games. At Apple, we strive to surprise and delight all of our customers regardless of their abilities and disabilities.

We envision a world where users like Patrick are able to do what they love and love what they do, without being hindered by technology. We want to make sure that users with disabilities can experience the same surprise, delight, magic moments and most importantly fun when using our devices.

As part of that vision, your apps play a key role. You play such an important part in helping us to deliver an inclusive, accessible and fun platform for all of our users. During our conversations today, we're going to be talking about four categories of disabilities. Visual disabilities, which are those pertaining to sight. How much a user can see, in some cases, the users that can't see.

Motor disabilities, which are disabilities pertaining to how a user interacts physically with the device, whether they can hold the device at all, how to use a keyboard or mouse. Hearing disabilities, which are those pertaining to how much somebody can hear and in some cases the complete inability to hear.

And lastly, disabilities related to cognition. And these are displays that affect how somebody thinks when using your game. How much they can remember from the original instructions you gave them. Or what type of UI works best for them. So, while talking about these disabilities, today we're going to break everything down into three key sections. First, I'm going to present how to make your media and videos more accessible in your applications.

We'll then build on some of those topics and talk about how to make your games more accessible. How do you think about accessible design in gaming. And this is really interesting. As we talk about this, it's really easy to sit back and say, "My game can't be made accessible." And one thought experiment that we often go through is let's start from a point where we think it can be done. Let's assume that somebody that's blind can use my video editing app, or somebody with cerebral palsy can play my first-person shooter game.

The challenge and the fun part is then figuring out how that can be done. And so, try to keep that in mind as we're going through some of these specific cases and general cases for adding accessibility to your games. And then last, we're going to show a demo game that we made, that highlights some of these accessible design concepts. And we'll give you some ideas on how you can implement them in your own app.

So, first, the accessibility of media and video. And generally, the most straightforward one that people think about is the accessibility of images. And anyone that's worked in accessible design will always say, always add alt text. But it's not quite that simple in all cases. For example, if we take a look at this image on the screen.

You might describe this as "Woman sitting in a hammock playing a guitar." An equally good description might be "Person playing guitar with custom artwork drawn using a marker." Equally, you could describe this as "Image demonstrating the use of the camera's aperture to make the foreground blurry while keeping the subject in focus." So, the important part when trying to describe your images, trying to populate that alt text, is thinking about how this image is used in your application.

What's the context for the image and what's it conveying to the user? Once you can think about that, you can then create a really good, clear and concise description for users that might not be able to see the image. So, for images we always want developers to think about clear, concise descriptions that convey the same information that somebody, that a sighted user market get.

For audio, we're generally thinking about disabilities pertaining to hearing. How does somebody that can't hear, or has difficulty hearing perceive the audio in your application? For audio, there are generally two categories of audio that we see in apps. The first is spoken audio. And this could be something like a recording of the "I Have a Dream" speech. Or it could be a podcast that you play back in your app. But regardless, when dealing with spoken audio you want to think about in the context of your application how can you provide somebody that can't hear that spoken audio access to the same information.

So, this is generally some way of providing a transcript. This could be a popup that shows up in your app to show a transcript. In the case of many great podcasts, they have online transcripts that somebody who's deaf could go and read to get the same contextual information that is conveyed in the podcase. So, it really how and why you're using this spoken information for how you're going to adapt it to somebody that can't access it through hearing.

The next way that audio is used in apps are for sound effects or ways to convey information. So, for example, in a game, you might convey where an enemy is by audio on the left side or the right side of the user. And in these cases, you want to think about how can you provide that same information either through visual or haptic feedback to the user. So, again it's thinking about if somebody can't hear this audio, which is really easy to simulate, you can just mute your device, how can we convey the same information in our game or application to such a user.

So, for audio, we are always thinking about how to convey the same information, what is an alternative way of conveying that. The last part of media and video is thinking about video, which is kind of the combination of images and audio. So, if you're not familiar, video has three key tracks, sometimes called media groups. The first is a visual track. This is kind of the moving pictures that we see. The next is the audible track. This is the track that contains all the sound, the sound effects, the spoken dialogue.

And last is what's called the legible track. And this one most people are familiar with. I know most people are familiar with it because we intro'd this with a closed-captioned video. But you've probably also seen closed captions in the gym or the airport. And legible tracks can be more than just closed captions. They can also be subtitles for users that don't understand the language.

They could also be closed captions as I said, but another term for closed captions is SDH, or subtitles for the deaf and hard of hearing. Subtitles for the deaf and hard of hearing are different than normal subtitles and subtitles for translation. Subtitles for the deaf and hard of hearing can include lots more information about the content. So, they might include things like what song is playing.

Was there a loud crash in the other room, or in some cases, even the absence of sound. So, it will just say silence if that's important to understanding what's happening. So, again it's taking a step back and thinking about if I didn't have the ability to hear this video, what information needs to be conveyed to such a user.

And so, for the most part, this is, you're playing back videos, it's working with your content providers and making sure they're aware that you want a closed-captioned track created. And in some cases, a subtitle track. But it's always thinking about closed captions and thinking about users with hearing impairments and providing information.

Playing this back is really simple in your apps. If you're using AVKit, on iOS or macOS, you don't have to do anything. As long as your content creator put the track in the video, it will playback appropriately, based on the user's settings. If you're working a little bit lower level at AVFoundation, you will need to do a little bit of work to let a user select which captioning track they want to see. So, if you're the lower level as I said at the AVFoundation level, you will need to first get all of the media group for the legible tracks again, we said closed captioned subtitles are the legible tracks.

You'll then need to do a little bit of pruning, because this can be a really big list. And you want to exclude forced subtitles and auxiliary subtitles from that list. Then you' have a final list of all of the tracks that should be displayed to a user and you'll iterate over them and just add them to some kind of menuing, or some kind of other UI that you're displaying to a user.

If you're working even lower level than AVFoundation, if you're taking responsibility for drawing your own video, rendering each frame, playing back your audio, you'll also then be responsible for rendering captioning content. And depending on what content you're playing, there can also be legal regulations that you have to honor, system settings for the user has for how they want closed captions to be displayed.

In order to do this, we have a great framework called Media Accessibility. And this allows you to get all of those user options for things like foreground color, background color, font size, font choice and other things along those lines. So, that you can render captions the way they look everywhere else in the system. That's where most people stop, right? Most people are aware of closed captions, they work with their content creators. Make sure they add a captioning track.

And it's easy to overlook the audible track. It's easy to say, for somebody that can't see, there's already audio there, we don't need to do anything else. But if you think about watching your favorite movie, or a TV show that you watch regularly. If you couldn't see what's happening, yes, there's audio but in a lot of cases it doesn't convey everything that's happening.

So, there's a technology to solve that and it's called Audio Descriptions. And this is a secondary audio track the user can select that contains not only all the normal narration, not only all the normal speech and sound effects, but also narration for what's happening. Now most people have never experienced this. So, I want to play a short clip of that intro video, that includes the audio description track.

Playing them back, just like captions you will get for free, as long as you're using AVKit. Again, just like captions if you're working lower level than AVKit, at the AVFoundation level, you'll need to do a little bit of work, and almost the same work. First you'll need to get the media group for the audible tracks. Instead of pruning this time, we're going to look for just the track. We're still pruning but instead of excluding tracks this time, we're going to get just the primary audio tracks.

And we'll then iterate over those to add them to a menu or some other UI where the user can select them. So, for video we now have a complete story. We've thought about users who are blind, users who are deaf and how they interact with our content. And how they consume our videos.

So, we talked about images, make sure you add a clear and concise description to your images. Audio, make sure that you're adding an alternative way for users to access the audio. And video, make sure you're working with your content creators to create closed-captioned tracks, audio description tracks and that application will support the playback of these correctly. Now, I'd like to hand it over to my colleague Charlotte, who's going to build on some of these concepts and talk about how they can be applied to games and how you can make some extraordinary games that are accessible to users with disabilities. Thanks.

Thank you. Thanks. I'm Charlotte Hill and today I'm going to talk to you about accessible games. When you create a game, you want to make something that your users will get lost in. You don't want them to get pulled out of that experience because of difficulties interacting with your software.

Today I'm going to talk to you about accessible games and how to make a game that's fun to play for everyone. Earlier, we saw a video of Patrick and how he uses VoiceOver on his iPhone and his Mac. VoiceOver users like Patrick do a lot of things with their phones, including playing games. When you're developing your game, think about VoiceOver users like Patrick with different abilities and disabilities and how they'll be able to interact with your game.

Let's say I'm designing an app based on the Four in a Row game. This is a two-player game, where each player takes turns dropping colored tokens into slots on a vertical 2D gameboard. The goal is to get four of your tokens in a line and block the other player from doing so.

How can I design it to be accessible from the ground up? Today, we're going to talk about how I can make my game accessible thinking about four areas of accessibility; vision, motor, hearing, and cognition. These cover huge spectrum of abilities, but we'll focus on some key points in each category and how they relate to Four in a Row, and likely your game as well.

First, let's think about vision. In the context of our game, the key areas we'll focus on are color, motion, dynamic type, and VoiceOver support. One of the key visual elements of this game is the differentiation between the two player's tokens. Here, one player is green and the other is red. These colors can be really hard to distinguish for someone who is colorblind, or doesn't see color at all. Here's what the same colors may look like to someone who has a type of colorblindness.

Let's take a look at how some other apps differentiate without color in their games. In Causality, the characters are defined by the colors of the stripes of their helmets. So, take a look at their helmets, and then this is what it looks like in the colorblind mode. You can go back.

So, they've added these shapes that go along with the colors to help distinguish between the characters in their color colorblind friendly mode. We can build off that idea and use shapes as well as colors in our game. Now, let's talk about how we'll use motion in our game. And you may want to close your eyes for a second if you get motion sickness.

Okay, so this is the really cool animation I wanted to do in the background of my game. Not at all distracting. I think it will be a big hit. OK you can open your eyes. Some people experience motion sicknesses from these kinds of animations. Motion sickness happens when some of your senses tell you you're moving, but others don't. When you're using your phone, your eyes see motion, but you don't feel it in your muscles, or your inner ear. Motion sickness from computers is actually really common. In one study, 70% of adults reported motion sickness while playing a game.

In my Four in a Row game, I'll make another background with less motion to follow the reduce motion setting on iOS. This doesn't have to be a static image, just tighten up the physics of the motion so there is less motion overall. You can find out more about designing for reduce motion on the WebKit blog.

Another accessibility setting to design for is dynamic type. Dynamic type or larger text sizes for users with low vision lets users go beyond the standard text sizes to something that's easier for them to read. Your game should support the standard text sizes as well as an accessibility text sizes, which go up to accessibility extra, extra, extra-large. We'll want to test set out our game with some different text sizes to make sure our layouts accommodate them. You can learn more about making your app work with dynamic type at the Dynamic Type Session and Labs on Friday.

So, we've done a lot to make our game more accessible for sighted users with vision problems, but what about low vision or blind players. A lot of players may be playing your game using VoiceOver, the screen reader that comes built in on all your Apple devices and let's players who are blind hear everything that's on screen.

Here's an example of how VoiceOver works. On my phone, I'm going to swipe across the screen to move the different elements and hear VoiceOver read them to me. Then I'll tap on the screen and hear what's under my finger, and then I'll double tap to press an element, in this case open an app.

If you do have custom controls or images, you can use the accessibility API to finish making your app accessible. For example, in my game I have these tokens that are custom views. And I want VoiceOver to say the right thing when it lands on them. Let's go with player A and player B.

I can use these strings of the accessibility label property to label these elements. Now, when VoiceOver lands on a token it will say its name properly. And you can learn more about that in the What's New in Accessibility Talk that was just on the screen before that. It's going to be available online and they went into some more detail about using the accessibility API.

So, for vision we thought about three key points for our Four in Row game. First, we used shapes and colors. So, there's more than one type of indicator for players who are colorblind. We provided alternate experiences for reduced motion and dynamic type. And we made sure all of our elements were accessible and labeled for VoiceOver.

When you're designing a game for iOS or macOS you're usually thinking about players swiping on a screen or typing on a keyboard. However, your players may be playing your game using assistive technologies like Touch Accommodations, the Accessibility Keyboard, Alternative Inputs, and Switch Control. Some of your players might have difficulties with fine motor skills like from arthritis that make it difficult for them to accurately swipe and tap in your game. These players may use the Touch Accommodations and Accessibility Settings to make it easier for them to interact with their phone. However, it's still a good idea to think about how you can make it easier to swipe and tap in your game.

In my Four in a Row app, players tap on a column to drop in their token. I want to make sure these targets are large and clearly defined, so it's easy for users to hit them. The Human Interface Guidelines recommends a minimum area of 44 by 44 for topical elements.

If you're designing a game for macOS, players might be using the Accessibility Keyboard to play your game. Instead of typing on a physical keyboard, players use the trackpad, mouse or dwell control to select keys on screen. Your game should just work with the Accessibility Keyboard, but you may have to provide some options for keys.

If your game requires pressing and releasing multiple keys at the same time, offer an option for just pressing a single key instead, or allow a press and hold so players can use sticky keys. You can hear more about the Accessibility Keyboard in the What's New in Accessibility Talk.

Other players may be playing your game through alternative external inputs like braille displays, HID controllers, and external keywords in iOS. These are great options for players who have difficulties in motor skills, or repetitive strain injuries. Supporting keyboards for things like typing could be straightforward in your game, but you may have to do a little extra. For example, in Four in a Row, we needed to implement a selection mechanism for players to be able to select a column using a keyboard.

Other players might be playing your game using Switch Control, an assistive technology available on macOS, iOS and tvOS. Switch Control lets you control your device with one or more switches which are like buttons that can be pressed by a hand, a head, or another body part. When you're using a single switch, Switch Control scans all items on screen and then you can tap your switch when it lands on the item you want.

So, here Switch Control is on and it's scanning everything in Sushi Go, and then when it lands on my sushi, I tap my switch. And I select my sushi. So, the good news is that if you already made your game accessible using the accessibly API for VoiceOver, which is used by players who are blind, it should also work with Switch Control. That means that games like Sushi Go can be played with just a single button.

In my Four in a Row app, I added a cool erase gesture that clears the board when a user swipes in a zigzag. However, this could be really difficult for players who are using Switch Control or Touch Accommodations. So, I'll put an option on my settings to use buttons instead of these gestures, so Switch users can use that instead.

When designing for players with difficulties of motor skills, you want to make sure that your targets are clearly defined and easy to hit. Try to provide for different inputs in your game, like testing it out with Switch Control or HID controller. Provide options for custom gestures so your game is still playable with a single switch.

Players in my game might be deaf or hard of hearing and I want to make sure they're not missing out on any of the experience by providing captions and alert options. In my Four in a Row game, I want to have a single player mode where the AI speaks taunting remarks throughout the game to increase the competition. However, my players may be deaf or hard of hearing.

Grim Fandango is a complex adventure app with a lot of dialogue, but it lets users pick between just speech, just text, and speech with text. I'll use the techniques Greg described earlier to add audio to my game. Accessibility Settings let your players customize the environment to fit their needs.

In Grim Fandango, there's also different sliders for adjusting the different types of volume in the game. And this is a great option for players who are hard of hearing, because that way you can select what you want your music volume to be versus what you want the volume to be for any speech, for example.

Depending on your game, you might also want to offer sound effect options, so players with hearing loss in one range can select sound that's in a different range. In my Four in a Row game, I want to use sounds that fill the spectrum, so more people can hear them. I also want my sound effects to be really noticeable for all players. So, I'll combine them with screen flashes or haptics on supported devices.

Deaf or hard of hearing players will enjoy your game a lot more if you provide captions for any text. Depending on your sound effects, you'll want to try to use full spectrum sounds, or add audio options so players can select what sounds work best for them. Try using multiple ways of getting player's attention, like screen flashes or haptics. Some of my players might have cognitive and neurodevelopmental disorders like ADHD, autism, or dementia.

These cover a wide spectrum and so it will be really helpful to get feedback from my users about what challenges they encountered while playing Four in a Row. However, there are some key areas I can focus on while developing my game, such as options for audio and visuals, providing help and tutorials, supporting Guided Access, and offering different game modes.

When we looked at Grim Fandango, one great option was the combination of voice and text. This is a great option for players to comprehend better when there's a mixture of sound and visuals in the game. I'll make sure to provide an option like that in Four in a Row.

Another thing I want to provide in my game is easy access to the instructions. For players with attention or memory problems, it can be really difficult to remember instructions or rules they encountered when they first played the game. Solivad [assumed spelling] has a ton of versions of solitaire and it can be hard to remember all their nuances, but they all have a rules button available from the same screen where you play the game.

I'll do the same thing in Four in a Row, and add a button to the game playing screen, so players can get back to the rules. Some of my players will want to play my game through Guided Access. Guided Access offers players another way to experience your app. A parent or caretaker can set up restrictions on the game so the Guided Access user can just focus on what's really important to enjoy playing the game.

For example, in Four in a Row, I have this Auto Play button that's a little confusing, it lets the phone play against itself and so players who stumble across it might not know what happened or how to get back to just playing the game. In Guided Access, a caretaker could put a restriction around that button so that way the Guided Access user can just enjoy the game without worrying about accidentally pressing it.

If your game has things that it would make sense to restrict, you can use custom restrictions in the accessibility API. These are used in setting up Guided Access in an app. And then when the player plays them, those things are hidden or locked, so they can't accidentally use them.

You may also want to provide different modes in your game. Real Racing has different options for drivers that lets players focus on one aspect of the game at a time. Depending on your game it might make sense to use different levels instead, but you want to make sure you're not preventing players from completing your game.

I want Four and a Row to be timed, so players really feel the pressure when making their move. However, a timed version could be a lot less fun for a lot of players, including those with cognitive disabilities. So, I'll add another option for an untimed mode in the game as well.

I've added leaderboards to Four in a Row, but I have to think about what ranking means to my players. I have diverse set of players playing my game in many different ways, and I don't want it to feel unfair. I'll create leaderboards for the different modes that you can play my game in so that way players in the untimed mode can compete against each other without worrying about what times the players in the timed mode are achieving.

Cognitive accessibility covers a wide range of users, and there's a lot you can do in your game. Use multiple methods for providing content like speech with captions. Make instructions and tutorials easy to access while playing the game. Support Guided Access so players can focus on just what's important to enjoy the game. And offer different game modes or levels, each with their own leaderboards, so players can compete fairly. OK, so now I want to show you some of the Accessibility Settings we did in our game.

So, I have my game running in the simulator right now with the Accessibility Inspector. And I found the Accessibility Inspector in Xcode under Developer Tools. And there's a lot you can do here. And I'm not really going to go into it because there's a really great talk in it last year.

But over here you can inspect your app. And so, it will tell you what different accessibility information it can get about the things in your app. Or you could run an audit which is really helpful, because it will just highlight some different accessibility bugs you might have in your game.

But, today we're going to look at the settings tab. And this is really cool, because it lets you test different accessibility settings without leaving your game. And I'm just doing this on the simulator, but up here if you have your phone plugged in, you can actually pick your phone too and so you could do it on device and that way you don't have to leave your game and like go into settings and go back. So, it's really handy. So, one that we did was reduce transparency. And for that we just did a simple black background to make the gameboard really stand out.

We also supported different font sizes for dynamic type. So, there's a slider here and I can just make the fonts bigger or smaller. And the next one is invert colors. And to know why we implemented it the way we did, you have to know a little bit about how it's used. So, invert colors is used by players who have difficulty reading dark text on a light background because the brightness of the background interferes with their vision. So, they'll use invert colors, to make that light text on a dark background instead.

Since our game is already like that, we used the accessibility API to tell it not to invert colors for this screen. So, when I do invert colors, nothing happens. But if I go to like settings, which is normally white, it's all black. And then if I uncheck it, it goes back to white. So, we have a lot of different settings here in this game that we added for accessibility reasons. So, let's check that out as well.

So, here's my Four in a Row app. I can just tap on a column to make a move. But more exciting than the game is the settings. And we have a lot here. So, first is volume. We added different sliders for the speech and the sound effects because people who are hard of hearing might want to emphasize the speech over the sound effects. So, we could turn down the sound effects.

We also have captions. And so, those are just going directly from the settings for your phone. And we have Differentiate Without Color. And if you noticed before, right now it's all circles. But when I turn on Differentiate Without Colors, we made it so now it's circles and triangles. So, it's easier to tell apart the different players.

So, we have a lot of other settings here, but the one I wanted to focus on was timing. We added a Turn Time slider which is great because that gives players who have trouble coming up their next move or difficulty moving that quickly some extra time to make their move.

But for now, I'm going to turn off the timed mode because I'm going to be testing this out with some assistive technologies and the combination me using them and talking about them is going to slow me down a little bit. So, oh, yeah and I'm already very close to losing. So, I'm glad I turned that off. Okay, so now I'm going to test it out with VoiceOver. And I have VoiceOver set up on my accessibility shortcut. So, I can just triple tap on home and turn on voice over.

But now, when I go into the gameboard, it's going by column instead of by cell, which it did originally, and so I was able to use accessibility API to change that experience to make it better. So now I can just press my switch and make my move. So, let me show you some other things that we did. Let me turn off Switch Control.

And right now, if you're noticing I have my screen setup to be a switch too. And that's an awesome option for testing, because that way you don't need to set up a switch. So, you're just, your phone works as a switch. Okay, so we also made it work with a keyboard.

And so, I mentioned earlier for that, you needed to do a selection mechanism, so that way when I hit the arrow keys, I can move by column, and then just press space to make my move. So, providing these options for these different assistive technologies really broadens into my audience. It means a lot more people will be able to play our game.

We were able to make a lot of improvements to our game by testing it out with different assistive technologies and accessibility settings. When you're doing this, it can be easy to fall into the trap of following the letter of the law, but not the spirit. As well as being accessible, you want your game to be fun to play for everyone. The accessibility API offers a lot of great tools for improving that experience.

Now, you may be thinking about your own app and how you can get started making it accessible. The three things you can do the minute you walk out of here are to audit your app, test it out with different accessibility settings, and most importantly, talk to assistive technology users. So, we already mentioned this talk a little bit, but it's really good. So, you can learn more about both auditing your app and testing it with accessibility settings and Auditing your Apps for Accessibility talk from last year.

It's really important to test out your game with assistive technologies, but you should also try to talk to the users who use them every day. They'll have a firsthand experience of what separates a good experience from a great experience. And they'll probably have a lot of ideas for how you can make your game more accessible. You can connect with accessibility users and developers at our get-together tonight. Designing your game for accessibility really means more people can enjoy it.

When you're designing for accessibility think about how accessibility is really a spectrum. So, you want to create these different great experiences so each player can find out their own perfect fit. At the end of the day, your game will be stronger and richer, because designing for accessibility is really designing for everyone.

You can find out more information on and the developer website. You can see this session again, and I think we have some notes and information there. There's a lot of other related sessions going on this week and there's more that you can see online from some previous years. Thank you and enjoy the rest the conference.