My name is James Dempsey. I'm a software engineer on the Cocoa Frameworks team, and I focus on application accessibility. So what are we going to be talking about today? Well, we're going to do a brief accessibility overview. Then we're going to take a look at some of the major use cases of application accessibility, both for users with disabilities, as well as how application accessibility is very useful to power users of your application and how you yourself can use application accessibility as part of your development process. Then we'll walk through the accessibility architecture on Mac OS X and spend the bulk of the time talking about how to make your application accessible. So let's get started.

So Apple has a very strong commitment to accessibility, and Mac OS X has numerous accessibility technologies and features built in. If you've ever visited the Universal Access Preference pane, you've come across these features. So there's things like screen zooming or flashing the screen for an alert, or sticky keys or mouse keys, a variety of features for users with visual impairments, hearing impairments, or motor skills impairments. Now, it just so happens that the vast percentage of these accessibility features are completely transparent to your application. You don't need to write any code to take advantage of them. So for the purposes of this talk, we're gonna kinda take that vast percentage and set it aside so we can focus in on those areas where your application needs to provide additional accessibility information work really well with the built-in accessibility technologies on Mac OS X.

So let's consider for a moment the case of a person with a visual disability using a graphical user interface. Now, there needs to be some alternate way for a user to interact with their computer. So they need to have the interface described. They need some way to navigate and interact without using a mouse. And that user also needs to find out when things happen, like when windows and sheets come and go.

Now, software that provides this alternate interface to a computer is commonly called a screen reader. Mac OS X, starting with Tiger, has included a built-in system level screen reader called VoiceOver. Now, the reason why I'm focusing on VoiceOver is that the way VoiceOver works to provide this alternate interface, it's really a collaborative effort between VoiceOver and all of the other applications on the system. Those applications provide accessibility information about themselves, and VoiceOver takes that information and provides a standard spoken interface to all of the applications on the system, or even a refreshable Braille display interface, in fact.

So another reason why VoiceOver, we're talking about VoiceOver, is that VoiceOver ships with Mac OS X. Every version of Tiger, every version of Leopard, in the future every version of Snow Leopard has VoiceOver built in. And so 16, 17 million copies of this software are already out there. and there's an ever-increasing number of users of your applications whose only way of interacting with their Mac is through VoiceOver. Now, we have one such user here with us today, and I'm very happy to introduce Dean Hudson, who is a QA engineer on the VoiceOver team, and Dean is going to give us a little walkthrough and tour of VoiceOver. Take it, Dean. Thank you.

I didn't know this was a performance. So, yeah, so voiceover. I am a visually impaired user, and like many visually impaired users who use the Macintosh, use voiceover to access applications. A couple things I wanted to do. One, show you kind of a fun application called iTunes that's been made accessible, and then show sort of a practical application, like say mail.

So before I do that, you might notice as I'm moving around that there is a black rectangle around the UI elements that I'm navigating over. And that's so that the sighted user has an idea of which element I'm hearing. And speaking of hearing, since Leopard, we've introduced a new voice from the speech teams, Alex, and it's actually tuned especially for screen readers that you can turn up the speech rate really fast and have it speak quite well and blind users understand it quite well. I'll do that a little bit later. Also, what I have showing on the screen is a text representation of what Alex is speaking. So that if you are following along and perhaps I have the rate really fast and you're like, what the heck, you can still read it. And for all you grade 2 Braille readers, there's also some Braille underneath that. So what I'm going to do now is just navigate a little bit in iTunes and have you sort of listen to what VoiceOver is reading to me and the types of information that I'm getting that James is kind of discussing a little bit. Dean selected.

SL selected. Let me go to iTunes first. iTunes. iTunes. Previous dim button. Got to love that bass, man. Play button. Play button. Play button. Next dim button. Idle image. Mute button. 100% volume horizontal slider. Full volume button. Text list button. Album list button. Cover flow button. Edit text blank. Now, as I was navigating, you were watching the rectangle, and you were seeing what was spoken in text. And you might think, boy, that's a lot of information. I mean, geez, you know, can't we just say button? But as a blind user, that information is very important to me.

If I land on something that just says button, I don't know if that's a mute button, a play button, it just says button. I land on something that just says edit, it could be a secure edit text field, could be a regular text field. That information, description, gives me a bit more information about what's going on and helps me use the application. more effectively. So I'm going to try and do something interesting, play a song, I think.

Sources, table one, interact, audiobooks, live radio, store, iTunes store. Now, I was just going to play a song, but Carl said that's too boring, so let's try and search for something on the store. Stop interacting, edit text, blank. Let's see if I can... S I G A E E D D E E E D space A N D space D N A A E A L E D sine d wield Clear search button Sine d, sine d wield D D E L A E A space D E N G G A E D space A N D space S E A L E D, signed and sealed Clear vertical splitter, iTunes store, dim songs, table, enter name, sign, sealed, delivered, I'm yours. Oh yeah, baby. I'm back and not ashamed to cry.

James? Oh, I'm sorry. Shoot. I forgot. Mail. I was getting too excited. Just having too much fun. Finder. Mail. Okay. So let's do something serious. Horizontal split messages table. Mail, you all know, it's a very, very popular application. So I'm just going to use this to send a message, something that users typically do. And again, pay attention to some of the feedback that I get. I'm also going to get out of this slow boat mode. We're gonna speed it up the way a blind user would use it. - 60%, 78%. Window, new message, to edit text, K, C, scram, m at apple.com. Subject, edit text, customize header, menu button, subject, edit text, insertion at beginning of text, L, U, N, C, H, customize header, Gmail, pop-up button, text.

L-E-T-O-P-S space G-O, period. - Let's see if that'll send. Maybe it should. - Gmail pop up button. Menu check mark, map account offline. Check mark, Gmail, map, map, let's go, text. Map account off. Menu check. Gmail. Gmail. Let's go. Text. Messages table. No selection. Messages table. That probably won't send. Horizontal splitter. Ah, it did. Sweet. All right, James.

Thank you very much, Dean. So Dean just showed us using VoiceOver. And you may be wondering-- I had mentioned that it's a collaborative process between VoiceOver, and in this case, the iTunes application, the mail application. Let's talk a little bit about that interaction between the two before we jump into what you need to do in your application to provide accessibility information. What's going on is it's a fairly straightforward client-server relationship. VoiceOver isn't doing any odd things within your process.

It lives in its own process space. And Mac OS X has a very well-defined public accessibility client API. That's shown in this diagram in green. The prefix we use on those APIs is AX. You'll see that a lot, AX for accessibility. I kid you not. So, Those APIs allow VoiceOver to get information from your application, all the applications on the system. So all the apps, in essence, are their own mini little accessibility information server. And the red sections up there represent the fact that that server infrastructure, the piece that can respond to accessibility requests, is built into the Cocoa framework, built into the Carbon framework. So your applications, if you have a Cocoa app that you've written, it's already doing some accessibility, even if you didn't do a darn thing. Now let's narrow it down a little bit to one application, the back and forth between two. You'll notice that VoiceOver has changed to assistive application. That AX API, that client API, is a public API. Therefore, not only does VoiceOver use it, but in fact there are other applications that can use that API. And we call that general class of applications assistive applications. And so an assistive application might want to talk to your Cocoa application. And to start off, one piece of that client API is -- takes the process identifier of some app and calls into that and says, give me back the application UI element for that item.

And on the client side, the UI of your application is represented in a very lightweight but rich way. It's represented by these little UI elements ref think of them as a little token or a little proxy referring back to the real object back into your application. And the top of this, the one that you get back and deal with or an assistive app, I should say, deals with, is the one for the application itself. Now these UI element refs, they're very powerful in themselves. They have a set of attributes. And a client such as VoiceOver is able to get the list of attributes, able to get the value, be able to find out if it's settable, set the value. It also has the ability to have actions. So things like if I have a UI element in an app like VoiceOver and I want to press a button and I have the UI element for that button, I can tell it to press. And that will trigger the appropriate action within the application. There are a few other things that we can ask of this element ref. In fact, we might want to know what UI element is within or at a particular spot in the screen. So hit testing is something that these clients can get. And finally, we might want to know, say, in a given application, let me know when a window shows up or when a sheet drops or when something goes away. So we can also register for notifications on that UI element rep. Now, again, this is all from the standpoint of somebody interacting with your application. But since it's a client-server relationship and your app ends up being essentially the server of this information, it's important to know what your clients are expecting. Now of all of the functionality built into one of these little UI elements that an assistive app can get at, attributes is really where the heart of it is. So let's take that first UI element and look at a couple of its attributes. A few that we have are the role. So we can tell what kind of UI element is this. Is this the application, a window, a button, what have you? But then a UI element can also have children.

And from this one root application element, we can build the entire tree of user interface in your application. So an application might have, let's say it's a window with a menu bar and one window. So it has two children. One of them is an AX menu bar. One of them is an AX window. The menu bar and the window both have children of their own. But then there's also an AX parent attribute that points back up to the parent. Now this is just a small subset of attributes that a UI element can have. Each of these have many more. I just wanted to highlight the importance of the role attribute and the children and parent attributes. Now, let's take an example, say, a checkbox. And what information through accessibility could an assistive app get from a checkbox? Well, it certainly can get its role. We also have something called a role description, which is a user -- it's a localized user presentable string. VoiceOver will typically read this allowed.

We also have a checkbox has no children underneath it. It's kind of a leaf in the tree. But it does have a parent. Also, what window it happens to be in. And this thing called a top level UI element. And this is-- sometimes we'll have things in a sheet, or a drawer, or a window. And the top level UI element is which one of those three things it happens to live in. Those items happen to be children of the window that they're attached to. We also have geometry, so the screen position and the screen size. This is what VoiceOver was using to draw that VoiceOver cursor that was following around what it was speaking. And then information about this control itself, like its value. Is it checked or not? Is it enabled or not? Is it the focused element? Does it have the keyboard focus? And finally, very important, a title, so that a voiceover user knows what the heck this checkbox is for.

And so with these UI element refs, attributes, and the other functionality, there's a great deal that an assistive app can do interacting with your application. And, of course, VoiceOver is just one of many assistive applications. In fact, on Mac OS X, we ship a number of features and technologies that use this client API. Now, another one is accessibility tools. We'll take a look at accessibility inspector in a bit, which uses it to give you a more raw, unprocessed look at the accessibility information that's being returned by your application. But also user level features, GUI scripting, which allows somebody to write Apple scripts or using the new scripting bridge, actually a number of scripting languages to basically control the user interface. Now advanced users who are building automated workflows love this feature. But for developers, this is extraordinarily useful for creating automated test suites that drive your UI the way a user would. So you can -- and we're actually going to have Carl is going to show you that in a little bit.

In addition, as a user-level feature and a developer-level feature, Automator has a feature called Watch Me Do that allows a power user to record and play back certain UI actions as part of an automated workflow. That same technology underneath, based on accessibility, is also used in instruments, which you heard about a lot yesterday, the performance tool, not only can you just drive it yourself, in instruments you can record a series of UI actions and then play them back again and again and again using various profiling tools to be certain that you are testing the exact same thing over and over again, which is especially handy when you have that one case that you need to do like these 17 steps in an exact, and then you try to debug it and you always forget step 13, and so you have to restart over. Very handy with instruments to be able to catch that thing by recording and playing back your actions. Now, all of these rely upon the accessibility in your application, that it gives out complete information and valid information. Now, to show you a little bit about how you can, your application can take advantage of this. using GUI scripting and instruments, I'd like to introduce Karl Schramm. He's an engineer on the VoiceOver team to walk us through automated testing and instruments. Thank you, James. Thank you.

Good morning, everybody. So in Dean's demo, we got to see how VoiceOver can interact with rich applications like iTunes and Mail because they're accessible. However, VoiceOver isn't the only assistive application. I'm going to show you two such tools that you can use in your development process as soon as you make your application accessible. So the first thing I'm going to show you-- let's get rid of that. Am I live? OK. Technical difficulties.

Great. And we're back. Thanks. Okay. So the first thing I'm going to show you is how you can make automated use cases using AppleScript GUI scripting. And the application we're going to be automating is Dicey. Dicey is available in the developer examples directory of your Snow Leopard installation, and it's this great single-player dice game. You play Dicey by rolling the dice three times and selecting the dice to maximize your score. So I'll just pick the threes and we'll try for a full house. Oh, we're short. So we'll just mark this up to chance. So that's dicey. Now for our script, all we're gonna do is click on the roll button and select the last die here.

Now, in order to do this, we're going to have to learn a little bit about Dicey. So we're going to take the accessibility inspector, and the accessibility inspector works by hit testing whatever is directly underneath the mouse. So we can see as we move over these dock items, we can see the information changing in the accessibility inspector window. And we can lock on to an element. I'm going to lock on to the roll button right here.

And this top part of the Accessibility Inspector window shows the accessibility hierarchy. We can see that this is an AX button, its parent is the AX window, and of course its parent is the Dicey application. Now below this, we have all the various attributes for this button. We can see that its title is role, it has a size and position, and a bunch of other stuff.

Now below this we have all the actions we can perform, and this button only has one action, and that's AXPress. We can actually perform it in the interaction window here. So we'll select AXPress, perform it, and we see the dice roll. We can also traverse the hierarchy by going up to its parent, and the highlight's going to change to show that we're now on the window, and we can return to the roll button by going to the children of the window and selecting the roll button. So now we know enough about the roll button, And let's look at these dice over here. And as I move over these and lock on to the last die, now the highlight is red and the die is red. So it's kind of hard to see that we have this highlighted. But if I perform an AX press, we can change the selection state. So we know we have the right die. So the die is accessorized as an AX checkbox. and its parent, the playing field, is an AX group. It's important to note that this is a custom NSView and we wouldn't know anything about this had it not been for the fact that Dicey's been made accessible.

So let's go up to the parent, the playing field, and we'll return again, look at its children. Now, the children are returned in the opposite order in which you see them, and that's just the way Dicey returns them. So the first child is the first checkbox, which is the last die, and we know we have it selected. Okay, so now we know enough information to make our script. And I have that script right here.

So the first thing we're gonna do is tell the Dicey application to activate. And then we're gonna talk to the system events application, and this will enable us to communicate to Dicey through the accessibility API. So we're gonna talk to the Dicey process. We're gonna talk to window one, which is Dicey's only window. And we'll click on the button with the title of role.

and we're gonna tell group one, which is the only group, to click on checkbox one, which is the last die. So let's clear the playing field before we start the script. Now, you also may have noticed that I made a couple calls here to the say command because we're gonna use the Alex synthesizer to annotate our demo. So here we go.

First, let's click on the roll button. Now let's click on the last die. Okay, we're off to a good start. So using this, we can make a script that'll play an entire game of Dicey. And I have the script right here, and we're gonna run it. Now, this script just plays the game randomly, an entire game of Dicey, and you can use this as a basis for other scripts. For instance, you can make it into a regression test by adding validation checks for the game state, or you can make it into a fuzz test by trying to perform some illegal actions or set some illegal values. And then once you've made this suite of tests, you can run them at the end of your build process on your daily builds, or use it as a validation before a major milestone shipped. So that's how you can make automated use cases using AppleScript GUI scripting.

Next thing I'll show you is Instruments. Now as James mentioned, Instruments UI recording feature utilizes the accessibility API to record and playback all your user actions, just like Automator's Watch Me Do feature. Now, because of this, the more accessible your application, the more robust your recordings can become. So here I've launched the Dicey Xcode project. Now, I could launch straight into Instruments. However, Instruments is just so nicely integrated into Xcode that I can launch it while I'm coding.

I go up to the Run menu of Xcode, select Start with Performance Tool, and we're going to go to the UI Recorder Instrument Template. And as soon as I click on this, Instruments is going to launch, and it's going to launch a debug version of Dicey, and it'll begin recording all our actions. So here we go. Here comes instruments, and there's Dicey, and we're off and recording. Just gonna do a few simple actions.

That's good enough. So we can see in the track view, all our actions have been recorded. Now, Instruments has a whole wealth of instruments that you can tune and tweak all sorts of aspects of your application. You can look at graphics performance, I/O usage, garbage collection. We're just gonna look at memory allocation and leaks. So we'll grab these two instruments, drop 'em in over here, And we'll kick off leaks in about 10 seconds.

And as soon as I hit the drive and record, it's gonna play back everything that we've recorded, and the object alloc in Leaks instruments will begin recording trace data. So, Instruments is taking control of the mouse, performing all our actions. We can see the object allocations in the detail view of Instruments. Now, Leaks is prompting me for my password.

And we'll stop it, and we can see our object allocations are okay, there's no spikes. However, we do have some leaks. So if we go to the detail view here and get some more information, we can get the stack trace for where this object was allocated, and it's in the Dicey project. And as soon as I double click on this, Dicey's gonna bring us back into Xcode where the object was allocated. And it says counted set right here. If we search for it, it's only used twice and it's never released. So we do have a leak.

So if we go back to instruments, we can save this file and pass it back to the developer, and they can use this file to reproduce the issue and fix the issue. Later on, you can use the same file to verify that the bug's been fixed. So as you can see, the accessibility API serves as a foundation not only for great user technologies like voiceover, but great developer technologies as well. Back to you, James. Thank you. Thanks a lot, Carl. So Carl has just showed us two ways that you can take advantage of the accessibility information that you put into your application. You can use it in your own development process.

Now, we've taken a look at sort of the client side of things, but by now you've got to be wondering, well, what do I, tell me what I need to do in my application. So for starters, every one of those UI elements that an assistive application uses, gets from that client API, maps back to some Cocoa object in your application. It's a Cocoa object that implements a 14-method protocol called NSAccessibility. And in fact, all of the heavy hitters in the Cocoa framework, NSApplication, window, views, controls, cells, they already implement this. And so there's already a pretty good default level of accessibility already built into the frameworks.

What happens when an assistive app makes this call? How does this all work? Well, the assistive application calls those client APIs. That comes into our app kit's accessibility infrastructure. And based on that UI element ref that was asked about, we map that back to the right Cocoa object, and we send it a method in the accessibility, the NS accessibility protocol. We get back the response, we package it up and send it on back. So again, fairly standard client-server sort of thing. It's just that in your application, you're always dealing with Cocoa objects. You're never dealing with those UI element refs that assistive apps deal with. You're just treating windows and views like you always do.

So a little more graphical representation of application accessibility. It really is a very lightweight representation of your UI. So we have your applications running. Maybe there's a window with a very boring button in it with no title. Shame on that button. But anyway, and that hierarchy is exposed in accessibility as an application with a window, and that window has a button in it. It's very clean and simple. And in fact, from the window on down, you'll see that it kind of maps to the view hierarchy. However, in Cocoa, we know that the view hierarchy is a little more complicated than that. The application has some windows. The window has a frame view, which has a content view. That view has a button in it. And then that button has a button cell, which is really doing most of the work of a button.

And so it's that button cell that is reporting itself as a button, but there's all this extra stuff in the way. We have a concept in accessibility called ignored elements. In this case, we don't report those items in between because to an accessibility client, they're not terribly important. They're just noise. We only need what the user's seeing, which is an app with a window and a button in it.

Now, let's take a brief look at this accessibility protocol. For all of the things that an assistive app can ask for, as you might imagine, there's a method to return that information. So for attributes, we can send back a list of the names. We can send back the value for a particular attribute, whether it's settable and setting a value. You'll also encounter something called parameterized attributes, and this is often a mystery to folks, for complicated elements like text elements, sometimes we need more than just give me your value. Sometimes to navigate through text, we need to find out things like the bounds of a particular range of text. Parameterized attributes are things that allow you to get more specific information, pass in a parameter when you get back a value. Typically you will not have to implement these methods, though. Actions, again, things like pressing something, incrementing or decrementing a slider, An object can return its list of action names, so things that it supports, a description that can be presented to the user, again, localized, and then finally be told to perform a particular action.

And then hit testing, as we saw in accessibility inspector moving the mouse around, accessibility inspector is able to get what UI element was under the mouse. This is done by a hit testing method. We haven't seen it in a demo, but we can also -- or an assistive app can also find out what item has the UI focus. And this is done by implementing the accessibility focused UI element to return the focused UI element. And finally, we saw that things might ignored, that's what this last item is. If it's yes, I am ignored, that doesn't mean it's hidden and all of its children are hidden. It's not like an NS view being hidden. It just means that it is skipped in the -- as the hierarchy is reported. Its children are reported as if they were the children of the parent of the ignored item.

And finally, notifications. So these are mostly automatic. NSWindow already sends them out when the windows come and go. Sheets do the same thing. But if you write your own custom control, especially if you are not going through object value, if you override that, you need to let accessibility know your value has changed. And if you have a custom control that within the control you're updating the UI focus, you need to let accessibility know that the focused UI element has changed. And so those are simple strings that you send out, and it's one function call in AppKit. The element is the thing reporting the notification, and the string is the string constant.

Now, that all said, let's talk a little bit about default accessibility. So one of the things that I spend a great deal of time doing is making sure that the built-in classes to AppKit, so all the Cocoa widgets, have great default accessibility built in. So that if you're using standard controls, there's a lot less work that you need to do to make your application accessible. And so we definitely recommend using standard controls whenever possible. However, we are unfortunately unable to read minds. We're working on it, but we have nothing to announce today.

So things like I have an image in the user interface, the framework doesn't have any idea what that image is for. Or you've written a custom view or a custom control, the framework has no idea what its purpose is, what its value should be reported as. And therefore, in those situations, we need your help.

So in past WWDCs, past conferences, is what we've typically done is we've written a sample application, Dicey would be one example, and we just do it without doing any accessibility. Then we walk you through how you would make it accessible. And then we release the source code. And, in fact, those pieces of source code, I highly recommend them. They're available at -- in your Snow Leopard Seed in developer examples, accessibility. So please take a look at them. But the one problem with sample code is that, well, it's sample code. Whoever wrote the sample code had a particular thing they wanted to show and they wrote the code specifically to demonstrate something in particular.

And so this year we wanted to show a real world example of making an application accessible. And so we thought, what if we went and grabbed a Mac application out of the wild and took Took a look at it in VoiceOver and then figured out what we might need to do to change it and then hopefully at the end of the session show that same user interface again but with its accessibility issues corrected. Now, we had a hunch that possibly a hot topic this week might be telephony. Just a slight hunch. And so a great Mac application for making phone calls is, of course, Skype.

So we called up the Skype folks a little over a week ago, and we've been working with them. And what we'd like to do is show you, one, the Skype that you can download today, and show you just one little piece of the UI, then talk you through how you would go about auditing just an application out of the wild, your own app, auditing its accessibility, and then addressing the issues that you find. And then at the end, we'll show you the results. So I'd like to bring up Dean again. who's going to walk us through just a small piece of Skype's API, or UI.

All right, so while I'm getting sound back on my machine. So visually impaired Mac users are no different than sighted users. We don't buy a program or a computer just to use iTunes or Mail. We really do wanna use other applications. And just like sighted users, you hear about things, you see things on the Internet. And especially when they've got the 30 day free trial and in the case of Skype, it's free. It's very exciting. You're like, man, let's download this, let's check it out. I'm gonna be able to use this program, make some free phone calls. And it's all gonna be good. So I've downloaded Skype. I suppose I should turn voiceover on.

Welcome to Macintosh. VoiceOver is running. System preferences. Window. Universal access. Toolbar. Skype. Skype. Welcome to Skype. - All right. So here's what I hear. - Skype. Welcome to Skype. Image. Image. - Wow, that was loud. - Welcome to Skype. Skype. - Not much there for me. Now you can see, of course, what's going on, that there is a login screen and password. But for our voiceover users, there's no way for me to know, unless I have someone sighted come along and say, ah, you're actually in the login screen and maybe try doing this. For a user who just downloaded it, that's all they would get. Now, James is going to magically make this work. James?

no magic involved. Just a little bit of elbow grease. For your Cocoa applications, you've shipped them. They're out there. VoiceOver users are downloading them. They're looking at them. They're interacting with them. if you've not thought about accessibility. It's not that you've necessarily -- you've not done anything wrong. You just haven't thought about accessibility and taken any steps to address accessibility issues. So what we're going to talk about now is, all right, let's take this login screen. Let's see what will we do to get it to be more accessible so that Dean would be incredibly happy with it.

So an accessibility audit. Sounds very fancy, but it's kind of simple. What you want to do is first use the accessibility tools we provide. They're located in your developer tools, developer applications. In the utilities, there's a folder of accessibility tools, accessibility inspector and verifier. You can also turn on voiceover, as you saw me do on Dean's machine, with an impromptu demo. to see what VoiceOver reports when you put it over your application.

So using those tools, the other thing to do is to focus on areas that commonly need additional accessibility. We noticed images was one area. We had that Skype logo that just said image, image, kind of useless. Also, custom controls and views tend to be hot spots where you need to do some accessibility work. It's good to start with the most commonly used areas of your application.

Those often tend to be the places where you've done the most custom work as well, but they're also the meat and potatoes of your app, so that's a good place to start. And proceed window by window. The thing to note about accessibility is certainly we would love every application, every just part of every application to be wonderfully accessible, Adding descriptions to images, that's a great start. Every release, make more and more of your application accessible if you can't do it all at once. Any work that you do is really, really a fantastic extension of your application. So now to walk us through an accessibility audit... is Carl.

Here's our Skype window. It exhibits three of the most common issues you're gonna encounter when accessorizing your application. And the first issue is that images need descriptions. So let's look at the Skype logo over here, and we'll lock onto it using Command + F7. And we can see that it's an AX image. It's rolled as AX image. However, when we look at the description, it's empty. Now, to an assistive application like VoiceOver, it really needs this description so it can describe it to users like Dean. So the next issue I'm gonna show you is related to the first. So just like images need descriptions, controls need title elements. So let's look at this combo box over here, and we'll lock onto it.

Now, there's nothing in any of these attributes that tells us that this is a username field. However, to sighted users, we can see that this Skype name label directly above it should apply. However, to assistive applications, it could be the Skype name element, or it could be this don't have a Skype name button that's right below the combo box. So to eliminate any ambiguity, you should provide a title element, an AX title UI element for all of your controls, and you can do this in Interface Builder while you're building your application.

So the final issue I'm gonna show you is that you need to make sure that your accessibility hierarchy is traversable. So let's go back to the Skype logo, and we'll lock onto it. So it's an AX image, and its parent is the Skype window. So let's go up to the window and look at its children.

and it only has seven children. The first three are the close, minimize, and zoom buttons. The next one is the welcome to Skype label. The one below it is an AX image, however, it's the white background image and not the logo. And then we have the Skype window title, and finally we have the grow area, which is the control in the lower right-hand corner of the window that used to resize the window. So the window knows nothing about that logo, and therefore, VoiceOver will know nothing about that logo because it traverses the hierarchy from the root down. We can also see this issue in the controls over here. So we'll lock back onto our combo box.

And we'll go up to the parent, which is an AX group. And as we recall from looking at the children of the window, there is no AX group as a child. So you can use Accessibility Inspector to find all these issues, like I just did, or you can use VoiceOver. And when you do find these issues and remedy these issues, your application will be much more usable by a VoiceOver user and much more scriptable to boot. Back to you, James.

Oh, give him a big round of applause. Great. Dean showed us kind of some of the ramifications of accessibility from his perspective. It's also interesting to note that if you were trying to write, say, an automated GUI script, you would not be able to in that log in panel because the hierarchy problem would prevent you from actually accessing the text fields which you probably would want to be using in your automated script. So how do we deal with these issues? So the found issues are, we came up with, images with no descriptions, elements, like those text fields, the combo box with no title, and we had a hierarchy issue, some parent-child issues. And who doesn't have parent-child issues?

So solutions. Two of the three actually have relatively easy solutions, which is excellent. The third one, it's not a lot of code, just conceptually you need to know why it's a problem and then what to do to fix it, which we're gonna tell you right here. So adding instance specific accessibility information. So certainly buttons and text fields have a set of attributes that they always have, like their role and their value and what have you. But we can also attach additional attributes specific to each instance. And that's what we'll do to give images descriptions. It's also what we'll do to associate controls with their titles.

And then for correcting parent-child reporting issues, there's two things that we might do there. We might override the NS accessibility protocol. So when we're asked for the value for our children, we might return a different set of children than the default. Or, as might be the case here, we might just refactor our view hierarchy in a different way such that the right information is reported automatically. So let's look at the first set first. So descriptions.

Descriptions are very important. We also have a-- convention for what those strings should be. They should be localized because a user is able to switch VoiceOver into other languages. They should not include the role itself. Don't say this is the login button. Just call it login. Because VoiceOver is going to use that role description we saw before and tack that on to the end. And then it will just sound stupid. Login button button.

The way I think about these is if this weren't an image for this button, if this were just a button with a word on it, what would that word say? That's usually about the length that you want for your description. And don't confuse this with the role description. One of them is just telling the role description is describing what kind of UI element this is, whereas the description is more information specific to this particular item.

Now, those titles, we want to know, especially when a voiceover user is tabbing through the fields, they don't want to tab to a field. They don't know what's around. They have to go to the left. Is there a title over here? No. Let me go back over. Is it on top? They just want to land on the field when you tab to it and have that described. And it's very easy to do that. We just connect the control to its title in interface builder.

And then one that we didn't see come up in the login page or the login window, but is very important also, are linked UI elements. So, in fact, you saw Dean use this in interface builder -- excuse me, in iTunes. When he was in the source list, he got to the iTunes store and then he was able to jump to another part of the UI. By linking related UI elements, it allows a voiceover user to move to related spots. Examples would be from the search field down to the search results and then back up. Or from a source list to the stuff on the -- that's being displayed for the selection as we see here. So how do we do all this magic?

I'll actually go in reverse order, because you can set this up in Interface Builder. You select a button, select some UI element. In the inspector, you want to go to the Identity tab, and under the Accessibility Identity section, you just type in the description. That's very difficult. Now, for attaching things or relating things, it's a control drag, just like any other connection that you make in Interface Builder, except that this time we drag from the text field to its title, and instead of picking an action or an outlet in the very same pop-up menu, we just pick how we want it related for accessibility. In this case, this is a title, so we would pick title from that list to make the connection. If we were linking, we'd pick link.

Now, sometimes you're not building your UI in Interface Builder. You have some things that happen to be programmatic. Sometimes there's things you can't access in Interface Builder, like the segments of a control, of a segmented control, in which case we can also set these attributes in code using accessibility set override value for attribute.

Now I wanted to talk a little bit about these overridden values, cuz sometimes they cause confusion. People see that method and they think, oh, I can just override everything with this. Now, we saw before how an attribute tends or is how a request for an attribute's value is responded to by a Cocoa app.

In comes the request. We map it to your Cocoa object. That accessibility protocol method is called. The return value is sent back. When you add an overridden attribute, either in Interface Builder or using that API, you doesn't happen quite the same way. The request comes in and AppKit's accessibility infrastructure, it's keeping track of that UI element ref that somebody has a hold of in their assistive app, and all of the overridden attributes associated with it. And when the request comes in, we say, oh, we have that overridden. Let me just send that back. And your Cocoa object never finds out about it. We don't want to bother you. We already know how to deal with Now, this leads to just a couple of caveats.

The first is you should only use overridden attributes for those instance attributes that I talked about. Don't try to change the role of your user interface element using an overridden attribute because your object doesn't know about that change. And the Cocoa frameworks often are doing different things depending on when it asks itself, what's my role?

It'll do different things. If you're changing things with overwritten attributes that are so central to that Cocoa object, but it doesn't know about it, just bad things ensue. The other thing to note about overwritten attributes is that they are static. We set a value, and that value is what is returned until another value happens to be set.

So that static portion can make dynamic image descriptions a little bit tougher. So this is just the example of, right, some chat client, it has a status image, right? It's away, then you're idle, then you're online. You don't want to, for this dynamic information providing image, you don't want to put a description that's called status, because that's not very helpful. Okay, it's the status, but I still don't know what it is. You want to provide the description of what the status currently is. Now, there's a few ways you could do that. You could, use an overridden attribute, and every time that you change the image, you could re-override the value with the new description. And that's not so bad.

However, if that happens to be in a table or an outline view, that gets very difficult because in tables and outline views, accessibility is keeping track of overridden attributes based on what row things are in. So if somebody resorts everything, then you have to redo all your descriptions. It's kind of a mess.

So we're thinking, well, what can we do to make this a little bit easier, or actually a lot easier? And we're thinking, well, strings don't have this problem, because a string essentially carries its own accessibility information with it. Its value is its value. You set a string and we just pull the value out of the string because it is a description.

So we thought, why don't we do the same thing with images? So what we've added in Snow Leopard and is in your seed is an accessibility description on NSImage. And the way it works is pretty simple. Take a localized string and you set the accessibility description on an NSImage. And now any time you set that image in any kit class that displays images, so image cells, button cells, the like, that description that you set on the image just comes on through. That makes it a great deal easier.

As I said earlier, we can't read your mind, but sometimes we can do a little bit better than we have been. And one thing that was introduced in Leopard were named system images. So in NSImage, you'll see a long list of common images that you can use in your applications and that we use in our system applications. We thought, well, now that we have the ability put a description on an image, why don't we provide a default description? And so if you use these name system images in your application, you will get a default description.

And then thinking a little bit more, okay, so image description, we have it down to one line of code. You can call on an image. could we get it down to zero lines of code? Does anybody want any guesses on whether we could? I think yes would be a yes. We added string file support. And because we noticed that what people do is they have a bunch of images in their app package, and they use image name to bring that in, and then they stick it in a button or stick it in some spot in the user interface.

I thought, okay, let's allow folks to add a specially named strings file, accessibility image descriptions. The keys in that strings file are the names of the images. The values are the descriptions you want to use. And if you pull in an image, set it in a user interface element, we'll automatically pick up that description.

So what's the search path here? We'll always use a programmatically set string first. Then our first call to accessibility description, we're going to see if you've provided one in the strings file. If not, we'll use the default framework description if it's present. And just a few notes on this new accessibility description on NSImage. The first is images with the same name are going to have the same description. So if you use that plus sign to add things in five different places in your application, you don't want to set it to add account because it will change all of those instances. In the case where you're using one image in multiple places, it still is the best practice to provide a specific overwritten attribute for each of those places. The strings file lookup is by name as you pass it into image named. So if you use an extension, put the extension in the strings file. And then if you are writing your own custom view and you want to return an accessibility -- or you're presenting an image, it's a very good idea, best practice, if somebody is asking for the accessibility description, to ask the image that you have set within you if it has a description and pass that along. So within your custom classes, also that description will be sent.

And then the last thing that we saw were parent-child relationship issues. And in an accessibility hierarchy in the tree, a parent can have multiple children And each child has one and only one parent. And the children, they have to match. You can't have one thing thinking somebody is its parent and this other thing thinks it's its child. It's like a bad episode of the Jerry Springer show at that case.

So why do we get into these parent-child issues? Not very Dr. Phil here, more accessibility related. If we take a look at what was going on in that Skype login panel, It was this issue, is that there is a, sometimes you have a control, an NS control, and NS controls often use a cell to do their drawing and event handling or tracking. We call those single-celled NS controls. And those controls are expecting to report their cell as their child. The control itself is ignored, and it's the child that reports its accessibility information.

The control isn't reporting its subviews at all. But in this case, we had an NSImageView. The image view itself is ignored as far as accessibility is concerned. It reports its image cell as a child, but that image cell doesn't know about its views, subviews. It doesn't know it should report them. And in fact, it really shouldn't report them because it's a little odd to have subviews in an image view.

And so there's a few solutions that we could -- or approaches we could take for a solution. The first would be to subclass NSImageView and return both the cell but also our subviews as children, in which case they would all show up. We'd put them all into one big happy array and send it back. Or the other thing we might do is refactor our UI a little bit. So we might make a custom view that displays an image. We're going to see in a moment, maybe instead of using an image view that displays a white background, maybe just a custom view that draws whites. It's ignored itself. It already reports its subviews.

Possibly refactoring is a better way. So we went through and saw the issues. Here are the ways we can solve those issues. Now let's take a look at... the result. Now, what we're about to show is just a very, very early peek at the accessibility directions of Skype for the Mac.

VoiceOver on Skype. Window. Skype. Skype name. Combo box. So we've got the Skype up here. And take two on this for me as the new Skype user. Welcome to Skype. Oh, that's nice. Skype name. Skype name. Combo box. Don't have a Skype name. Button. Password. Password. Secure edit text. Forgot your password. Button. Remember password. Uncheck checkbox. So you can hear it's a lot more accessible, said a lot more things. I, as a user, am getting a lot more information out of it than it was before. So great job. Also, I don't know if you've noticed, through all of my demos, I've been using the keyboard exclusively. So when you make your apps accessible or usable, I know it's easy to think about making a rich graphical interface and mouse-driven. but think about accessibility in those who may not be able to use a mouse, may have to use a keyboard, may even have to use some alternative input devices to control your application. The last thing that I wanted to say is, as James mentioned, voiceover is included on each computer shipping Leopard and upcoming Snow Leopard. So for you to check your accessibility really fast, it's just a matter of turning voiceover on and just navigating the UI. And that will give you a pretty good sense of what someone visually impaired would experience with your application. Thank you. Thanks. Thanks, Dean. Thank you. Actually, let's give Dean and Carl a big hand for the demos today. Thank you, guys. Thank you. So to summarize, application accessibility affects your users. It also can affect you and your development process.

So absolutely encourage you, exhort you, to audit your application for common accessibility needs. Images without descriptions. Take a little time, go through the images on your app. You can drop in one of these accessibility image description strings files. Don't even need to touch a line of code or even a nib file to get that done. Connecting controls to their labels. Again, a simple control drag within Interface Builder.

Looking at hierarchy issues, which could be a little more complex, as well as custom views and controls. And start making your applications accessible.. Now, for more information, our evangelist is Deric Horn. I highly recommend, if you are looking at making your application accessible, which I very much hope that you are, is to join the accessibility dev mailing list. You can go to lists.apple.com. You'll find it on the list there. But certainly all of us working on accessibility at Apple, we read that list, we respond on that list. It's a great place to get your questions answered and your answers questioned.

The documentation, there's the accessibility documentation. I would also recommend taking a look at the AppKit release notes. There are a number of minor changes we've made in AppKit that are in your Snow Leopard seed that we didn't talk about in the presentation. Finally, related sessions. On Thursday, we'll be doing a chunk of accessibility as part of polishing your Cocoa application. There you'll see if I have a custom view, what do to do to make it accessible. And then a few other sessions where some accessibility will come up, but I'm not allowed to say how.

Some labs. Today, 3:30 to 6:15 in Mac Lab A is the Accessibility Lab. Please bring your applications, your accessibility questions. I'll be there, Dean and Carl will be there, members of the VoiceOver team will be there, all ready to answer your accessibility questions. I also mentioned at the Cocoa Open Lab on Thursday at noon, I certainly will be there to answer other Cocoa questions, but also accessibility questions.