I'm Travis Brown, and I'm in the Technology Evangelism Department at Apple. And many of you might have recognized me from some of the graphics sessions. I am responsible for graphics and imaging at Apple, at least in working with developers and helping grow those technologies. But one of the other areas that I actually play a significant role in is accessibility.

I'm sort of the technologist who keeps track of what assistive technology needs in terms of operating systems requirements, and also how to -- and I bring that message directly into engineering to effect change. And one thing we've been doing is a team inside Apple has been working very, very hard on accessibility.

And so one thing I want to make really clear is that a lot of developers and a lot of users were worried when Mac OS X came out that Apple wasn't committed to doing anything with accessibility. But we really are, because we consider that it's a great idea to make our platform open and accessible to all users. And so in Jaguar, we spent quite a bit of time doing that. I'm doing just that.

So let's look at the agenda real quick. What we're going to do is we're going to give sort of a real quick overview of accessibility in case there are developers in the audience who are not familiar with issues relating to accessibility. So we'll bracket why accessibility is important. We'll talk a little bit, a really, really high level on Section 508 because that's the way many developers have been introduced to accessibility is through Section 508 requirements.

Then we're going to quickly begin to focus on accessibility in Jaguar. And we'll talk about our universal access features that you've seen demonstrated in a variety of sessions. I found that the zooming feature that's now built into Jaguar has been very, very popular. And it's been used in practically every session I've attended to focus in on something. So we'll also then continue on to what's the most important part of this session, which is the developer accessibility features.

This is for developers to use to make their applications accessible. And that'll culminate, obviously, in a lot of detail provided by Mike Engber, who will be on stage, who will present the Jaguar accessibility APIs. And so we're really proud of the fact that now we have an API dedicated to accessibility that's part of the operating system.

Why is accessibility important? If you're a developer and you've heard about this, why is it important for you to really take accessibility seriously in your applications? Well, an important and stunning fact is that 54 million people in the U.S. have some form of disability. And that's a huge number. And that's a pretty up-to-date number, considering we checked it with the U.S. Department of Education on Friday.

But again, that means a large number of customers, potential users of your software or your peripherals, could be affected by some form of disability. And therefore, they might not be able to access the full functionality that your software performs. So it's a real key point. These are mutual customers, disabled or not. Those are customers, and those are sales opportunities for your products, peripherals, and in our case, hardware.

Increasingly, this issue has become something that the state and local governments have taken up as a requirement. So when you sell product in through a federal, state, or local channel, frequently you're going to have to make some sort of declaration on how accessible your product is. This is also being done in education.

So it's a real significant requirement, and it's becoming an increasing requirement, particularly on the back of Section 508, which we'll talk about in a moment, that really dictates what minimum accessibility standards your application needs to follow to be considered accessible, and therefore, in many cases, able to be put on, you know, purchased by the federal government.

And then the last point is it's really the right thing to do, because before I started working with community with disabilities, I really didn't understand how much the computer, in the case of this Macintosh, can be fundamentally enabling and empowering to users with disabilities. It's something that's, you almost have to experience it, because we do many operations with our computer that is trivial, and we can, we don't really view them in a serious and necessary light.

But, for example, text-to-speech may be absolutely vital to a user who's unable to speak, and that might be a whole entree to communicating with the world at large. So it's a really good thing to do, and, you know, coupled with the fact that it's now a sales requirement, and, you know, the potential of the fact that there's a large number of customers that you may be missing when selling your software or hardware solutions, it's a really important thing to do.

Let's talk just really briefly about Section 508. Essentially, Section 508 is an extension of the Rehabilitation Act of, I believe, 1973. And so there's been laws regulating support for persons with disabilities for a long time in the books. It's just they never really reached in and sort of provided minimum standards for things like computers, operating systems, applications, websites, and general electronic devices.

And this was a problem because those areas have grown up so fast over the past 20 years, in many cases, little or no effort was put into making them accessible. So this is a significant problem that the functionality that we find every day in our computers, in many cases, is not available to a person with a disability.

So the federal government has weighed in and says, "We're going to have to make sure that we have a standard for things that it purchases. Those products need to be compliant and adhere it to some of the minimal rules laid down in Section 508." And the key thing is that this is good and bad.

One is this has caused accessibility to sort of percolate up to becoming an important issue for software developers and hardware developers to deal with. But it's also bad because Section 508 is being taken as the only standard. So that if you do Section 508, you're fully accessible and you're done.

That's not really the attitude you should take as you approach accessibility in your products. Because one thing is the spectrum of disabilities is quite large, and Section 508 doesn't necessarily address every single one of them. But it's important that it started and opened the dialogue and caused people to take the whole matter seriously.

Now, if you haven't already done this and you want to find out more about Section 508, what you need to do is go to the Access Board's website and download or take a look online at Section 508 requirements. They're pretty interesting. It's actually a fairly sizable document, but the parts that really apply to operating system software and peripherals is fairly short.

If I remember, the part that applies to operating systems is about a column and a half inside the document. And you look at it and you'll first go, hmm, this is not that bad. Then you'll look at it again and go, okay, this has some difficulties in some areas. Then you'll look at it again and you'll go, I'm going to have to redo my complete application. So the key thing is don't panic, because first place you want to look at is the commentary documents that are up there on the website.

In many cases, if you're having an issue with a particular part of Section 508 that might be mandating a certain functionality in your application, and you don't think that might not apply to your application or you don't know if it applies or how it applies to your application, often someone has asked that question before and those comments are going to be, you can find them up there on the website. So it's a good first start.

But the key point to really remember is that there's really three players in accessibility. There's going to be, obviously, you, the application writer. There's going to be the operating system vendor, which is Apple. And then also there's going to be something called assistive technology. That we'll actually get at in a moment. But essentially the burden of accessibility is shared by all three.

And the way Section 508 works is that you don't have to implement all of Section 508 if another part of, let's say, for example, the operating system supports that feature. And if your application is based on the operating system, it might pick that functionality up for free and therefore meet the requirement.

The same is true if assistive technology can actually perform that function as well on behalf of your application or the operating system. So you're not alone in this. So the key thing is when you look at Section 508, look at the commentary, understand what it's talking about. But then also the next questions to ask are first come to us, come to Apple and say, hey, how's my application impacted by this particular part of Section 508? And we might be able to reply, you're covered. The operating system, Jaguar now supports that.

So the key thing is take a look at it. It is the document to sort of get your feet wet with accessibility. And it also happens to be the document that a lot of the people use. It's not just the document that a lot of the federal, state, and local and educational requirements are being based on.

So let's actually talk more about accessibility in Jaguar. Obviously, we saw the keynote. We announced that we have support for a lot of accessibility features. And we really sort of factored the problem into two levels. One thing is we want the platform to be reasonably accessible right outside of the box. And so we've created a bunch of features which we call user-level accessibility. And those are what we call universal access features. That's sort of our umbrella term for them.

And then the more interesting part for today's discussion is really the developer-level accessibility, which is what you as a developer have to do inside your applications to either expose them to our accessibility APIs or work in conjunction with assistive technology. So we'll start first and talk a little bit about user-level accessibility. And that's universal access. Now, some of you may have seen this little icon.

It appeared actually in 10.1. We actually snuck some of this functionality in 10.1. And this little icon resides in the press panel. So if you're not familiar, you can go to the system preferences. You can click on them. Universal access. And that's where most of the features dealing with accessibility will live.

There's some exceptions. For example, some of the speech-related features are actually under the speech area in the operating system. So let's look at the Jaguar features. The first thing we have is the zooming feature that I spoke about earlier. Essentially, this is meant to aid people who have low vision.

So if you need the screen magnified, this is a method that you can dynamically zoom in on any part of the screen. And the interesting thing here as compared to things like CloseView that ran under Mac OS 9 is this is done at a very low level inside the operating system. So it supports things that CloseView never did.

For example, we're able to support zooming of QuickTime movies and also zooming of OpenGL content. So it's a very fundamentally integrated component. It enables you to zoom. I can't remember the exact size, but you can zoom up to a huge degree. And it also has supports for zooming in and out very rapidly so that it has some nice usability features there as well.

The next one is Inversion, which is also relating to vision. Many users might have issues with contrast or brightness or a contrast disability that means they need to have maximum degree of contrast to detect edges. And this is a mode that you can put the whole screen in, which essentially does a screen inversion taking white to black and black to white, and enables you to get the maximum contrast ratio to see the screen.

We also have the Visual Beep, which on Mac OS 9, if you turn the volume off, we'd actually flash the menu bar. We've done something a little more comprehensive than Mac OS X, so we actually will flash the screen. This enables a user with a hearing disability to understand that the system has actually sounded the alert, that the system needs attention and needs for the user to look maybe at the dock or some other part of the system or a dialogue that's popped up and address a particular alert condition.

We also have Sticky Keys. Sticky Keys is essentially an input device, a form of input method that essentially is going to allow users with coordination issues to, instead of chord keys like we do for many operations, like let's say creating a folder, they're able to hit each key individually in sequence, and Sticky Keys will automatically coalesce them into that key press.

So this means things like creating a new folder, such as hitting Command-Shift-N, you can do singly, and the system will essentially show you the actual key presses in a little transparent floating window on the upper left. And then once you actually complete the sequence, it will actually perform past that coalesced key event into the system, and the system will act accordingly. And that works system-wide. It works in applications. Obviously, in this case, it's shown working in the Finder.

Slow keys, another form of input method that is essentially going to take and give a user with maybe a coordination issue the ability to type and hold a key to indicate that's the key press they intend. So this prevents users who may hit other keys in the action of trying to press a key the ability to really indicate to the system they mean this particular key.

We also have mouse keys, and this is useful in general. You can actually navigate, move the mouse with the keyboard. So it also gives a very fine degree of control as well. And then actually something that's another example of assistive technology being really useful in general is keyboard navigation.

Carbon and Cocoa now both support keyboard navigation. And this means most applications are going to support keyboard navigation because this functionality has been integrated into the frameworks. There's also ways, and those will end the sessions later today, for Carbon accessibility and also Cocoa controls. They'll talk a little more at length about how to sort of customize and add some additional functionality to the keyboard navigation that comes automatically on your behalf in Jaguar. But essentially, you can fully navigate the user interface using the keyboard in Jaguar.

Now let me segue over to a different type of development that's done relative to accessibility, and that's assistive technology. This is the part that I said is part of the three parts that support and meet accessibility requirements relative to Section 508. And that is software developers who specialize on creating solutions that are going to change or modify the behavior of the system in order to accommodate someone with a disability.

And, for example, here I have Niedermeyer Consulting's rather excellent Keystrokes, which is essentially an on-screen keyboard. So without even having a keyboard, you can actually use your mouse and push and type into the system. And this is very useful for someone, for example, who may use a head mouse or some other sort of pointing device to be able to work. type.

So, the technical definition of accessibility is this. It's essentially taking your system, your Macintosh, and modifying it with hardware or software to accommodate a specific disability. And there's a huge industry, well, actually a small cottage industry is actually more to the point, that creates these solutions. And also creates these solutions to address the significant number of different types of disabilities.

And this is an important community to understand and interact with as you start looking to make your applications accessible. Because these, in many ways, are going to be the way that someone with a disability is going to interact through assistive technology to actually make your application accessible. So, let's actually spend a little bit of time on how this relates to Section 508.

Section 508, one of the key things it does, in addition to providing sort of a minimum list of things that applications must do to be considered accessible, there's actually sort of a subtext there, which those requirements, for the most part, are for the enablement of assistive technology. They are to remove the roadblocks that prevented the assistive technology community, which provided that finishing step to really making platforms accessible on behalf of application developers and hardware manufacturers. They removed the roadblocks because there were many. So, for example, one of the first things, right off the bat, I think it's Section A and Part A in Section 508, is applications must support keyboard navigation.

The reason that's done is because keyboard navigation is one of the primary ways assistive technology can be used. It controls applications and makes those applications accessible. And then also certain low-level things, such as exposing a lot of user interface information, such as the controls that are on the screen, what type they are, where they are, if they're in a particular state, and also how they work. Focus. I mean, that's pretty obvious to us. We can use our eyes to track where the insert point is or where the mouse pointer is, but someone with a vision disability might not.

So that information needs to be programmatically exposed. On screen. On screen text is another issue. Your application freely puts text up on the screen, but that's a very important thing to make available to assistive technology because they may need to act upon that or communicate that text in a different manner. And there's a couple others inside the specification.

And then a very interesting form of assistive technology is a screen reader. And it does essentially what it says. It takes the textual contents on the screen and typically reads it out loud with some sort of text-to-speech synthesis. In many cases, it will also work in conjunction with a Braille display so that the text on the screen is going to be displayed on an instrument that allows someone who can read Braille to understand the characters that are actually on screen.

This is probably the most tricky form of assistive technology right here. It's technically very, very difficult because actually we had a bird of a feather just the other day and the screen reader manufacturer was there. And they made an important point, something I hadn't thought of, is like an operating system, a screen reader has to support all the applications on the system.

And that's a very, very daunting task. And this is why we need to work in conjunction. Apple Application developers who are interested in accessibility and also the assistive technology providers, in this case the screen reader developers, are all going to need to work together because this is a very difficult problem for any one party to solve alone. A typical sort of OS-level functionality that needs to be provided to support screen readers is going to be methods to discover the user interface state. Focus, where the mouse is, where the insert point is, and also to be able to determine exact mouse location.

Now, an important thing happened when we moved to Mac OS X, and this is one of the reasons sort of for the delay why Apple is just now coming out and saying, hey, we have this great accessibility message in Jaguar. It's because assistive technology on the Macintosh used essentially methods that don't work on Mac OS X. The old extension, install an extension, trap and patch something, insert some code at some random specific place in the operating system to change the behavior.

So this was typical. Most assistive technology that was developed on the Macintosh essentially worked this way. Basically, you installed a set of extensions. Those extensions dramatically changed the behavior of the operating system, but they did it without the application's knowledge. This is why you, if you're an application developer, in many cases had no idea that someone might be using a screen reader to interact with your software, because it wasn't something that you ever really had to develop for. But when Mac OS moved, when Mac OS X moved, when Mac OS X came out, we moved to a protected memory model. So this means one application has one memory partition, and it cannot affect another running application. So this technique of creating assistive technology completely and fundamentally broke.

But also has some benefits because this technique was also very, very fragile. A lot of you may not have been aware that as we all like to revise our products, in many cases as we made a new version, particularly if we went to a major new version of an application, we would simultaneously break assistive technology that was making assumptions about where they could patch in and modify things to get the hooks they needed to make an app which was fundamentally inaccessible and not designed for accessibility under Mac OS 9 fully accessible.

So what do we want to try to do on 10? The end goal of what we're going to be talking about here, essentially the Accessibility APIs, is to allow an assistive technology application, which again is in many cases going to be finishing the accessibility picture on behalf of your application.

It needs to be able to run just like any other app on the system. No different requirements, no special privileges. It's just going to run like another application. It's going to have to do things in conjunction with the operating system to be able to reach across those memory partitions and do things like parse UI state and do things like that.

So these pathways are available in JAGUAR. So assistive technology applications have several ways to now interact with the system. They vary from I/O Kit, which is where you do device driver development and really kind of touch the hardware in Mac OS X. There's also some API in Quartz, which is the windowing system and also the drawing system. The frameworks have been modified in JAGUAR to support accessibility. And the assistive technology application, and to a lesser extent your own application, are going to be clients or targets to the accessibility APIs that are new in JAGUAR.

So let's look at a couple different techniques where assistive technology vendors can actually approach the system, and I'll go into a little more detail. At the highest level, a lot of assistive technology, particularly input methods, are going to be able to rely on the keyboard navigation that's part of Jaguar. They can essentially generate macros of keystrokes and put them into the system, and the UI is going to respond to them as if it was driven from a keyboard.

One level below that is going to be some APIs such as CG Remote Operation, which essentially allows you to synthesize an event and pop it into the system, and it'll go through the regular event queues. So that gives you this sort of synthesis aspect that a lot of assistive technology needs.

Then also, one thing that's important is if you are an assistive technology developer and you're bringing product over to Mac OS X, I think it's important for you to look at the USB human interface device classes, because we find a lot of assistive technology vendors are, they're bringing USB peripherals over and they do a custom device implementation under USB, and that requires a custom driver to be plugged in to the Macintosh. There's an interesting sort of huge specification that's part of the USB spec called HID.

HID has several hundred different categories of input devices. If you rely on HID, it makes it much easier for applications, even your own companion assistive technology application that may support the actual input device, easier to write. The operating system can offer more functionality. Things are much more discreet than if you do a custom implementation.

And then at the lowest level, if you absolutely have to, you can always develop a custom driver for your device. We generally, if you're going to do that, please talk to us because in many cases there's ways to accomplish just that in a different way that you might not have thought of. It's very common for assistive technology developers to immediately jump at the IOCit level and try to do a device driver, and in many cases that's sort of the wrong way to approach things.

As you notice, the sort of thread between what your application needs to provide, vis-a-vis Section 508, and what assistive technology needs to be able to drive your applications, is mainly relating to discovering and understanding and driving the user interface on the system. And this is where we spent a lot of work in Jaguar, is creating the infrastructure to support this. For the first time, we have API-level support for discovery of UI elements inside the operating system.

And this is very, very important, because this moves us away from the model where assistive technology vendors are going to have to rely purely on their own cleverness to make these solutions work. And essentially, it just performs a function where I can have an assistive technology application running in one memory partition, and it can have a dialogue with the framework that's supporting another application, and figure out what the status of the user interface is, where controls are, and it can even send messages to control that user interface.

So this gives assistive technology the pipeline to control applications. And the important point here is this really provides a common ground for Apple, the application developer, and the assistive technology developer to come together and work together. Because obviously, what we're showing here is the first implementation of the accessibility APIs. We happen to think it's pretty good, but I'm sure there's areas where we've missed. And that's where we're going to rely on response from this community.

And that's where we're going to rely on the accessibility API to tell us where we need to iterate and improve. And then once we deliver that functionality in API, that becomes sort of the de facto way to do it. So it's very important that we all work together to continue to involve accessibility on the platform.

Because if we all cooperate, it's going to make the whole act of making all users able to use your software, our hardware, our operating system, and your peripherals that much easier. So what I'd like to do is invite Mike Ingber on stage. And he's going to talk more directly to the engineering aspects of the accessibility APIs. Thank you. Thank you.

My name is Mike Engber, and I'm a software engineer in the Cocoa Group. And for the next 20 minutes or so, I'm going to talk to you about the new Accessibility APIs that we've introduced in Jaguar. Accessibility APIs allow one application to examine the user interface of another application. The clients of these APIs are intended to be assistive applications.

For example, a screen reader could use these APIs to discover what windows are in some other application and then dig around in the window and find out there's a button, discover the text in the button, present that text to the user, and then finally press the button. There's a whole other half to this story, and that's what the target applications have to do to allow this to work.

This session is going to focus on the Accessibility APIs themselves. The two sessions immediately following it are going to cover what target applications have to do. The next session, Session 209, is going to talk about what Carbon apps have to do. And the session after that is going to cover Cocoa Controls. And then about a third of that talk is going to cover specifically what Cocoa apps need to do to work with the Accessibility APIs.

So I'm going to start out by mentioning the goals behind the design of these APIs. The first thing we wanted to do, as has been mentioned before, is we wanted to allow applications to examine the user interface of other applications. We also wanted them to be able to interact with the user interface of other applications, press buttons, move sliders, things like that. And then finally, we wanted them to be able to be notified of changes in their target applications. If a window goes away or a value of some control changes, the assistive app wants to know about it and behave appropriately.

A second goal is that we wanted a single set of APIs to work against both Cocoa and Carbon applications, so that you basically don't have to do twice the work. The third goal was to make sure that these APIs worked out of the box with the standard user interface elements. And this means that most existing applications already have some degree of accessibility.

and it also means that developers have a minimal burden in implementing or making their apps And the final goal was to make it possible for developers who have non-standard user interface features to make them accessible. So I'm going to bring Kevin Aitken of the Speech Group on stage, and he's going to demonstrate what the Speech Group has been able to do using these APIs.

Can you hear me? Oh, boy. Okay, sounds great. Okay, I think I need demo machine number one. Demo one. I'm from the Speech Group and the Speech Group has been working hard on a number of new features for Jaguar. I'm going to show two of them in this session. The reason I'm going to be showing just two is because we're using Accessibility API to implement these. We're really excited about this because this allows us to extend what we're able to provide out of the box.

The two features, the first one is an enhancement to our text-to-speech system. It allows us to speak the on-screen text that the user sees. And the other is an enhancement to our speakable items application. This is our built-in speech recognition command and control system. And now you'll be able to speak menu items and the items inside the front window. So let me go to the speech preference panel here.

You may have seen this in the keynote, but I'll just... So I'm going to run it really quick. Speaker 2: Speak recognition. As I move around the screen it... Speaker 2: Speak text under the mouse. It speaks to visible text and we're all using accessibility API here so as I go down through the doc.

Speaker 2: System preferences. What we're doing is we're basically making two calls here. We're asking accessibility API what is the object under the mouse at this moment and then given that object we ask what is the title. In this case it comes back as system preferences and that's what we synthesize to the speaker.

So we have access to virtually all of the system. Find it. We can go up here to the-- Go.

So let me go back to the speech preference panel. I'm going to turn that off. And I'm going to switch gears and go to the speech recognition to show what we're doing with that. That uses a little bit different method because, you know, the mouse really isn't involved in that case. And so what we want to do is come up from the bottom and explore what the application is providing to us. So let me turn that on.

And instead of actually speak to it, I'm just going to, I think it may be more effective that you can see over here in the speech commands, you may not be able to read all the commands that are in there. But you'll see that as I change the user interface, we get notification and we update that list of spoke commands that are possible at that moment. So we're able to via the accessibility API keep up with the user whether they're using a mouse, whether they're using some other kind of input device.

And so I can switch to another application. We get notification that the application has changed. We can go up to the window here and as it appears, we then get notification. We get a reference. We can easily find out that this is the front window and then begin exploring the full contents of that window and make decisions about what we want to provide to the user.

So that's a real quick demo of that. We have a speech session. We have a speech session tomorrow afternoon. We'd love you to come attend and see how you can use more of a speech in your application. And I think the time of the session will be at the end of this presentation. So back to you, Mike. Okay.

Bring the slides back on. Okay, so the first problem that has to be solved in implementing the Accessibility APIs is how we're gonna represent the user interface. This diagram is an abstract representation of an assistive app trying to examine the user interface of two target applications, one Cocoa application, one Carbon application. And the question marks are meant to indicate that it's not exactly obvious what kind of information is going to be passed back from the target application to represent its user interface.

Each of these applications has a natural representation of its user interface in terms of its own framework. Cocoa apps have NSWindows and NSControls. Carbon apps have ControlRefs and WindowRefs. But there are problems trying to use these natural representations. One problem is that they're specific to the framework, which means the assistive application has to deal with all the complexity of Cocoa applications and all the complexity of Carbon applications.

Another problem is that the natural representations contain a lot of extraneous detail. If you consider just the simple button in a window in a Cocoa application, what you actually have is a button cell inside a control, inside the content view of the window, inside another view, and finally you get to the NSWindow. And the only thing the assistive app is really interested in is the fact that there's a window and there's a button in it.

So this next slide shows our solution to the problem. The boxes in green are the new layers of code that were added in Jaguar. The box in front of the assistive app are the Accessibility APIs, which we're discussing. And the boxes in front of the target applications are the framework-specific protocols that allow them to work with the Accessibility APIs. Now you'll notice that the question marks have been replaced by the phrase or the term UI element, which is short for user interface element.

and our solution, or our representation, is to use a generic user interface element object, or UI element for short. And everything is a UI element. A window is a UI element, a menu is a UI element, a button is a UI element, and there's even a special UI element that represents the top level or the whole application.

So now that we know what the representation is, basically a hierarchy of UI elements, we're going to talk about the details of what UI elements have to offer. First, there's attributes. And attributes provide information about a UI element. There's attributes to return its title or its value or its children, which would be the basis for walking the whole hierarchy. And UI elements also support actions. And this is a way for you to interact with the application. For instance, a button UI element would have a press action. A slider might have an increment or decrement action.

Now we're going to spend a couple of slides talking about attributes. So as I said before, attributes are designed to provide information about a user interface element. And there's basically four operations that the Accessibility APIs allow with attributes. First, you can get a list of a UI element's attributes. Second, you can get the value of a specific attribute. Third, you can check if that attribute can be set. And finally, you can set the value of that attribute.

Now one special attribute is the role attribute, which basically describes the class or the kind of UI element you're dealing with. So there's going to be roles for checkboxes, buttons, menus, things like that. Now this attribute is really for developers to use. It's not to be presented to the user. It's something you could test off of and make a decision in the logic of your program based on the value of it.

If you want to present to the user what kind of UI element you're dealing with, then there's the role description. And that's a localized string that you can present to the user. Another attribute that's commonly used is the title. For instance, a button, the text in a button is the value of its title attribute. If it's a graphical button like an arrow, its title might be down.

Another common attribute is the Value attribute. A slider's value would be a number. The value of a text field would be a string. And then there's two more attributes that relate to the hierarchy of UI elements. The parent attribute is your parent in the hierarchy, and the children attribute is, if it's present, reveals the child UI elements.

So the other thing that UI elements support are actions. And you should think of actions as very simple operations. Basically, about what you could do with a single click of a mouse. Press a button, pick a menu item, things like that. And there's three basic operations that are allowed with respect to actions. You can get a list of the actions a UI element supports. You can get the description of a specific action, if you want to present that to the user. And finally, you can perform that action.

A couple of points to make about actions. One, they're very generic. They're going to be things like pick or press. There's no instance-specific information. If you have a print button or a cancel button, its action is still going to be press. The Assistive App doesn't really know anything apart from the fact that it's a button and it can be pressed.

It does have the option of presenting the title of the button to the user, and if the title says "Print," then the user is going to have an idea of what this button's going to do. But apart from that, there's no attempt to make this information available to the Assistive App.

And another point to make is that actions aren't the only way to interact with the target application. So, for instance, you might be surprised to discover there's no action to select the text in a text field. Instead, what you do is set the value of the selected text range attribute.

So now we're going to talk about a couple of special UI elements. And the first is the application, or the top-level user interface element. This is basically going to be your starting point into the hierarchy if your assistive app sort of works from the top down. Your children will be the windows in the application and the menu bar in the application. There's a special attribute to return the menu bar so you don't have to dig through the window list to find it. And there's another attribute to return the list of windows.

There's also an attribute to return the UI element that currently has the keyboard focus. And there's an attribute to return whether the application is active. There's also attributes to return the main window, the key window, and there's a whole variety of other attributes that I'm not going to go into.

There are also two special calls in the API that only work on application UI elements. And one of them is for hit testing. If you wish to discover what UI element is at a specific position, you can use the application UI element, pass it the coordinates, and you'll get back the UI element at that position. and there's also a special call in case you wish to post keystrokes to an application. So if your assistive app needs to emulate a keyboard for some reason, you can use this API to send keystrokes to an application.

A similar and related concept is the system-wide U-Element. It's kind of an application U-Element, but it represents all applications. But depending on what your assistive app is trying to do, it might be convenient to discover what U-Element is under the mouse, regardless of what application it is. And so if you do hit testing on the system-wide U-Element, you're going to get back some U-Element, and you don't know what application it's going to be in. And for instance, that's what the reading the button, this speech demo was using. The system-wide UI element also supports posting keystrokes, and they go to the active application. And there's also an attribute to return the current keyboard focus, again, regardless of the application.

The last section of the APIs has to deal with notifications. Assistive apps want to be notified about changes in their target application. There's a series of notifications having to do with Windows, when they're created, when they're moved, they're miniaturized. There's also a series of notifications having to deal with changes in the state of the application, if it's hidden, if it's activated, deactivated, things like that. There's also a notification that'll tell you if a specific UI element has been destroyed or if its value has changed.

When you register for a notification, you specify the name of the notification you're interested in, and then you have a choice. You can either specify a specific UI element you want to observe, or you can pass the application UI element, meaning any notification of this type that happens in the application, I want to hear about it.

So we've covered the basic concepts behind the Accessibility APIs, and I thought I'd like to sort of wrap that part up by pointing out a few things the APIs do not do. So one of them is the Accessibility APIs do not really address the same problems that scripting does.

If you think about just the relatively simple example of printing a document, in Apple Script, you can do that with a single line. You tell the application to print the print file whatever, and it prints it. If you try to do this with the Accessibility APIs, first you get to dig through the menu bar and find the file menu. Then you find the open item, and you get to pick it. Then you have to use the standard file dialogue and navigate to your file and eventually open it up.

Then you get to dig back through the menu bar, find the print menu item, and so on. You get to deal with the print dialog. So it's very tedious. So if what you're trying to do is drive the application, AppleScript is really better suited to the task. But if what you're trying to do is help a user with the user interface of the application, that's what accessibility does.

Another thing you might notice as you use these APIs is that there's things we could have done that we didn't. For instance, the title of a window, that attribute could be settable, but it's not. Or we could have allowed you to set the value of a radio button directly. Instead, all you can do is press the radio button.

So again, bear in mind that these APIs are designed to allow you to do what a user could do if they were driving the user interface. They're not designed to allow you to program this other application. And in the case of setting the value of a radio button, that could be pretty dangerous because if you uncheck a radio button, you might leave the radio cluster in a state where nothing's selected, a state that the programmer may not have anticipated being possible.

And so my final comment on this is that I'm sure these APIs are going to be put to all kinds of creative uses. And I'm not going to pass judgment on what's an assistive app and what's not. But one thing you should keep in mind is that this functionality is going to be off by default. In your C disks, it's turned on by default. But when we finally ship, the user is going to need to go into the universal access system preferences and turn it on.

So now we're going to bring Kevin back on, and he's going to demonstrate a tool

So what it's doing is it's following the mouse around and picking up what object or UI element is under the mouse at this moment and displaying all the information that is available through the Accessibility API. So let's stop up here and lock on this startup disk icon. I'm going to push F10 and that's going to lock us on there so I can have my mouse back.

So as we look through here, at the top part here, we see the Startup Disk Push button. And we can work up the hierarchy. It's inside a toolbar, which is inside the window named Speech. And that is inside the application name System Preferences. So that gives you an idea of where this push button lives.

And then, as Mike talked about, here are the attributes. So we have the role, the role description, the help, there's no help, some state information, some pointers to its parent and its window, the position, size, and finally the title. And this is what you saw in our text-to-speech demo where we're grabbing the title, and that's what we're going to speak.

And then at the bottom down here are the actions. And so this supports a single action, press. So let's go try it out and see if that actually works. So what I'm going to do is I'm going to click on this perform button and it's going to use the Accessibility API to send that press action to the startup disk icon.

So I'll push that and you'll see that it changes. So let me go lock on something else and I'll lock on the actual window itself. And it doesn't have any actions per se, but it has some rules. So let me bring it forward. So the ones that have W's after them are things that we can actually change. So let's do something fun. Let's change the position of it.

Set it back to zero, which is Google in default, and it pops back up. So this tool is really great because it allows you as a target application developer to go through your application and to test to make sure that your accessibility features work. And as an accessibility client application developer, you can look around through all the system and different applications to see what is available for you. So that's all I wanted to show. Back to you.

Bring the slides back on. Okay, so now that we've gone over sort of the high-level overview of the concepts behind the Accessibility APIs, now we've got about seven slides that are more or less the header files. So I apologize in advance if it's a little dry. I think for the programmers in the audience, this is what you want to see.

So as I mentioned before, accessibility is not going to be on by default. And if your system app wants to determine whether or not it's on, there's an API, AX API enabled that will let you know whether it's on. And if it's not, you'll have to explain to the user what they need to do to turn it on. You'll notice all these APIs start with the letters AX.

One issue I've been kind of skirting around is, what is the data type of these UI elements that we're talking about? They're a CF type. It's a type of our own invention, and they're reference counted like other CF types are. You have to use retain and release to manage that. You could use them as elements of CFArrays, as keys in CFDictionary's. If you want to determine if some ref you have is a UI element, you can compare its type against the return value of AX UI element get type.

So this slide presents basically the starting point for most of you into the user interface element hierarchy. If you're doing sort of a top-down approach, like the speech recognition system does, every time an app is activated it starts at the top and starts looking around at the active window and digging through and finding out all the commands it should recognize. Then you'll use the create application call. It takes the process ID of an application and it returns you a UI element for its top or application object.

If, on the other hand, you're doing more of a bottom-up approach, as Kevin was just demonstrating, for instance, you want to know what UI element is under the mouse, then you'll probably start by creating a system-wide UI element with Create System-Wide. And then you can use the call Copy UI Element at Position to get the UI element at a specific position. And then also, this slide also shows the keyboard posting API, if your application needs to emulate a keyboard.

So this slide shows all the API that's involving attributes. Given a UI element, the first call, Copy Attribute Names, is going to give you a list of all the attributes it supports. Copy Attribute Value will give you the value of a specific attribute. Is Attribute Settable will let you know whether you can set that attribute. And finally, Set Attribute Value will let you set its value.

There are actually two more APIs involving attributes. These are special ones for attributes whose values could be large arrays. You could imagine a table view with a large number of children.

Another issue I've kind of skirted is what is the data type of these values? In some cases, it's pretty obvious. CFStrings would represent strings. CFNumbers will represent numbers. But there's a few other odd cases like sizes or ranges or points that there's really no, or at least there's currently no CF type that represents it very well.

So in these cases, you're going to get back a generic CF type rep. And you can pass it to AX value get type. And that's going to tell you whether or not it's one of these points or sizes or ranges. Or if it returns a legal value, it means it's just something else.

Once you determine you have one of these special values, like a point, and you actually want to work with it, then you pass it to axValue.getValue, and that will return the point as a CG point, and then you can extract its x and y coordinates. and then the opposite direction is also possible. For instance, if you want to set the position, you need to create one of these things and you use AX value create to create one of these types.

These are the three APIs that are dealing with actions. The first one lets you get a list of all the actions that a UI element supports. The second one allows you to get the description of a particular action in order to present it to the user. And the final one lets you perform the action.

And these are the notification APIs. You start out by creating an observer, which basically associates a callback routine with a particular application. Then you use Add Notification to hook that up to observe whatever notification you're interested in. And then you could use Remove Notification if you lose interest in that notification.

Now, the reason we have this extra step of creating an observer instead of having Add Notification take a callback directly is that we wanted to give you flexibility over what run loop your callbacks would be executed in. So the thing to remember is after you create your observer, you have to get its run loop source and then add it to some run loop. If you fail to do this, your notification will not get called.

And that's it for the APIs. This slide shows some of the documentation that's available. And the first document covers these APIs and how you, you know, from the point of view of someone writing an assistive app. And the second document is the topic of the next two sessions, basically what Cocoa and Carbon developers need to do in order to work with these APIs.

I have a road map slide somewhere in here. Yes, it's the next slide. Just to let you know, there's actually two other sessions today. I think they're in this hall. I think they're one after another. Yes, they are. And those sort of finish out the accessibility story because, like I said, there's a relationship between the Accessibility APIs, which then talk to some of the new stuff we've put in each framework to support it.

And we have an Accessibility and Carbon, which will go into details of your Carbon application developer, what you have to do to support these APIs. In some cases, you get a lot of functionality free. But in other cases, like custom controls, you may actually have to implement something.

And then Cocoa Controls and Cocoa Accessibility is actually going to cover both Cocoa Controls with, towards the end, a little bit of information on the accessibility for Cocoa because a lot of the Cocoa Controls automatically support accessibility now. So those two sessions are a little bit later. And please come see those to get the whole picture of accessibility on Jaguar. Now what I'd like to do is host a question and answer. I'd like to invite...