So Apple approaches accessibility in sort of a cross-functional way. And this is sort of our mantra, our world view of accessibility. We consider, you know, we don't break the world into users without disabilities and users with disabilities. It's a continuum of users. And we want to support users with all abilities.

And so it's important that you understand the way we approach this is we want to make sure that we can deliver an operating system that has a great experience. And this is sort of, you know, our world view. And this is definitely how the cross-functional team, which involves people from the relationship side at Apple, involves technology evangelists like me, it obviously involves several different groups in engineering who work on accessibility as a problem on Mac OS X.

And the key thing is that we believe accessibility is important. And if you look at the statistics, these are U.S. government figures where technically 54 million people in the U.S. have some form of disability. That's a huge number. Now obviously there's a large continuum of disabilities and not all of them require assistive technology to overcome. But I think the key thing is that if you look at statistically, there's a large number of users out there that can benefit from having your application work with another party's assistive technology that works in conjunction with our infrastructure and the operating system.

Interestingly, since the ascendancy of Section 508 and the issue of accessibility, you have accessibility becoming a sales requirement in the education, the federal, and local government spaces. We'll get into what Section 508 is, I think, on the next slide. But the key thing is if your product sells, your application sells into certain markets, you're going to get asked about accessibility. What is your Section 508 compliance is going to be the question you're going to get asked. And it's something you want to be able to answer correctly because if you don't have an accessibility story for your application, you might not be able to access those markets.

And then lastly, it's really the right thing to do because I think that it's difficult for people who have not worked with the disabled community or the assistive technology developers, may not have a full understanding as how simple changes in your application can have significant effects in terms of opening the usage of that application to a person with a disability. It's very interesting.

One of my first contacts with assistive technology was I was at a conference and I was waiting to go up an elevator and I heard from behind me the Macintosh Pro voice asked me to push the second floor button on the elevator. And it turned out to be a person who used an iBook that had been converted to do work with them as a piece of assistive technology that essentially acted as that person's voice.

Now I use speech. I often have the text-to-speech capability in Mac OS X, read me emails. But I didn't really fundamentally understand that this technology, it was a bonus technology for me, was a pivotal technology to a person with a disability and enabled them to communicate. And so you really have to think, little things you can do inside your application can have significant reciprocal benefits to the disabled community and it's very much worth your time to engage that community, to engage assistive technology developers to learn about this. Because the real thing is, it's a very important part of the process. So really change your mind on how you approach accessibility. Instead of just being some checkbox on some sales form, just something that's really the right thing to do.

Quick note about Section 508. In many ways, Section 508 has been driving this. Section 508 is a federal requirement, and essentially it's a specification that the government's published that sits down a minimum accessibility standard for a variety of new devices. Now, new devices are things like operating systems, applications, electronic devices such as computers. And it's essentially sort of the same body of legislation and law that mandates curb cuts and various handrails and things like that.

And it just took a little while for those standards to be applied to items such as personal computers. Section 508 obviously was reauthorized or basically it said, hey, it's coming down the pipe in 1998. The various communities that are affected by this got together and started -- Section 509 was re-reported to the state of California. And it's a standard that's been applied to all of the state's standards. You know, organizations working on defining what those requirements were.

And in June 21st of 2001, these standards actually became formal. So there's actually a document that you can go get that tells you exactly what things your application or your operating system or your electronic device has to do to be accessible. Now, one unfortunate situation that arrives with a hard document, particularly a federal document, you know, outlining accessibility is in many ways it's not complete. In many cases, people consider that it is complete.

Okay? And this can create some issues. It is a great starting place. Section 508 is a good starting place to learn about accessibility to figure out how to make your product accessible. But it does not define everything that you could potentially need to do to open your products up to users with disabilities.

So let's talk about our approach to accessibility. Because again, it's new. When Justin--when you introduced Mac OS X, a lot of people were concerned because they really didn't see much about support for assistive technology and accessibility built into the OS. But in Jaguar last year, we actually announced a whole bunch of additions to the OS to really support, you know, making applications accessible and support our users with disabilities.

We actually, when looking at this problem, because it is a big problem, we actually factored it out in two parts. One is something we call user level accessibility. This is our built-in functionality, those universal access features I'll talk about in a moment. And this is when we looked at the things that we have to do to enable users with disabilities to engage our platform, there was a subset of them that it made sense for Apple to do. And so we actually have built those and baked those into the operating system, and I'll show you those in a moment.

And then on the second hand, we realized there's a whole lot of adaptation that needs to be done to engage the whole community. And we're not expertise in every form of adaptation. So we wanted to actually build infrastructure in the OS to enable the assistive technology community to engage the platform.

And so that's what the developer level accessibility is. And that's really characterized in our Mac OS X Accessibility API. Apple, what's the difference between the two? So Universal Access. Some of you may have already seen this icon. It's in the pref panel in Mac OS X. And this is really the hub, if you will, for our assistive technology features that are built in. The user level accessibility that I spoke about.

So I'm going to actually come over here to, I think it's Demo Machine 3, and give you a real quick tour of the Universal Access pref panel and show you what we have built into Mac OS X. Some of this you may already have seen in Jaguar, but we've actually made some enhancements to it, very much in the case of user feedback, what people wanted to see.

So I come here and I see right here I have my universal access press panel. I click on it and it's going to bring up essentially the hub of our user level accessibility features. We sort of break it down into categories of disabilities. So for example we have seeing, hearing, keyboard and mouse. And if we take a look at the first option which is seeing, this is actually where, I'm a graphics guy at heart so this is where I think the coolest feature is.

We actually have full support for screen enlargement inside the operating system. And this enlargement is done in a very technically advanced way because of the way our windowing system works. And the fact that we use technology such as OpenGL to accelerate our display. So we're actually able to do some pretty fantastic things here.

For example, I can zoom in and I have full speed zoom. It's very high quality and I can zoom in and out very fluidly. And the other interesting point is that one of the problems with enlargers that people have been familiar with in the past is in many cases these only worked with Windows and text documents.

So if there's other rich media that happened to be being used at the time, they really couldn't participate because of the bottlenecking and slowdown that having to do all this copy, pixel copying and replication would be. And so we wanted to make sure that we could architect a system that really didn't suffer from that.

So we actually created the system to where it's able to do things like, for example, enlarge even multimedia. So it's a quick time pop-up. Tell it to loop. and I'm able to freely zoom up and experience multimedia without having to have any conflict with the screen enlarger. And I can actually enlarge to extremely large proportions.

and I now have a 35 foot tall pointer over there. So it's a really powerful feature. Now one thing that we also got as feedback was that people wanted more control over the zoom so we've actually made it pretty flexible in here. We're able to set your zoom, how far you want it to zoom in, zoom out, if you want it to show you where it's going to zoom into, a lot of options like that.

We also have the ability to turn on and off the smoothing so that if I am enlarged that I can turn on and off the anti-aliasing because Mac OS X is fully anti-aliased and in some ways if you have issues with contrast the anti-aliasing can make the type sometimes difficult to read. This was sort of a first solution to be able to address that problem but in Panther we've actually gone one further. If I actually zoom out a bit we'll see we have a new option.

And that is going to be the enhanced contrast. And this is a really cool feature because we had a lot of feedback that the anti-aliasing, the soft edges gave people with low vision problems and second is the pin striping. The pin striping can be very difficult if you have issues with perceiving contrast. So we actually sat down with one of our crack graphics engineers who thought about it and came up with some very funky math to actually be able to do this effect where, and let me actually go one farther in.

You can see the pin stripe and I'm actually able to take it and move my screen and just change the slider here and it's going to be able to make the pin stripe and drop out and it's going to increase the contrast of the windows. Additionally it's going to harden up the edges of the type.

So even though we have anti-aliasing the type can go, we can harden up the edges of the type so if we look at it the type itself is going to become bolder and bolder. And this is again a request that was put in specifically for features. Right. by our users.

and its system-wide effect. So, and it's also variable, so you can dynamically tune it. We also have full support for flashing alerts for people who have hearing disabilities. We have sticky keys and slow keys, which are also alternate input devices, if you will, for people with dexterity issues who may not be able to manipulate the keyboard.

Travis Brown, Mike Engber, Kim Silverman We also have full support for flashing alerts for people who have hearing disabilities. We have sticky keys and slow keys, which are also alternate input devices, if you will, for people with dexterity issues who may not be able to manipulate the keyboard.

Travis Brown, Mike Engber, Kim Silverman We also have full support for flashing alerts for people with dexterity issues who may not be able to manipulate the keyboard. and the, up on the screen in a semi-transparent way, the keys that are currently, the modifier keys that are currently cached are going to affect whenever our main keys hit.

You also have mouse keys to enable the numeric keypad to out like a mouse. We also have some other options. So, for example, if I cruise over to something that's sort of interesting, a lot of the features that we've built in the Universal Access Press pane, we found a lot of people liked using them regardless if they had a disability or not. For example, the screen enlarger turned out to be very popular in last year's WWDC for people to show code snippets.

We also have some features that work the other way. So, for example, if I go into my keyboard and mouse, I see a new option called Keyboard Shortcuts. And essentially, this enables users to define hotkeys. Now, anyone can use this, but for a user with a disability who might need to assign a hotkey, one key press to do a lot of different operations, it's very, very powerful.

And then also, we're going to have a demonstration a little bit later of another very important piece of assistive technology that's built into Mac OS X. Which is our speech, which keeps on getting better and better. So that was a quick tour of the Universal Access PREF panel. And let me go back to slides.

So let's start to drill down on the developer side. Because I talked about the user side, the experience when a user gets a machine and has the ability to essentially engage it, even if they have a disability. But now let's sort of work our way down into the infrastructure to talk about what application developers need to do and also talk about what assistive technology developers need to do.

We're going to have to do a little bit of storytelling here because we had to really take a look at accessibility, how it's handled in other platforms and also in Mac OS 9. And we built our infrastructure for doing accessibility in Mac OS X. And one of the key things that sort of created this scenario is we built an entirely new OS. Mac OS X is entirely from the ground up new operating system. And that actually had sort of two challenges. One is we had a lot of infrastructure to build. Second is for our developer community.

It was a little bit of a curveball, particularly for the assistive technology community, because their landmarks in terms of how they affected accessibility on the platform totally changed. So one of the key things is that I think the biggest thing that really affected them was the protected memory model. Now, obviously, everyone was overjoyed to get protected memory on Mac OS X, but it was horrifying if you were an assistive technology developer. Because that meant that your app, assistive technology application, could not tell what another application was doing.

Because there was a hard wall. There was protected memory in between them. And then also some core changes in the way we handle I/O. Mac OS X has a very abstract and modular and object-oriented I/O architecture. And another thing that's actually an interesting challenge is that our visual environment is really complex.

The ability to seamlessly composite 2D, 3D, multimedia, to do the dynamic enlargement. It's a very different world up in the frame buffer than it was on other platforms and in the past. But in many cases, some of these challenges actually turn into reality. And some of these challenges actually turn out to be benefits.

And we'll talk about those a bit later as we go on. But the key thing was we took this opportunity by the fact that we built a new operating system. And we had to come with innovative solutions for the assistive technology community and developers who need to make their applications accessible. We decided to say, "Hey, we're in here. We're building infrastructure. We might as well formally include that infrastructure." And make it to where developers can rely on key parts of the system to do things on their behalf.

So the first thing we wanted to look at is how it was done the wrong way. So we actually looked at the way it was done on Mac OS 9. And it wasn't done the wrong way because anyone was making a choice to do it the wrong way. It was really because it was the way that accessibility and assistive technology organically grew up on that platform. Because Mac OS 9 architecturally was a short stack of hardware with an irregular layer of system software and then the application sat on top of that and it was all running in one memory partition.

So it was pretty easy for the assistive technology vendor to basically put themselves up on top and if they needed to change the behavior of running application or adapt the functionality of the operating system, they could just sort of have at it and go around and change things. So in many cases, assistive technology would essentially eject patches between the application and the system software.

Obviously, assistive technology developers weren't the only people who do it. We had all these issues with extensions on Mac OS 9 and that also was used the same techniques. But in many cases, assistive technology vendors had to be more aggressive. In some cases, they'd even change the way they were operating the application.

They'd insert their code into a target application to modify the behavior. Or they would do things like get in between the hardware and start pulling events directly before the system software could get them. And in many cases, they would actually do trickery with the way the hardware presented itself to the Macintosh. So you actually had a scenario where assistive technology devices and assistive solutions were in a "do it yourself" environment.

And we really realized that this was the wrong way to do things. We did not want to replicate this behavior on Mac OS X. So that meant that Apple had to get involved and create this infrastructure. And so we realized in our new operating system, we have a common layer that's called the application frameworks layer.

And between all these different ways to develop for Mac OS X, each application framework, there's actually common infrastructure that runs between a lot of them. So we said, "Hmm, this is a nice choke point, if you will, to start building accessibility into the operating system to support the Mac OS 10." assistive technology and application developers.

We had to sort of factor the problem because each, one interesting thing about Mac OS X is it's got, you know, four discrete ways to develop for it. If I'm a Unix developer, I can use Darwin and BSD. If I'm a Java developer, I have full Java implementation. If I'm a developer who has core competency on the Macintosh from, you know, Mac OS 8, 9, whatever, I have those APIs that have been, you know, a significant body of those APIs in Carbon that have been brought forward and they're available on the platform. And I also have Cocoa, which is a fantastic way to do object-oriented development.

All these frameworks are very different from one another. And making them all accessible is very tricky. So we actually originally started with just Carbon and Cocoa. So last year, we were focused on just making Carbon and Cocoa applications accessible on the platform. But for Panther, we've actually extended that out to include Java as well.

So we're able to leverage the common infrastructure to essentially have all the applications sitting on top of the application framework layer. They're using this core infrastructure. And so the idea here is to take the assistive technology solution, which before had to do some very interesting things with the system, and sort of make it just another application to where it was an application that just ran using the same infrastructure and was able to use that to do what it needed to do, to have access into other applications. So we had to formalize some pathways for this access to occur. So that's what the Mac OS X Accessibility API is.

It's API-level support for accessibility. It's accessible. It's standard C interface, so it's accessible from Carbon and Cocoa. It allows applications to discover and control the UI of other running processes. Because when you look at it, when you look at the whole field of accessibility, that's one of the most common threads that a piece of assistive technology needs to be able to adapt the functionality of an application. So we have a lot of applications that are not even aware it's being adapted. So we give pretty low-level control for one application to peer across a protected memory and look into another application and determine the UI state and to be able to affect that state.

It supports some existing APIs that we've actually had in the system for a while that were more general-purpose APIs for synthesizing events, keyboard-mouse events. And so it can also be used in conjunction with those to create some very powerful solutions. But more importantly, this API set has really become a common groundwork for us.

So we're able to use this to create some very powerful solutions. So we have a lot of applications that are not even aware it's being adapted. So we give pretty low-level control for one application to peer across a protected memory and look into another application and determine the UI state and to be able to affect that state. But more importantly, this API set has really become a common groundwork for us. So we have a lot of applications that are not even aware it's being adapted. So we give pretty low-level control for one application to peer across a protected memory and look into another application.

And more importantly, this API set has really become a common groundwork for Apple and the assistive technology community to work on accessibility on the platform. In fact, a lot of the feedback we've gotten over the past year in terms of things you'd like to see in the way we handle text and text representation has been incorporated into Panther. And then obviously we introduced the core set of this in Mac OS 10.2, and we've done some enhancements, particularly in the area of text handling in Panther.

So what is UI element? Again, it's a framework-independent abstraction for user interface. And we'll actually have a little diagram of how a dialogue works, but you can think of essentially an application is appearing of a hierarchy of these UI element abstractions. Because a UI element can actually have attributes in terms of the title of a window or the value of a particular control or slider, for example.

And a UI element can also have children attributes. So, for example, if you have a window and it contains within it a child, which might be a button. And obviously certain controls are going to accept actions, user interface events. And so you can actually have, a UI element can have an action as well, such as a press action or an increment action. There's a whole suite of those.

So let's actually sort of take a look, because I think it's better to sort of describe how this really works in reality. And show how the UI elements in the Accessibility API work to overcome the problem of one application, a piece of assistive technology, needing to know what's going on in the user interface of another application. So here we have a theoretical piece of assistive technology, and we have another application that's just, you know, my app that's running. Obviously they're separated by a protected memory model. They're in different processes. So we need to somehow to start a dialogue.

To determine the user interface state. So the first thing I do is get the process ID. And there's a couple different APIs to do this. But once I have the process ID, that's going to, I'm going to have to basically ask for a UI element, the initial UI element for this process ID. So I will call ax_UI_element_create_application, pass in the process ID, and I'm going to get back a UI element that's going to contain some information, the top level information on that application.

So what I'd like to do is invite Mike Engber to the stage. And he's going to take us in--that is his actual job title right there. And he's going to actually show us the UI Element Inspector. The UI Element Inspector is actually sample code that you guys can pull down and take a look at. And it's basically a way that you can use the accessibility APIs and show them in action.

Okay, hello. My name is Mike Engber and I'm going to start off with a demo of UI Element Inspector that's going to sort of give a live illustration of what Travis has been talking about. And then I have a few slides, only six, so don't worry. And I'm going to cover some of the things we've added for Panther in the accessibility APIs. And then finally, I'm going to end with another demo of a fun little app.

So UI Element Inspector is available as sample code from DTS. It has been for a while, so some of you may already be familiar with it. And I'm going to start by launching it. And what this program does is every time the mouse moves, it checks what UI element the cursor is over. So as I move over different things, you can see that the contents of this window change. I'm going to increase the font size so people can read it a little better.

and now I'm going to launch TextEdit. We're going to do some demonstrations of what it can do with TextEdit. So here I'm over an untitled window in TextEdit and we can see that we get the, The list of attributes that the window supports, things like the role, the role description, the title. At the top of the window is showing the position of the window in the hierarchy. So you can see at the top of the hierarchy is the application itself. And then this window is one of its children.

One of the sort of challenging things with the UI of this program is since whenever I move the mouse, it changes what I'm looking at. If I want to operate on that thing, that's a problem. So use a hot key and you can lock in on the particular UI element that you're looking at and everything turns to red to indicate that.

And then this little interaction window appears. And I can't increase the font size in it, so I have to apologize, but I'll describe in detail what I'm doing. So we can see we are now locked into the window and one of the things you can do is use this pop-up menu and get a list of all the attributes in the window, which we've already seen above, and we can see some of them have W's next to them.

That means they're writable. So I'm going to pick the position attribute. We can see its position is given in XY coordinates and I can... Change the Y coordinate and hit set value and the window moves. and other experts have also been involved in the development of the app.

and it's a Boolean. It's now at zero and I can change it to one, set it and the window disappears and then I'll bring it back by setting it back to zero. and this illustrates a point that sometimes people new to the APIs forget, is that actions aren't the only way to interact with UI elements. A large amount of your interaction is going to be accomplished through setting the values of writable attributes.

Okay, so now I'm going to bring up TextEdit's Preferences pane. And I'm going to illustrate a couple more features of UI Element Inspector. I'm going to focus in on this text that labels this box. and there's this little check box in the interaction window that lets you highlight the thing that you're locked onto. And so this is useful for going around the app and just seeing what you can hit. So I'll demonstrate here we've hit another button. Here we can hit a pop up and I'll go back to that label.

And then also in the interaction window is a pop-up named Go To. And you can actually walk the hierarchy. So you can use your mouse to get close to where you want to be. And then we can go up to the parent. We see the highlight changes to reflect that that title is actually a child of that box. And then we can go up another level and the box is a child of the window.

and I'm going to show one more thing. So you might be asking the question, can UI Element Inspector inspect itself? And the answer is yes, it can. And I'll end the demo by having it close itself. So we can, here I'll perform an action, which is AXPress on the close button of the window. And we're done.

All right. So, as we just showed, we have this model of attributes and values and you can ask a UI element, you can pass it a name of an attribute, you can get back the value and this really suffices for most purposes and this is--and this is really all we had in Jaguar. But we found that sometimes just passing the name of an attribute isn't quite enough. So, in Panther, we added this concept of parameterized attributes.

And for instance, you might want to know what character the mouse is over. And in that case, you need to pass in the position of the mouse. Or you might want to know the bounding rectangle for some text. And in that case, you need to pass in the range of characters of the text you're interested in.

So, we added this concept of parameterized attributes. It's conceptually, it's similar to attributes. But now, when you ask for the value, you pass in a name and you also pass in a parameter. And the type of the parameter varies. And I'll go over some examples of that in an upcoming slide.

So to accomplish this, we added two new APIs in Panther. The first one is ax-ui-element-copy-parameterized-attrib ute-names. And you can see we kept brevity in mind when naming these APIs. And the second one is ax-ui-element-copy-parameterized-attrib ute-values. So the first one is very similar to an existing API that returns you a list of all the attributes. This one returns you a separate list of the parameterized attributes. And it's a separate list for compatibility reasons. We didn't want existing code bumping in to parameterized attributes and trying to get their value and not passing a parameter. So since they're separate lists, that can't happen.

and the second call is what you're going to use. Once you've got the name of the parameterized attribute you're going to use, you get to pass in the UI element, the attribute, the appropriate parameter, and then you get back your value. and for now, we only support reading, not writing of the parameterized attribute values. But if a need arises, we might consider extending that in the future.

So no presentation would be complete without a giant table of very small text. So I apologize if you're having trouble reading it, but I'll go over some of the items in detail. This is a list of the parameterized attributes that we have currently implemented in Panther. And they all have to do with text. So there were a lot of requests from developers to find out more detailed information about text, what style it's in, how it's formatted on the screen.

And this is where we found we really needed a parameter along with an attribute name in order to answer these requests. So the first one is KIX line for index parameterized attribute so this one is addressing the issue of the text layout If you pass the index into a text string, then we're going to tell you what line it's on. And then the second attribute up there does the opposite.

If you give me a line number, I'm going to tell you the range of text that's on that line. Then there's a... String for Range, which will, given a range of text, allow you to just get that much of it. Before, you could ask for a text object's text in its entirety. This lets you get it a chunk at a time.

Then there's one that'll tell you if you specify a character position, what range of text you're over, and you have to keep in mind that, you know, the mouse might appear to be over a single character, but in fact, because of surrogate pairs or multi-byte glyphs, you might be over several characters. So we return the whole range.

Then there's range for index, and this again has to do with multi-byte characters. If you specify an index in the text, we'll give you the full range of characters that comprise that glyph. So there's calls similar to this, like range of composed characters in the kit, and this is sort of the accessibility equivalent to that.

Then we have some that have to do with layout. Given a, there's the KX range for, KX bounds for range parameterized attributes. So you give a range of text and we're going to give you back a bounding rectangle. And then there's the, K-A-X R-T-F for range parameterized attribute.

So again, you can give us a range of text and we're going to hand you back the R-T-F representation of the text. And from there, you can extract the font, the size, the color, just about everything you could want to know. And then finally, there's this K-A-X style range for index parameterized attribute.

I'm sure you guys are going to enjoy typing these names in just as much as I am speaking them right now. And this last one lets you go through your text in style runs. So if you're interested in when things change, you can use this and find out, you know, the extent of this current style run and then move on to the next one.

There's a couple of other attributes we added for text accessibility in Panther, and these aren't parameterized attributes. The first one is just to tell you how many characters are in a text object. And the second one is to tell you what range of those characters are currently visible.

And the next two are a little more esoteric. These have to do with text that flows. If you have, for instance, a two column document, that might be two separate text UI elements and the text flows between them. So the first one, the shared text UI element attribute, tells you what's your portion of the text that you're displaying.

Because in general, most of the attributes, if you ask for the text, they just treat this, treat you as if you have all the text because that's most of the time what you want and that's most convenient. But if you really need to get to find out what exactly only this UI element has, you can use this attribute to find it out. And then the last attribute, AX shared character range attribute.

Actually, I got these backwards. The second one is the one that will tell you your share. The first one is going to return you an array of other UI elements where your text is shared with. So, you know, it would return you the other column if your text was split between two columns.

Now here's a few other untext related attributes we've added for Panther. First, we had a lot of requests for people wanting to be able to identify the default button. So we have a parameter on Windows that will tell you the default button and we also have one that will identify the cancel button. We also added a couple of notifications. So when drawers appear or when sheets slide down, you can get a notification. When they're destroyed, you get the normal UI element destroyed notification. So we didn't have to add anything there.

We also added some new sub-roles for windows. Again, this was a, we had a lot of requests from developers wanting to be able to distinguish a floating window, possibly ignore it, from a dialogue, from a regular window. And so, we added a variety of sub-roles that will identify dialogues, floating windows, and then there's an all-purpose unknown if someone puts up a custom window, we can't tell what it is. And so I'm going to finish up with a demo of a little app we call Monkey See, Monkey Do.

So I'm going to start by launching TextEdit. And this is a typical document you might work with on any day with a variety of tastefully formatted text. and the rest of the team. So, I'm going to show you how to extract rich textual information from text and different styles and different colors.

And next I'm going to launch--I'm going to first move this out of the way and then I'm going to launch MonkeySeeMonkeyDo. And this app, it's named MonkeySeeMonkeyDo because basically that's what it does. It uses only the accessibility APIs and it tries to replicate the user interface of another app. So, I'm going to--from the list it presents me, I'm going to pick TextEdit.

and it's computing its own, you know, it's creating its own window that's supposed to look as close as it can get to the original app. And again, all it's using is the accessibility APIs to get the information it needed. And it's actually completed a while ago. I'll just have to move it off. So, again, this illustrates that all the rich text information is available to you through the Accessibility APIs.

So I'm going to demonstrate some other features of MonkeySeeMonkeyDo. First I'll go to TextEdit. I'm going to close this window. And you'll notice MonkeySeeMonkeyDo removed its copy. That was an example of getting a notification of a UI element destroyed. And now I'm going to bring up preferences. Now this is actually a little tricky because when Textetics deactivated, its preferences panel disappears. So that adds a little bit of a wrinkle to this.

But MonkeySeeMonkeyDo has already done its best to replicate the preferences panel. And you might have noticed the change if you were paying attention. This one doesn't quite look right. And that's because there's no way through the Accessibility APIs to discover whether you're using regular size controls, tiny controls, or small controls.

So you could take a guess based on the size and here it's not guessing quite right. So I'm going to pull aside MonkeySeeMonkeyDo's version of the preferences. And again, TextSet, it's not in the foreground so that's why you don't see the other window. But as soon as I bring it back, you'll see it.

and using notifications, there's some more connections we can show. For instance, we can pick something from the pop-up menu. Of course, I picked one that was concealed. You can barely see it below the upper window. I'll go back to automatic and you can see it changed. Next, I'll click a radio button and you can see it changes between the two. can illustrate some typing. We can change the window width to 100. And when I hit Enter, it'll propagate down. So again, this application is just sort of an illustration of what's possible to do with the Accessibility APIs.

Okay, back to Travis. All right, thank you. Cool. So let's go back to slides. So now what I want to talk about is if you're an application developer, I'm going to touch briefly on how do I engage the accessibility APIs, all the sort of infrastructure that we've been talking about in the first half of the presentation.

And we'll go through it really quickly because we did quite a bit of this last year in terms of going into great detail. So there should be sessions on ADC TV and the DVD ROM on Carbon accessibility. There's also one on Cocoa Controls and also the accessibility overview session. I'll have a lot of this content in greater detail.

But the key thing is obviously there's multiple types of applications that work on different frameworks on Mac OS X. And the accessibility API can work with Carbon applications, but there are some caveats. Essentially, we've been telling developers of Carbon applications to use standard controls for a long time for other reasons besides accessibility. If you were one of those developers that took our advice, you're going to be in pretty good shape because we're able to make a lot of the standard controls accessible in Jaguar and Panther. However, there's a long story.

We've had a sort of past of Carbon developers doing their own controls. If you do this in your application, you will have to actually do some work to support the accessibility APIs in your application. The other thing is important is your app should support Carbon events as well.

And that also is one of the ways accessibility API works. The other thing that is sort of an area of difficulty and it's something that we don't have total control over as much as we'd like to is that in Carbon, there's also a history of people doing their own text engines or you name it, licensed text engines. They roll their own.

There's several different ones available in the OS and LTE. There's text edit. And it's too big of a problem for us to solve on our own. So Carbon developers want to use the parameterized attributes are going to have to add some support into their text engine for supporting these APIs. It's something we'd love to fix if we could, but it's too big a problem.

It's too big a problem for just Apple to tackle. We're going to have more information published on this topic on Tech Note 2092. And that is going to be forthcoming and it will have lots of detail on what Carbon developers need to do to properly support the accessibility API.

Good news in the other frameworks. Cocoa, it's a great story there. The Cocoa team actually modified AppKit to support the Accessibility API last year and they continued, obviously, to increase that support. And because of the way Cocoa works, essentially you're always inheriting from parent classes applications that may not have been aware of accessibility. And say, for example, Mac OS 10.1, when those applications are run on X.2 or in Panther, those applications will be accessible. There's still the possibility for Cocoa to be accessible.

There are ways to make those custom controls accessible. Java applications, an interesting story here, is you don't need a Java accessibility bridge like you do on other platforms. What a Java accessibility bridge is bridges out the accessibility from the platform native side where the assistive technology has to run over through to the sandbox. We don't need that on Mac OS X. And with Java and Panther, you just basically, we've done the work for you. And AWT and Swing applications, we don't need that. applications that use standard controls in those environments are going to be accessible as well.

So let's talk briefly about bringing your assistive technology to Mac OS X. This is more for the assistive technology developers who are looking to engage our platform. There's a lot of ways you can go wrong because it's sort of like a restaurant with a huge menu and we want you to choose certain items.

We have like a chef special to do the right technologies. So I'm going to give you some tips right here. One of the things that we have been dealing with the assistive technology community for three years for Apple and it's a continual point of confusion as to where you should start.

How should you begin coding your assistive technology application? And it's sort of become clear to us and looking at all the options repeatedly with a lot of developers that if you're in a situation where you're trying to bring a new assistive technology application up on Mac OS X, you want to do it in Cocoa.

It's just going to be faster and more efficient for you. You get a lot of platform benefits by using it. And you also get a lot of stability because the way the architecture works where Cocoa can actually innovate and change and you'll inherit those advantages. And the rapid application development aspect of Cocoa because a lot of assistive technology developers are small shops and this is going to be tremendous productivity gain for you guys. So we really recommend you take a look at Cocoa.

If you're making an assistive technology device, such as an alternate input device, a head mouse, an eye tracker, a switch, I think that one of the problems that we see a lot in this community is you guys use a lot of legacy I.O., serial, PS2 mice. The reality is Apple sort of really helped catalyze USB when we introduced the iMac a long time ago.

USB is ready. So, you know, if you have a PS2 assistive technology device, you're going to want to really think, how do I move it to modern serial I.O.? In most of your cases, it's going to be USB, but there's also interesting possibility with things like FireWire, where an eye tracker, instead of being this large multi-thousand dollar box and specialized custom image processor done in hardware, could be a relatively inexpensive FireWire camera with, you know, using the real-time capabilities and the high-speed throughput of FireWire, able to actually pump its data into... software that's running on Mac OS X, which is using maybe our platform tuned DSP and math libraries to do the actual calculations to track the eye.

So then you can take, by using the proper I.O. and clever software, you can essentially factor your problem from a very expensive hardware implementation down to potentially very inexpensive and affordable hardware. The other interesting thing is when doing any of the new serial I.O., please follow the standards. It doesn't do any good to do the I.O. and then not follow the standards. And this occasionally happens where people deliver an excellent PS2.

It's not a very expensive assistive technology, but they'll make it work like a keyboard. So when the user hits the switch input device, it just says letter A, letter A, letter A. And then it becomes a complicated software problem to determine, okay, where'd that letter A come from? Did it come from the assistive technology device versus did it come from the keyboard? And software, layers of software have to be put in a place to decode that.

Where if you follow, for example, the USB HID spec, which has all sorts of input devices, you could say this is a button. And that's much clearer than having a... a switch emulate a piece of a keyboard. There's also interesting for wireless opportunities such Bluetooth and also force feedback. We support force feedback in Mac OS X. And I think that could be an interesting dimension to provide feedback back to the user when using an alternate input device.

And we have some speech technology that's available in the operating system, a really good speech technology. We've been working on it for a number of years, and that's available for assistive technology developers to do text-to-speech. It's also available to do speech recognition. And I'd like to invite Kim Silverman up to the stage, and we're going to quickly do a configuration adjustment, and this is sort of an impromptu ad today, and do a quick demonstration of some of the excellent speech capabilities built into Mac OS X. Hi, Travis. While Travis is setting up, there are a whole lot of speech features built into the operating system. We don't have time to show you even a small proportion of them. We're just going to step through particularly what's relevant to the accessibility aspect. API.

Thank you. So there are speech synthesis and speech recognition capabilities that use that. Let's talk about the speech synthesis first. Excuse my back, I'll be looking at the screen here. I'm going to go to the system preferences and the speech preferences, where in the spoken user interface tab, I can turn these things on.

The first thing I'll turn on is selected text when the key is pressed. It's going to ask me for hotkey. I'll use F8, and that means in any application, if I have some text selected, I can speak it just by pressing F8. So we've got some... News, which I just brought up before the session today. I'll press F8 and we ought to hear this.

Security talks between Israeli and Palestinian leaders are making good progress. So there's selected text when the key is pressed. And this is one of the disability features that, like many, become popular amongst people who don't have disabilities as well. It means if you're typing a document, for example, and you want to hear how it sounds, you want to proof it, you just select it and hit your hotkey and it's read back. That is useful for detecting spelling mistakes because spell checkers can only detect typos that don't make words.

If you ask any cognitive psychologist, he'll point you to a lot of data showing that most of the typos that we make make other real words and therefore spell checkers can't catch them. Then we have this item here, text under the mouse. This again uses the Accessibility API. And when I turn it on, you'll hear it speak already the name of that checkbox. Text under the mouse. And as I move this around, it will speak other controls. SpeedTrack Ignition. I'm not doing anything to launch this. It's happening spontaneously. Default Voice.

Spoken User Interface. OK, we'll turn that guy off, otherwise you'll drive us crazy. Now let's go over to speech recognition. If ever you try out the speech recognition, the first thing you should do is give it a chance to normalize to the way your voice sounds in the acoustic environment in which you're using it.

We have built in automatic tracking and adapting to the acoustic environment, but you can speed it up. I'll show you how. And I'm going to do it now because I'm getting echoes from the back of the wall, which are quite a long delay, and it's got to learn about that.

So we built the speech recognition to work with the microphone that's built into most of our machines. iMacs, iBooks, PowerBooks, Emacs. Tower machines don't have a built-in microphone. You can get a number of third-party USB head-mounted microphones. I'm using this one, which was produced by our buddies over at VXI. Those guys really know what they're doing. There are other brands you can use as well. So you go to the speech recognition preferences, to the listening tab, and click on volume.

Now, as I'm speaking, I can adjust the volume so that my speech peaks just go into the red, and then my task is to read down these commands. It's listening to me now, and as it hears me say one of these and recognizes them, it will flash. What time is it? Quit this application. Open a document. Show me what to say. Make this page speakable. Move page down. Hide this application. Switch to Finder.

So ostensibly, we just let you set the volume on the microphone, but what you don't know is we just set nearly a thousand parameters there concerning spectral properties of this microphone and me and this acoustic environment. Now we're ready to use the accessibility features. I go to spoken user interface. Whoops, no I don't.

Speech recognition commands. And here there are a number of sets of commands that can be spoken from the speakable items application. Oh, I should turn that on first. So we go to on off, turn on here. This brings up speakable items and there's our old familiar command window. And let's just zoom in onto that for a moment. Woo, there it is. I like that.

and I don't normally do this, but I will turn on speak confirmation, which means it will speak back to me anything that it hears me say. What time is it? What time is it? It's 11:36. Okay. This is speaker independent. I've not trained this to my voice other than the normalization that you just saw me do. And that all of you will do when you try this in your hotel rooms tonight, won't you? Yes. All right. Down here, turned off by default is front window commands and menu bar commands. Both of these use that Accessibility API.

So we'll turn this guy on, and now I can speak through any front window commands in any application that's using real controls. I'm showing you this because you're going to keep using real Apple controls, aren't you? Default voice. Default voice. Spoken user interface. Spoken user interface. We have not programmed anything special into these preferences, so we can go outside of our preferences. It will still work. Show all. Show all. Let's go and do another one. I'll just pick one at random. Let's do doc. Doc.

Magnification. Magnification. So there you can see that speech was developed as a client for the Accessibility API. Travis mentioned that it gives you and us common ground. We use speech as a client so that we could make sure the Accessibility API worked before we gave it to you. It also means that if you use standard controls, you get a lot of accessibility in your application just for free. And through the Accessibility API, you can add even more. Okay. I think that's enough for now. Yeah.

Thank you. Go back to the slides please. So yeah, it's pretty cool stuff. We've got great speech technology in Mac OS X and it builds on top of the accessibility APIs to do some very interesting things. So let me talk about the last point. I've got to kind of hustle through it. Mac OS X has a unique visual pipeline. We have a compositing windowing system and also a windowing system that's very aware of the idea of having to layer things on top of one another. And this is sort of unique in the industry.

I don't want to go into technical description of how the actual pixels flow to the frame buffer, but the key thing is that there's one process that always knows what's going on in the frame buffer. And that process is lowest level primitive as a concept of a window. And this means assistive technology can do some very interesting things and also solve some problems for assistive technology. One thing is that the windowing system does support Z-ordering, so it's very possible for assistive technology applications to need to be always on top.

To have to do that, it's just basically a windowing system. So it's a very possible for assistive technology applications to need to be always on top. To have to do that, it's just basically a windowing system. So it's a very possible for assistive technology applications to need to be always on top. To have to do that, it's just basically a windowing system. is basically setting in the proper layer in the z order stack and then applications can't put their content on top of a piece of assistive technology content that needs to be always on the screen.

And then also the compositing gives all sorts of opportunities for the visual presentation of assistive technology because in many ways there's always a fight between screen real estate, between assistive technology, and the real estate that the application that's being adapted assumes that it's using. So I actually want to show you a really quick demo of a piece of sample code that actually sort of shows an interesting way to maybe solve this problem.

So let me go to, yes, that's right, the machine. So we're going to bring an application that an engineer wrote that's showing something called an overlay window. And this was just a simple application that I had one of our DTS engineers who works with Cocoa come up with that's going to essentially demonstrate a prototype, if you will, of what's called a scanning application, assistive technology, which is software that usually works in conjunction with a switch, that the user will hit the switch, and basically a palette will pop up, and then a selection will start iterating over that palette.

They hit the switch again, and then they can select an item of maybe actions that are encoded in that palette. And the interesting thing is, you know, this is showing the idea of assistive technology being able to move out of the way of applications because it's able to use Mac OS X's ability to composite and do transparency and blending to where this could easily be made as the user needs it. It could fade forward and begin its scan, and then it could fade away when the user needs to maybe view a web page or something underneath it.

And also one last point is this little sample is about $2. hundred lines of Cocoa code I believe, if I remember my numbers correctly. And the interesting thing is that one of the things that typically people need to do in scanning applications is support a large number of tiles or different little iconic representations of the actions.

And the interesting thing about the power of Cocoa and Mac OS X is the fact that I'm able to put a variety of different file formats into this, my little scanning application here, and the system just all displays them without me having to know anything as a developer how to decode them.

This will even work for PDF files. So I could use a, if I was a scanner, software manufacturer that did an assistive technology scanner, I could then use an image tool like Illustrator or Freehand to create my high quality artwork which would then be dynamically scaled down and have the best possible vector fit for the piece of assistive technology. This is a piece of sample code available up on the DTS website. So let's go back to the software. Last slides.

One last point I want to make is that there's an issue that is sort of happening right now between assistive technology and the propensity for applications to begin to grab the full screen, to take over the display. It's a functionality that we offer in Mac OS X and a lot of games use it.

And it creates a problem that, let's say, for example, someone wanted to use an alternate input device, which required a piece of assistive technology software running in a computer, to be able to display its content on top of maybe the game that's running. That would be difficult to do if the game said, the screen is mine.

So here's sort of a recipe or recommendation if you're a game developer or you have an application that does grab the full screen, how you can use the Z order capability of Mac OS X's compositing window model to essentially put your window at a particular level to allow assistive technology to work.

It's pretty basic. Instead of taking over the screen, you just simply hide the menu bar. You then hide the dock. Then what you do is you bring up a full-sized window without any sort of borders or controls around it, and you put it at a particular level, kind of a screen-soluble level. Plus one.

And there's another layer that's above this that essentially allows assistive technology to appear above your application. And this can be like 2D, 3D. It can be multimedia. So in theory, you could have a piece of assistive technology with a semi-transparent menu sitting on top of maybe a game, a 3D game, if it was designed that way. And the window layers are defined in CG window level.h, which is in the Quartz 2D headers.

So in summary, we've basically covered a lot of ground here. We've talked a little about accessibility from the aspect of why you should do it. We've talked about what technology Apple's built into the operating system to accommodate accessibility, both from a user and developer perspective. We've shown you some new APIs that we've put in Panther to really benefit assistive technology developers who need to get at text attributes. And then additionally, I've made some recommendations for both application developers and assistive technology developers, how they can better leverage the technology portfolio on Mac OS X.