Good afternoon and welcome to this session on USB. This is something like the fourth or fifth year that I've spoken at WWDC about USB. Most of the people around Apple know that USB is one of my favorite subjects. I worked on the original USB team that put it on the iMac, and I've got to tell you, even to this day, I still have a lot of fun with it.

So today we're going to talk about, primarily, the things that we're doing with regard to listening to developer feedback, collecting information, how we're addressing their issues, the kinds of tools that we want to work on and we are working on in the future. And we've got a lot of great stuff to show you today. So having said that, let me bring out Rhoaads Hollowell and Fernando Urbinan.

Welcome. Today we're going to do things probably a little bit differently. Rather than trying to tell you the basic building blocks of doing USB development for Mac OS X, we're going to assume that you know some of the basic building blocks, and we're going to discuss some of the issues that we have found over the last year or so that have been the questions and stumbling blocks for developers with USB drivers in X. We also want to share with you some of the new tools that we have developed and some techniques that we have put together over the past year or so for doing some sort of tips and tricks, if you will, with USB development.

So here's what we're going to teach you today. We're going to show you what documentation for USB development is available, which for those of you who are new to USB development for Mac OS X, this will be the place where you'll start. We're going to show you how to use open source and the Darwin repository to get access to the USB code.

We're going to teach you a little bit about I/O service termination, which has been one of the stumbling blocks for people who are developing kernel extension drivers for Mac OS X. But we're also going to show you some tips on how to develop a codeless kernel extension. If you came to the I/O KIPT managing kernel extension session, you heard Dean talk about that you could write a kernel extension that only has the the the the p-list file, and did not contain any code.

We're going to show you some specific examples of how you can use that to help you with USB devices in Mac OS X. We're going to teach a little bit about how to diagnose and get around exclusive access errors, which has been one of the biggest problems that our developers have seen with Mac OS X. We're going to show you a new tool, USB Prober for Mac OS X.

And we're going to do some live two-machine debugging to show you exactly how easy it is to do two-machine debugging if you have to have a kernel driver. So first of all, documentation. If you are new to USB development with Mac OS X, you need to start with the basic Darwin documentation. Then you move on to understanding sort of some fundamentals of I/O Kit driver development. And then finally, we encourage you to look at the document for working with USB device interfaces. This document tells you how to develop USB drivers that run in user space.

And this is where we prefer that most people do their work. We prefer that you stay out of the kernel if you can, but we understand that sometimes you have to do some things in the kernel. And finally, the overall USB technology homepage is listed here, and all of these URLs will be available on the main URLs page for the conference.

There's some tips and tricks documents that we've developed, and we will be developing more, but these are the ones that we have right now. We have tips on how to diagnose matching issues with USB driver matching for Mac OS X. We have a document for how to make sense of the various IOCit error codes that you might see coming back from USB calls you might make. And issues that people have run up against with boot time loading of their kernel extension drivers.

Okay, first of all, let's talk about open source. The Darwin repository contains the entirety of the USB code that we develop at Apple Computer. At least, we on the USB team develop. There are some drivers through other groups on the team, but all of the code that includes the I/O USB family, which the USB core team is responsible for, is open sourced.

The CVS module for this is called IOUsbFamily, and this is a live CVS module. Anytime we check in changes to the USB family, those changes are available for developers to look at immediately. We maintain tags in this repository, and these tags use the Mac OS X build numbers as identifiers for a particular set of code modules in the USB family that match a particular release.

So, for example, if you have the latest version of 10.1.4, which is, and you look at the build version in the About This Mac dialog box, you see that that's build version 5Q125. There is a tag in the IOUsbFamily repository that has the same name, IOUsbFamily-5Q125, and that tag corresponds to the version of the USB code that is in that release of Mac OS X.

Similarly, if you have the Jaguar CDs that were handed out, the tag for that would be IOUsbFamily-6C35, and that's the current Jaguar build. And every time a build is released to the public or seeded to developers, we will make sure that we have a corresponding tag in the USB family, so finding the code that matches what's running on the machine should be pretty easy.

So we're going to have a demo of that. Nano's going to show us how to get to the CVS repository on Darwin. Nano? First, you need to start at the new Darwin homepage, which is actually at developer.apple.com slash Darwin. And I'm going to traverse really quickly to where the IO USB family module interface to CVS web is.

All this is also available from the command line, and that's how we use it. But the CVS web interface is pretty easy to deal with. So first we go to the open source project. The USB family is part of the Darwin open source. And right here at the top, you can see the up-to-a-minute source in the CVS repository.

And in the next page, we see that the CVS web interface will allow us to see the actual repository at the top level. On the bottom here, you see all the projects that are part of Darwin. If we scroll down to IOUSB family... We can click on it, and then you'll be at the top of the IOUsb family repository. You can see that we have the CDC driver, the composite driver, our family itself is right here. That's where the USB stack is.

This does not show a file that I just checked in this morning called readme.rtf that I suggest that you read as soon as you download the IOUsb family in order to work with it and see what we're doing. It gives you some hints on how to build a family if you so desire, how to build your drivers unstripped, and any information that we want to communicate.

About our module will be in that file. So anytime that you check it out, you should actually go and read the readme file. At the bottom, CVS Web has this interface that allows you to select a particular tag that you want to download through the CVS Web interface.

And as you can see, here are the tags that we were mentioning. I'm not going to do anything else with it. I just encourage you to go to the web interface and download the family and see what we're doing. Now we can go back to the slide, please. Thank you, Nano.

It's possible with this Darwin repository for USB family for you as an interested USB developer to get live release notes, if you will, by being notified by email anytime anybody at Apple checks in any code to this family. So if you'd like to do that, here's how you do it. You send an email to Majordomo here, you subscribe saying I want the logs for USB family, and whenever any of us commits any changes to the repository, you can find out all about it.

And I added this information to the readme file, so you don't need to copy it down. There you go. We also encourage anyone doing USB development for Mac OS X to subscribe to the USB public list. And the way to do that is to just go to lists.apple.com and subscribe to the USB list. It also contains an archive, so if you have any kind of beginner-type questions, we encourage you to search the archives.

And remind you that if you use this... If you use this USB mailing list, that anytime you reply to someone else that's replying on the list, especially if it's an internal Apple developer, please reply to the entire list. Because it's difficult for us to try to manage individual private communication channels.

Now, I want to remind you, the IOUsb family repository contains A number of Apple drivers, the Apple Composite driver, the Apple Hub driver, the HID driver, the keyboard driver, mouse driver, and so forth. The naming conventions that we use for these drivers, some of them begin with the letters I-O. For example, the I-O USB HID driver is a generic HID driver that manages generic HID interfaces.

If the name of the driver begins with I-O, then we encourage you to use this driver and subclass it. And we guarantee that as we move forward through different versions of the operating system, your driver, which is a subclass of this driver, will remain binary compatible from one version of Mac OS X to the next. If, however, the name of the driver begins with the word Apple, such as the Apple USB keyboard driver or the Apple USB Composite driver, this is not a driver that is intended to be subclassed by third-party developers.

And we don't guarantee that such a driver will remain binary compatible with future versions of the operating system. However, because all of this source code is open sourced, feel free to borrow the source code from our drivers to develop your own drivers. But just do not subclass it if the name of the driver begins with Apple.

Now I want to talk a little bit about I/O Kit termination because this has been an area that has been problematic both for some internal Apple drivers and for third-party drivers. And specifically this is what happens when a device is unplugged. Beginning with OS 10.1, there were some new I/O Kit mechanisms that were added to help facilitate the process of terminating a driver, an I/O Kit driver, when a device actually went away. When your device is unplugged from the bus, we terminate the nub for that device, which causes a recursive termination to terminate the drivers and so forth. Before 10.1, it was difficult.

It was difficult to cleanly terminate if you had outstanding I.O. With 10.1 and going forward, there are now two new I.O. service methods, will terminate and did terminate. One is called near the beginning of the termination sequence, and the other one is called near the end of the termination sequence. And by using these two new methods in your driver, you can cleanly get yourself out of the I/O Kit tree without panicking the kernel, which was one of the problems.

We recommend that you... no longer, particularly in kernel drivers, use the kio service message is terminated. So, for example, your driver can contain a will terminate method, and when this method is called, your driver is inactive. If you call is inactive on yourself, you will see that it returns true.

At that point, you need to cancel any outstanding I.O. that your driver has pending. An interrupt read, for example, or any kind of asynchronous... read or write call to the bus. You need to issue the call to cancel that, but you don't necessarily have to wait for the callback to return.

Then, when you get the didTerminate method, you need to check. Termination is almost complete. You need to decide if you have to go away or not. If all the outstanding I.O., any callbacks, any timers, everything that you have that might be outstanding I.O. is complete, then all you have to do is close your provider, and your driver will automatically get unloaded.

If, however, there's I.O. that is still hanging around, you need to leave your driver open with a flag that says, OK, I still have outstanding I.O., and when that final piece of outstanding I.O. returns, you close your provider, and your driver will cleanly exit. And if you want examples of this, look at all of the Apple-supplied drivers in Darwin, and you will see how we use willTerminate and didTerminate.

Now, Codeless Kernel Extensions. This is something that we've been talking about on the USB list now for a few months, and we have had people interested in this and have been trying to put together some specific examples of how this can be used. This is a case where you create a kernel extension, a KEXT project, that has no code in it at all. It only has p-lists, bundle settings, that will cause I/O Kit to match drivers that are already in the system, whether they're supplied by the USB team or by something else, and it causes a behavior change on those drivers, or a behavior change in the default USB system.

So, where Apple supplies some class drivers for certain types of devices, you can provide a vendor-specific driver that has no code in it, that causes a different driver to match against your device, and therefore changing the functionality of the Apple-supplied class drivers. You can use Project Builder to create these codeless KEXTs, but again, you can make them with no code in them whatsoever.

So, for example, let's say we have a HID device. How about a USB to ADB adapter? And this device normally publishes a couple of HID interfaces, one for a keyboard, one for a mouse, but we're not interested in these HID interfaces. We're not going to use an ADB mouse or an ADB keyboard on this device.

Instead, we're going to attach a special ADB device to this, and we want Classic to be able to grab the USB device, and use the ADB with a Classic driver, a Classic ADB driver, to talk to the ADB device that's on here, and it's neither a keyboard nor a mouse. Well, the problem with Mac OS X is that because the device is a composite device, the Apple Composite driver loads, and then it publishes two interfaces, and the HID driver and keyboard driver match against those two interfaces, and Classic cannot get to this device.

Classic does not, by default, take any composite device. But we want Classic to have this device, and we don't want the Apple Composite driver to have this device. So we need to add the Classic must-seize Boolean to the nub of the device to make sure that Classic takes it, and we also need to make sure that the Composite driver does not try to control the device and publish the two interfaces.

So we need a vendor-specific text. Well, here's the personality that we create for this vendor-specific text. There are two personalities in here. The first one is an IOService driver. We tell it, we give the vendor ID and product ID for the device, and we tell it that the device driver for this is IOService. Now, those of you who have done text development know that IOService is the base class for almost all I/O Kit drivers. In fact, it is the base class for all I/O Kit drivers, and most I/O Kit nubs as well.

And so the thing about IOService that is very interesting for this particular device is that it has a start method that has one line of code, which is return true. And that's exactly what we want. We want a driver that will say that it started, but won't actually do anything with the device. So by providing this personality and telling it that the bundle we are using is I/O Kit itself, we can match IOService to this nub, and that's it.

The driver will just say, yeah, I'm running this device, and will do nothing else. It won't open it, it won't send any information to it, it won't communicate with it at all. Then there's a second personality, which uses an Apple-supplied driver that's been in the system since the very beginning.

And what this driver does is it actually does not even have a start method, it has a probe method. This driver's name is AppleUSBMergeNub. And the probe method of this driver takes its own personality dictionary, and looks for something called IOProviderMergeProperties. And any properties that are in that dictionary, it puts them into the provider dictionary of that driver at probe time.

And then once it's added those personalities to its provider, it returns false, saying, no, I'm not interested in this device. So it comes in, gets probed, changes the property list of its parent, and then goes away without, but has left a side effect in that the property list of its parent has changed.

So by providing this MergeNub driver with a property of IOProviderMergeProperties, this will then add the classic must-seize property to the parent. Finally, because this personality of this codeless KEXT needs to compete with the AppleUSB composite driver at boot time or at root time, we need to specify the OSBundle required string of root, saying this driver needs to compete very early in the boot process. By providing this particular property list, we can now completely change the behavior with this USB to ADB dongle.

Alright, we're going to have a second example. The Apple USB composite driver matches against any device that's a composite device. That is, the B device class and the B device subclass are both zero in the device descriptor. But there are some vendor-specific devices, that is, the B device class is 255, that need the same functionality that the Apple USB composite driver already provides.

And that functionality is twofold. It calls set configuration on the first configuration it can find in the configuration descriptor, and it handles any reconfiguration necessary if the device gets reset. For example, if there's a hub that seems to be misbehaving and we need to reset the hub, that causes a reset. And that also causes any devices downstream to also get reset. So we want the composite driver to match against our vendor-specific device. This will cause the interfaces to get created, and those interfaces might be class type interfaces, a HID interface or so forth.

So, here's a personality that matches against this rainbow dongle that we have. And basically it says, okay, if you see this vendor ID and this product ID, then take the Apple USB composite driver and match it against the device nub. And that's it. And now the Apple USB composite driver will match against a vendor-specific device, and yet we've added no code to the kernel. It's using existing code modules.

Finally, I want to talk a little bit about USB log. People have been using I/O log for debugging purposes with their kernel extensions, and it's not really the purpose of I/O log. It's intended to be a way to get information to the log for severe error conditions. But we do want people to be able to use logging as a way of debugging kernel extensions. So we came up with a replacement for USB log called I/O log.

I mean backwards. A placement for IOLOG called USB log. It uses the same printf style formatting that IOLOG uses, but it adds a filtering mechanism. You can specify levels from 1 to 7 for your log messages, and then there are back-end processes that we'll talk about in a minute that will take those messages and will strip out the messages at certain levels.

Now, the interesting thing about this is that if you have a klog KEXT, which we provide in the SDK, in your extensions folder, then these log messages that you provide will actually get sent to an application and user space, and in this case it's the new USB prober tool that we will show later.

But, if the klog KEXT is not there, then your messages will continue to go into IOLOG like they always have, only they'll be stripped to a certain level depending on how the family is. So, you can have these messages go into prober, and then you can use prober to filter out the messages in more than one way.

Now, the biggest benefit of USB log is that it has a really big buffer behind it, unlike IO log. And so you won't miss, you won't drop any messages, even if you tend to spew a lot of information into this log. It's very difficult to overflow the buffer. And the way you use it is you define a debug level that you're interested in, and then you include the IO USB log code.

Now, one of the neat features of this is that we've set it up so that if you define a debug level of zero, and compile your code, then all of your log messages end up getting stripped out because USB log is actually a macro. And if the debug level is zero, that macro defines to a no-op. So that's a way to reduce the code size of your kernel extension when it's time to ship it, at the expense of not having these log messages in it anymore. And here's a sample usage of this.

Log message. You'll see that one of the things that we did in the sample message that we have done in most of our own code is at the beginning of the printf string, we put a %s, bracket, %p, and then we fill those strings with a get name, which is the actual name of your class object, and the this pointer so that you can differentiate identical class objects from each other. Okay, I'm going to turn it over to Fernando Urbina, and he's going to talk about user LAN drivers. Thank you.

Let me go back one slide here. I wanted to talk about a couple of things related to user space USB drivers or applications. The first thing that you need to do if you're working with a USB device, you need to know when the device shows up and when that device goes away.

One of the questions that we get the most in the list is from people saying that they're writing a user space application, they try to open their device, and they get a KIO return exclusive error.

USB and I/O Kit use open semantics to arbitrate access to a USB device or a USB interface. This means that only one component can have access to the device at a time. And so, when you get access, the first thing, or you get the access once you open the device or the interface.

If somebody else tries to open that device or the interface, they'll get this error, and they won't be able to use it until you close the device. So when you get this error, you... It probably means that another KEXT in the kernel has gained access to that device before you want it, or another user client has a device open.

How can you tell if it's a text or a user client that has a device open? As we'll show in a little bit, you can use USB prover and the IO registry pane of the application to examine the IO service plane and determine who has the device open.

In this example, we have an Apple optical USB mouse, that's the IO USB device, and the indented component there is the Apple USB composite. In the IO service plane, this means that the Apple USB composite is the driver for this device. Underneath that driver, we have the IO USB interface for the actual driver. The IO USB interface is the actual interface to the mouse, and the Apple USB optical mouse has matched to that interface, and it's the driver for that interface.

In the case of a user client controlling, being the driver for a device, in this example we have this scanner, what you will notice is that we have the IOU USB device user client as the driver for the device. The user client is actually a KEXT that provides the interface to the user space applications.

So great, now you've found that some other object has the device that you want access to. How do you solve it? Well, it depends on whether it's a kernel extension or a user client. In the case of a kernel extension, like the examples Rhoaads was talking about earlier, you can write a codeless text with just having a personality in your infop list. The code will then be sent to the device, which will then be assigned to the device and the user will be able to return the code.

and its characteristic that it has of returning true from the start method to trump the class driver that will probably match to that device. And by trump I mean that because it's a vendor-specific KEXT, it'll match at a higher criteria and the system will load that driver instead.

If, on the other hand, you have a user client using your device, and in most cases, the user client that is going to grab your device is Classic, then you have two options. Like we showed, again, you can have a codelet text that says Classic must not cease in its info playlist, and that will just tell Classic to never, ever grab your device. That's a little drastic, and there is a better way of doing it. In our documentation folder under your developer tree in the disk, we have a document called Classic. USB device arbitration.

You can use that document to implement the arbitration scheme where you will be able to share the device with Classic, and so the user will have an overall better experience because his device is not shut out from Classic. In those cases, you will be using the USB device OpenSees or USB interface OpenSees to tell Classic that, hey, please give up that device or interface because I really want it, and then you go ahead and wait for it until you can open it.

In the Jaguar CD that you have, if you came to the Managing Kernel Extension session yesterday, you'll see that we now have a very rich kextload command that does everything that you ever wanted to do with KEXTs. For those of you that are doing kernel debugging in the current 10.1.4 system, it is a pain to generate symbols with a new kextload command. It's really easy, and later on we'll show you a little demo on how it works.

Again, now with the 1.87 SDK that was released a few weeks ago, we have the kernel login KEXT that we're very excited about. And if you use USB log in your drivers, you'll get back to the kind of login that we had in Mac OS 9. And finally, we have USB Prober, and it's just like old times, but even better. I would like to bring out now Nimaa Parivaar. He's an intern that works in our system, sorry, in our team, and he wrote USB Prover.

We just gave him a pile of code, and he went by himself and produced this wonderful application. This is the... Interface for USB Prober. You have four main tabs at the top. The bus probe, kernel extensions, I/O registry, and USB logger. We're going to be going through each of those panes. The first pane, the bus probe pane, shows you all the USB devices that are connected to your system.

We are obviously an application. We're from user land. And so what we're doing is issuing device requests to the family to get information about all the devices that are connected there. We get the device descriptors, we parse it, and we present it in a very straightforward manner. If there are strings, we go and get the strings. We get the configuration descriptor. In this example, it's a hit device. So there is also a hit descriptor that we go and get it.

We parse it, and we go and show it to you and interpret all the fields in the descriptors. It's very handy to know what's there. We're going to be improving the tool as we go along, so that hopefully it's going to become a really useful tool for you to use.

The next tab is the kernel extensions tab. This has two different flavors. In this case, we're showing all USB drivers that are in the system. We're kind of cheating a little bit. We're showing any driver that has USB as part of its name. And we also show the version number of that driver.

This is handy when you're doing developing to know exactly what was loaded in the system. We also can go ahead and show all the drivers that are loaded in the system. So if your USB driver kernel extension doesn't have USB in the name, you can still go ahead and find it.

The next pane is the I/O registry pane. This has two different flavors, again. If we go to the I/O USB plane, it is a representation of the I/O USB plane in the I/O registry. This plane shows the bus topology for USB. If you follow the indentations, it just means that you have a hub that has an Apple extended USB keyboard connected to it and an optical USB mouse, etc. It's handy to see the bus topology.

The IOService plane actually shows the client-provider relationship for IOService. So here you can actually see, like I showed in my slide, which driver is attached to a particular device. For example, in the iMate, we can disclose that device and see that IOService is indeed, like Rhoaads showed in his Codeless Text example, the driver for the iMate to USB adapter.

If we then go and look at the USB token, again, that is a vendor-specific device, and we can go back to the bus probe window, look at the device descriptor, see that it is indeed a vendor-specific device. That means that we really don't have a driver for that device.

However, I have loaded those kernel extensions that Rhoaads wrote, and we see that the Apple USB composite actually matched to that device. The composite driver, what it does is reads the configuration descriptor, does a set configuration, and that created the IOSB interface for that device. So that was a nice way of modifying the behavior of our built-in composite driver to actually drive your device. Finally, we have the USB logger window.

This is where you will see all the messages that are displayed by USB log. We have the different levels that you can set up from 1 to 7. In this case, we're going to leave it at 5, so we get to see lots of stuff. I have a family in this machine compiled. We've all logged in on. So if we go ahead and start, and then unplug the token, We see the hub driver realizing that there was a disconnect and blah, blah, blah, blah, blah, and everything, and you can see how we're finalizing and aborting pipes.

Then when he plugs it back in, we again get a lot of information on what is going on. There's also this nice filter button that you can say, you know, I want to see only the lines that contain token. And he refreshes that, and then you can see just your device. Or even if you want to put in your log some specific, you know, key that you can then filter, it's really useful as well.

You have this and on this Boolean thing there that you can expand your filter on. Finally, the other thing is when you stop it, before you start, you can decide that you want to dump the log to a file as well. This comes in handy when you're actually getting a panic.

He actually flushes the log to a file so that when you reboot, you'll have the last few lines in that file. And then you can decide what was going on. Anyways, we're really excited about this. It is, if we can go back to the presentation, available now. And you can go at this obtuse FTP URL.

I'm going to be sending this URL to the list because it's not in the conference's URL. It was written in Cocoa by Nimma. He did a tremendous job. We're really, really excited about being able to give you guys this because it's written in Cocoa. You can print out of it. You can cut and paste.

And in a few weeks, we are actually going to be checking in the code to our USB family module in Darwin. We want to give it a little test drive inside and fix any obvious bugs before we put it out there. But then it'll be there. Anybody can go and modify it, and you'll send us any improvements, and we'll check them out and put them in.

Now, you know, the message is stay out of the kernel, stay out of the kernel, stay out of the kernel. However, sometimes you have to be in the kernel. So, for the rest of the presentations, I'm going to give you a little bit of kernel debugging overview. And then we're going to go to the demo machines and actually show two-machine debugging. You have two types of debugging when you're developing a kernel.

And after the fact debugging, where a customer tells you that your kernel extension made his machine panic. Or, when you're actually developing your driver, you want to do a little bit of symbolic debugging to see what's going on. In both cases, you need to generate symbols so that you can use GDB to look at your source.

So, you have to build the keck-swift symbol. And then you actually use kecksload, the new kecksload, to generate the symbol file that has the actual addresses of the running kecks. You can do this either locally in the machine that is going to panic in the future. Or, if it was a customer, you can use the addresses that the kernel panic message presented and generate the symbol file. at your, uh, local machine.

How to build with symbols? With the new development tools, you need to put this huge line. Don't worry about it. That's in the README that I spoke about earlier. The idea here is that you're building a kernel extension that is unstripped. The results when you run this

The syntax to generate the symbol files, depending whether you are running on the machine that is panicking or on another machine.

The new user interface presents a very nice multi-language picture that tells the user to reboot his machine or her machine. However, upon reboot, the old panic message with all the addresses that are interesting for you developers is actually saved at this location. However, there is a way to disable this new user interface, and it is done by specifying this command, setting the boot arguments for the machine. Again, you would use these addresses to generate the symbol files and then use GDB on the backtrace to find why it was panicking.

If the machine is not available, what you do is you generate the symbol file, then inside of GDB, you go and you do an add symbol file of your symbol file, and then do a list, that's the L here, of the reference address. And then it should show you, as we'll see if the demo gods are with us, a listing of your code and the line number.

If you have the machine available, I would encourage you to read and go through the Hello Debugger tutorial in the developer site. That gets you going on how to do it, and then all this might make a little more sense. And we should see if we can do it now.

What we're going to do is two kinds of debugging. The first kind is a... The first kind is a debugging while you're first starting your driver, development of your driver. So we're going to attempt to break at the start method of the driver. So we have a mouse here.

That I wrote a kernel extension for. And now, on this machine, what Rhoaads is going to do, he is going to load that extension without the mouse being connected. That'll bring it up into, that'll load it into the kernel, and then we'll be able to generate those symbols. We're going to move those symbols from his machine to my machine.

And then I'm going to start GDB and we're going to set a breakpoint at the start method. Then he's going to go and plug the mouse in and we'll see that we do break in the start method. So the first thing he's doing is he's loading the kernel extension.

And we have about a minute before it gets unloaded if we don't plug the mouse. Now he is generating the symbols. Kexload-S, go ahead. We have mounted the... We mounted my volume so the symbols are here. So on this other machine, I'm going to go and gdb slash mac underscore kernel.

Target Remote KDP. That's what you need to do. Attach. Oh, I need the address. And I think you can go and do that. He's actually hitting the NMI switch now. and let's see if we can attach to it. Connected. You can see on his machine that his mouse is not moving at all. And on my machine, it said connected, so we are in GDB and we have attached to his machine. So now, what we want to do is add the symbol file that he created of the kernel extension that we want to debug. So we go add symbol file.

You can use tab completion, which is very nice. It tells you if you really want to add that. And we... Excuse me. We added the symbol file. Now we can go ahead and put a break in the start method. So we say break. Nanos, Maus, if you hit escape, question mark, it lists all the methods. And I want this. The start method. And it says the break-on was set. I hit continue. And now his mouse is working. He just plugged in that other mouse. And bingo, we broke up in GDB at Nano's mouse start.

I can go ahead now and do a list that will show us where in the code we are. We have broken a line 91, which is right at the beginning of the start method. Hit next, and it'll just single step through the code. Oh, it's going. I hit continue. Why don't you go ahead and break again using the NMI button.

Okay, we are back attached to his device. What I wanted to do next was put a break point in actually the interrupt handler routine. Something that was very hard to do in Mac OS 9 is to debug interrupt routines that are running at an interrupt level. So here we can do break nanos. Oh, that's right, we have to add the symbol file. Thank you, Rhoaads.

Nanos, good. Nanos Mouse. The mouse moved routine is what actually the interrupt ends up calling. Sorry, it's called mouse-- sorry. Move mouse. Hit continue. And the other one. We lost control of it. Well, we had another demo that we can use that is actually showing how to debug after you get a panic and you get the panic back trace. So if this works, when he hits the middle button, the machine should panic. Okay? It panicked.

So what we do now is... features of the kexload command that is in Jaguar is that if you take a listing like this that has a bunch of IOCit modules, which include your driver and all of your driver's dependencies, and each module happens to have a particular address, with the new kexload command, you can actually just, kexload will ask you what the address of every single dependency in your dependency chain is. So you're no longer typing kmod sims dash d this, dash d that, dash d the other.

It asks you what each dependency address is, and it builds the symbol file for you accordingly, so that he can actually build the symbol files on the machine that he's trying to use to look at where the panic is. So how you do that is, again, you'd use kexload minus... Minus S is the directory where you want to put those symbol files.

And then... The kernel extension. Again, this is my unstripped kernel extension.

And finally, Nano's Mouse. 1-E-7-6-9-0-0-0 Okay, it generated those, it put them in slash part 10. So what do you do now? You go again into gdb slash mac kernel. You add the symbol file.

and the rest of the software. You say yes. And now you look at the backtrace You find, you know, the first suspect address. In this case, because Nano's mouse was the first kernel extension in that dependency, that's probably the one that caused the panic. And so we looked and see which address looks like it is the culprit, and we find 1E7 6B 468. That's the fourth address in the backtrace. and we list that.

That's why we panicked, because we had an IO panic instruction there. But again, this tells us that it's in the move mouse method, line 569. You notice 569 is the line after the actual panic occurred. You could actually also display the actual PowerPC instruction by typing the address, minus 4, and then you can see that the address is in the move mouse mode.

and it's a call to the IO panic. You can, again, use GDB to look at the previous 40 instructions. We could also look at the next address in the backtrace, which was...

and you see that it is actually in our interoperable handler routine, line 821, but it's actually line 820 where we called moveMouse, which is the... The reason we panic because we put this IO panic.

Of course, when you have your KEXT, and obviously you're not calling IO panic, you have to do a little detective work to figure out why you were panicking. But at least this gives you an idea of how to use GDB symbol files to determine what was going on.

I apologize with the previous demo. We were going to show you you can use print of the referencing that this pointer, and then you'll get access to all the fields in your method. You can then see what their values are. One of the particularly important fields is the retain count of the object because a lot of panics, and I mean a lot of panics, happen because somebody released the object and you didn't go away. Your driver didn't go away for X reason, so that retain count might be zero, which means that you have been released and you have no business being around. We can go back to the slides, please.

Again, these are resources. There's the USB Implementers Forum.

I don't have a time machine, so, you know, the only thing I can say is tomorrow is the feedback session for FireWire and USB, and we are hoping that you'll come and give us some feedback on what we're doing right and what we're not doing right.

Craig Keithley is the evangelist for USB on FireWire. You probably all know him, so Keithley at Apple.com is his... Email. Again, USB developer list. It's a free resource. We are always hanging around. We try to answer as soon as we can, but sometimes we're busy, so we don't answer right away. But we are always reading the emails, so please use it.