So I'm Thane Norton. I'm with the I/O Kit Team. And I want to talk to you a little bit about designing and debugging device drivers on OS X. What I'm going to talk about today is how you can drive hardware from user space or kernel space, the kinds of drivers that have to be in the kernel. And some ways you can debug kernel level code, and some of the special challenges that are involved in doing your own I/O Kit user client. So what is I/O Kit? I/O Kit is the set of frameworks that Apple uses for driving hardware.

There's a piece in the kernel and it percolates up into user space. Not only is it the device driver model, but it's also a framework that applications can use to drive hardware. Now I/O Kit is only available on OS X. It is not available for the iPhone. If you want to know how to drive devices on the iPhone, you want to go watch the Developing Apps That Work with iPhone Accessories session that happened earlier today. So the best possible place for you to support hardware is in your application in user space.

If you just need to drive hardware from your particular application, you can just link against the I/O Kit libraries, the I/O Kit framework and build the support straight into your application. This gives you the benefit of being able to do a drag-and-drop install and not having to have a special installer. And even if you have multiple applications that share the hardware, I/O Kit can arbitrate that sharing for you, and you don't have to worry about all the nitty details.

Now, if you have multiple applications that need to share the same device support code, then you're probably going to want to create some kind of framework. You can still build that into your application, letting you have the drag-and-drop install by putting it in the resource folder of your application, otherwise you're going to need some kind of installer to put it in the right place, so all of the applications can use it.

The last way to support hardware from user space is when you need to do some kind of services, moving the cursor, generating keyboard events, or other services to the system, you need to have some kind of background application. Launchd is your friend. It is a utility that can be used to create launch-on-demand applications that anybody can connect to. It's very well documented in the man pages, such as launchd.plist, and you want to go to the Launch-on-Demand session, which I'm pretty sure hasn't happened yet. Now what being in user space buys you is your device support will be a lot easier to debug.

Not only will it be easier to debug, when you have bugs your application will be more robust. If your application exits suddenly, the user can relaunch it or the background service can be relaunched. If your kernel driver exits unexpectedly, the user's going to reboot and they're going to lose whatever they were working on. You also have a lot more access to system services. You can choose to use Objective-C, which you cannot do in the kernel.

You have access to better logging facilities, such as the ASL log system. You can create your own log file and log your messages out to that. And you should not be able to cause panics. This is not to say that you won't, but if you do you're going to want to log a bug against it and then try and work around it.

Now from user space,- you can get nearly full access to any I/O Kit driven hardware. Almost any USB device can be supported. You get much better memory management. You can actually page memory in and out. You don't have to worry about your memory all being wired down. And one common misconception is that kernel threads get special treatment when it comes to thread scheduling. That is untrue. At a given priority, kernel threads and user space threads get the same scheduling. So that doesn't; being user space versus kernel space does not buy you anything in terms of priority of access.

And you end up with a testing matrix that is very similar to any other application. You do some testing on both processors on all the versions of the OS that you support. When I say all the versions of the OS you support, I would highly recommend only supporting the most recent revision. Have your users update to whatever the latest revision is. It collapses your testing matrix significantly.

Now you can't drive hardware from user space if the hardware is supplying services to someone in the kernel; for example, if you are writing a file system driver. If you require access to other kernel resources that are not published through I/O Kit, you're going to have to be in the kernel to get those. And if you need to respond directly to primary interrupts, that is to say, if you are a PCI driver you're going to have to be in the kernel. There's just no other way to handle it.

If you have to be in the kernel, you're going to need to write a kext. Even if you're just thinking about writing a kext, if you've already written a kext and you haven't done this already, go read the Kernel Extension Programming Topics. It has a lot of very good walkthroughs. Going through that over the course of a day, you can get a very good workflow for building your kext, installing it on your test machine, testing it, debugging it. It's an excellent set of tutorials.

All memory that you allocate in a kext is, by default, wired. What that means is if you had a memory leak in your driver in the kernel, you're going to take the user system down just by depriving other applications of memory. Memory from user space is not wired.

As I was saying, user space applications can page. But, since the driver, since kexts, expect memory to be wired, you have to, before you do any direct I/O on memory passed in from user space, you have to make sure that it's wired down and then unwire it after the operation completes.

When you're in the kernel, logging is very limited. Most of the system logs use some kind of ring buffer. Which means to you, that if you try and log a lot of messages, you're going to overflow the buffer, the messages are going to get garbled, things will be dropped, you're not going to see what you expect to see.

You can't use ASL, which is a very nice logging system. Now you can use kprintf to do FireWire logging. This is not a solution that you can deploy. This is something that is very handy to use in a test setup, so that you get all of the message coming, the messages coming out of your driver. If you want to learn how to do that, the place to look is the man page for fwkpfv, which is the FireWire kprintf something.

Your test cycle's going to be longer. Often it will require a reboot, especially when you encounter a bug, because you'll have to be rebooting your system. It requires advance preparation. You need to have two machines. That's going to be your best debug solution. You're going to connect them either through Ethernet or through FireWire. Ethernet tends to be more convenient, because all your machines are hooked up all ready.

FireWire is available earlier in the boot time, so it just depends on what kind of environment you need. Additionally, you can download the kernel debug kit for the specific kernel that you're running on. And that will give you the ability to step through Apple code as your driver uses it.

So you can see what's going on and who is expecting that pointer to be not null. Again, read the Kernel Extension Programming Topics. This will show you how to set up a two-machine environment. If you wanted to use FireWire, another FireWire utility, fwkdp, very handy for getting FireWire up and running.

You must test your kext on every supported kernel, every release. I tell you from experience, it is very easy to build a kext that will not run because you linked against something that isn't there. Your kext also must match the kernel architecture that you're running on. You don't get to choose if you're going to run 64-bit or 32-bit. If it's a 32-bit kernel, you run 32-bit. If it's 64-bit, you run 64-bit. So this expands your testing matrix slightly. Each of these tests is going to be a little more involved as you're testing to make sure that your kexts runs on the various kernels proper.

One of the problems you can run into is that various functions have changed, for example, memory descriptors. They had to change in 10.5 to support x86 user processes, or 64-bit user processes. The kernel was still 32-bit, and so we had to deal with the memory address translations. To do that, you have to use conditional compilations and availability macros.

Weak linking doesn't really exist in the kernel for all intents and purposes. Now when you're deploying a kext, one way you can support multiple platforms is by having multiple kexts nested inside of one package. Dean will talk about this a little more later, but it can make things very easy for the system to find the proper kext.

You can fairly easily support all Leopard or better kernels with one kext. You can support earlier kernels. And if you want to do that, you're going to want to go back and look at last years Creating I/O Kit Drivers for Multiple Architectures and OS Versions. They have a very detailed overview of how you have to link them all together and munge them so that they work. When you're working in the kernel you have to deal with panics.

Panics are even harder to debug than your regular kernel environment. You've got a static slice of the operating system. You can poke around, you can look around, but it's sort of like CSI, you're trying to figure out who killed you. Again, two machine works well if you already have it set up. Once the panics happen, too late.

One thing that can be very handy is having your system dump core. What this means is, it, you can set it up so that it will write out the kernel memory to a core file that you can then look at later, or at a different location. So, anyhow, if you have a panic that happens infrequently, you can set up a bunch of test machines and when they crash, they can dump core and you can take a look at them.

If you want to know how to set up a kernel system for dumping core, you want to see Tech Note 2118`. That will show you how to set up an Ethernet core dump. And again, fwkdp is how you can dump core over FireWire. All of the panic debugging is covered in Tech Note 2063. It's a very good place to look to figure out what went wrong.

And it is another one of those things I would highly recommend that anybody working in the kernel reads. So why do you work in the kernel? For whatever reason you need to have closer access to hardware. You need kernel resources. The kernel, being in the kernel gets you that special access. It's more challenging than user space. I'm sure that's not a surprise to anybody here.

It opens up, you up to a whole new class of defects, panics, which can deprive your user of the use of their system. And you get much more limited access to system resources. You can't hit the hard drive, for example, and access config files. If you need a custom interface for crossing the user kernel boundary, you may need to write your own custom user client.

This is even harder to debug than a kext, because you've got two separate processes in two separate memory spaces with two separate debuggers attached to them, and figuring out who's calling whom can be very confusing. There is almost always a better choice. I would highly recommend that you not do this.

Code running in the kernel is inherently trusted. So if you create, the reason you create an I/O User Client is to cross that trust boundary. If you make your own user client, you become the gatekeeper. What that means is, you have to validate everything. Any data that comes in from user space, you have to check to make sure that it makes sense. Any pointers that you pass out, you cannot trust them to be the same pointer that comes back in.

Even if just a connect, an application connecting to your user client, if that could elevate privileges or affect another user task, you need to verify that the connecting application has the right to access your user client. If you want to know how to do any of this, the SimpleUserClient and the AppleSamplePCI driver are both very good examples.

And this is another big reason you don't want to do it; your test matrix explodes. You have to test all the various architectures of user clients on all the various architectures of kernels, and it just becomes a testing nightmare. You can remove Rosetta support, but this only helps a little bit. It's not a big win.

And now for something completely different. I've been asked by members of the team to give a couple of public service announcements. If you're doing a PCI driver, you want to try and use Message-Signaled Interrupts. They are preferred, because there is less interrupt sharing and it allows us to demux the interrupts, so that only one driver gets called.

The way you enable MSI is by supplying an index that is greater than 1 to your I/O interrupt event source. When you do that, you don't need to supply a filter. So you can use the root class I/O interrupt event source, interrupt event source. If you enable MSI, you disable the standard interrupts. If you've set them up, don't expect them to be called.

This is supported on all Intel hardware. So as long, if you're still supporting Power PC, you're going to have to still do some filtered interrupt support. Power Management: If you're in user space, better performance equals better battery life. Periodic activity is worse than batching your activity. If you're constantly pinging the hard drive, it's going to keep it from spinning down; it's going to use more power.

If you can do all of your access and then let it go to sleep, this is going to be better for your users. Another tool you have available to you is the System Load Advisory API. Now this is as it might sound, an advisory API to let you know what kind of power save is the user in? Is this a good time to do background tasks? Is this a bad time? Is it kind of OK? Some users, the way they use their system, it never is a good time to do background tasks. So you want to use this to guide your behavior, but you're going to want to have some kind of time out, so that, you know, OK, I've waited long enough, I've got to just go ahead and do it.

If you need to know about wake and sleep, the IO Kit.framework lets you subscribe to the system power messages. If you need to prevent the system from going to sleep, you want to use the IOPMAssertion APIs. They can set it so that your screen won't go to sleep, for example, if your users watching a movie. Or that the system won't go to sleep, if the users listening to music, the screen will still darken, the music will still play. And for those of you who know about SafeSleep, this is what happens when the system runs out of battery when it's already asleep.

You should not have to do anything different. It should look just the same to you: the system went to sleep, it woke back up, the hardware was turned off, it got turned back on. There shouldn't be anything special that happens. If it looks like there's something special, there's probably something wrong. Now Power Management's slightly different in the kext. Since any sleep can become SafeSleep, in the kernel, you again, should not have to do anything different. It should look just the same to you.

You can use IOService APIs to join the power tree so that you can be notified of Power Management state changes. Most kexts don't need to know that, so they're not part of that tree. It's important that your kext, anything in your kext, only implements the mechanism. You do not want to hard code policy into your kext.

Have a user space daemon that pushes information to the kext and lets it know what the policy should be. You'll have some default policy in your kext, but you definitely want to be pushing policy from user space. You also may have noticed that in Snow Leopard we introduced something called maintenance wake.

This was added for Bonjour sleep proxy. So your system could go to sleep and still maintain system resources, or network resources, available on the web. Sleep Proxy, the Maintenance Wake is a brief wake with the screen and audio off. It will not last more than 30 seconds. If you try and do something that takes longer than that, you're just not going to get to. And it's only, again, it's only triggered by Bonjour sleep proxy being active. And with that, I will hand it off to Dean.

Hi. Thanks Thane. I'm going to talk to you a little bit about taking your driver sort of the next step. You've got it written. It's working in your lab. But how do you take it from your prototype all the way to being a product? What do you need to do to harden it and prepare it? And also, what resources Apple has to help you if you get stuck along the way.

So how to qualify a driver is an interesting question. And most engineers have a pretty good intuitive sense of things that they need to test, because they know what the challenges were when they were with the driver. But there's a lot of basic correctness tests that often get overlooked, because they seem very basic. But I work with a lot of developers and I see a lot of kernel extensions that have been shipped with some pretty silly common mistakes.

And as soon as we point them out to the developer, it's like, oh yeah, of course, I need to fix that. But I'm going to go through kind of a laundry list of things that you might want to look at before you even consider putting a driver into full QA.

And also I recommend keeping records of a lot of these results between releases, because we've found it's actually valuable to compare from release to release how we're doing, particularly with Power Management and with general performance. It can be very instructive, and you can start to notice trends, and particularly if something steps out of line. So first on the sort of easy checks is, look at the Info.plist.

This is an XML file inside your kext bundle that is sort of used by the kext management system to know under what conditions to load your kernel extension, how to load it, what it depends on, and so on. But there's a few particular properties I want to call your attention to.

IOKitDebug. This is a property you can add to your personalities in your plist, to turn on some additional logging messages in the IOKit system. And its value is the decimal number; I believe it's from zero to 65535. Well you need to either make sure that it's been absent or set to zero when you ship your kext, because you don't want these additional messages clogging up your users logs. And it's slowing down their system, because it has to write these out to the disk at some point.

Did you advance your version number? This is important, because it's one of two properties that the kext system uses to uniquely identify your kernel extension. If you have two different versions of your kext on the system, this is how the computer can differentiate between them and load the newest one. It's also handy for support, when somebody calls in with an issue you'll know exactly what they're running.

In the CFBundleIdentifier is the other property used to uniquely identify your kernel extension to the system. Make sure it's correctly formed. Make sure that you're using a reverse DNS style identifier, com.yourcompanynamehere.driver.whatever. We don't want to see a bunch of kexts out there with their CFBundleIdentifier being com.mydriver, or something like that from one of the examples.

Because, again, if these collide, the kext management system doesn't know that they're different kexts, it assumes that they're the same. Two additional properties that have been misused over the years, OSBundleCompatibleVersion. This is perhaps confusingly named. This is what tells the kext management system that the kext is a library kext.

It is describing the lower bound of a version range that that text is binary compatible with. If it's presenting and interface vessel or lower bound to the range. The actual CFBundleVersion being the upper bound. But if your kext is not a libary kext, and most of you probably are not making library kext, then you do not need this property. Having it in there means the system will keep around additional information assuming it's going to need the link. Use that information for linking later. So make sure that property is absent unless you know you need it, because you're a library.

And then also the OSBundleRequired property. This is used to determine if the booter has to load your driver, or if it can be left later for the kernel. The only reason that the booter has to load the driver is if it's necessary in order to go about mounting the root file system, so that other drivers can be found. So most of your drivers probably don't need to have the OSBundleRequired property, either. But if you've kind of developed your driver by cutting and pasting from some other drivers you might have kind of absorbed these properties along the way and you want to review their use.

Bundle structure. Run Find on your kext and look at every file that's in there. There's really only two files that have to be there: there's the Info.plist and then the binary that actually gets loaded into the kernels. You should be able to account for every file that's in the bundle.

I've seen kernel extensions ship with header files in them. That their build process somehow copied in there. I've seen lots of stuff in there, a GDB debug sessions files left in there. Yeah, go through and do some hygiene there and make sure those files are all removed if they don't need to be there.

And of course, ownership and permissions, you kext won't even load if these are wrong. But this is a good opportunity to make sure they're owned by root, group is wheel, and they're set up so they can't be modified by any other user. Kextutil-tn is a great tool to help you find, sort of structural problems with your kext.

Any kind of an issue that, you know, maybe would be fatal, maybe even not fatal, maybe it's just a warning. But it does; the t means test the kext, perform the strictest test that it can. And n means don't actually load it. So this is just doing all the proofing that it would do before it loads the kext. So that's kextutil on Snow Leopard and later. Prior to Snow Leopard, that's kextload. In Snow Leopard we split out the developer tool into kextutil.

We also have a tool called kextlibs. And this is a great way to find out what your dependencies ought to be. If you don't know exactly what other library kext you should be linking on, the kextlibs tool will actually even give you the little XML blurb that you can copy and past right into your plist to make sure you got your dependencies right. So we would hope you'd use that to make sure your dependencies are as tight as they can be. Now the binary inside the bundle is the PC gets loaded into the kernel. And, but it also contains a lot of symbolic information.

Obviously we have to link that into the kernel, so you got to have all your undefined symbols defined there. And if you're a library you have to have your exported symbols there as well. But you want to make sure that that was built with the release configuration of Xcode.

Because if you have it set to debug, you're going to get a lot of debug symbols. You're going to get like ten times as large a symbol table as you would actually need. So the binary gets very large. And you're actually leaking a lot of information about the structure of your code.

You're also going to want to verify that your architectures are correct. Make sure that every architecture you need is present, and that none that you don't need are present. So there's three that you would encounter in this development space: i386 for 32-bit Intel, x86_64 for 64-bit Intel, and ppc for 32-bit par PC. You should also run nm on the kernel. Nm is the, on the kext, nm is the utility that dumps all the symbols in the symbol table.

And there's a variety of options and other things you can do. I've given you a line here that runs it through a filter to convert the C++ mangled names into human readable names. And then removes all of the undefined symbols. So what you're going to be left with here is a list of symbols that your kext defines. Make sure that they're named correctly.

You really want all your global symbols to have a prefix that uniquely identifies them to your kext. Ideally the reverse DNS that matches your CFBundleIdentifier, but with the periods replaced with under bars. And the reason for this is the kernel uses a single, flat name space. All the kexts, when they link, link against all their libraries, and those all link against the kernel.

If you have any symbol collision along the way, it won't load your kernel extension. So if you have a function called qsort and then a version of our kernel Apps qsort, which I think we actually view now, your kext will not load, because you have duplicate symbols. So you want to make sure that your global names are all well qualified.

OK, so far we've just been looking at the kext passively. Now we actually want to try loading it, and here's some things you can do. Watch the system log and kernel logs while you're loading your kernel extension. Look for any warnings, or errors, or any other kind of messages that come out that seem scary, and go chase those down. Figure out why they're there. The messages that we spit out are usually meaningful. They usually are important. And if you're having trouble understanding what they mean, the various mail lists that our own lists can usually help shine a light.

Also, unload your kext. I recommend getting this working very early in your development cycle, and here's why. When you're developing the kernel, you want to be able to load your kext, try something out, unload your kext, make a few changes, reload. You know, hopefully you won't panic every time and have to reboot the computer. So some of the time unload can save you, can actually make your development cycle much more rapid.

But it's also important to make unload work, because it really helps ensure that you've got a lot of your references right. You're not leaking references to your objects. You're not leaving dangling objects around. So if unload won't work, that probably means that either somebody is holding a reference to one of your objects or you've leaked a instance of an object and we won't unload you.

Also, you know, while you're doing this, make sure that the system sleeps and wakes correctly while your kext is loaded. And make sure your driver will unload after the device has gone away. If you unplug your device after about a minute, your driver should automatically unload. And also try Safe Boot, that's where you boot holding down the Shift key. This bypasses the kernel caches, the kext caches, and actually tells the system to rebuild them.

But that has the booter load the drivers individually as opposed to a cache. And this gives you a way of testing a few more configurations that your users will encounter under some circumstances. Now cycling is a really powerful tool. This is something we use pretty extensively at Apple and we find a lot of things. This helps us harden the code quite a bit.

But it's not just about repeating a task over and over again, it's about watching the impact of that task on the system. If you load and unload your kext one time, it's unlikely that you're going to notice any leaked memory, because it's probably not a lot of memory.

But if you load it and unload it 1,000 times, you can see a pattern. So we have some tools like ioclasscount, ioalloccount, zprint, and top that all monitor various things around the system. And if you can cycle some of these activities, then you can watch for trends. You can look for, you know, some particular zone of memory blowing up, and realize, OK, maybe I am leaking some resource here.

These are important things to fix. As Thane pointed out, kernel memory allocated by kext is, by default, wired. And so that's memory that if it gets leaked, the only way to recover that memory is to reboot the computer. It's gone as far as the kernel is considered, as concerned.

So take leaks like that very seriously please. And of course, you know, I've listed a few things you can cycle here: sleep, wake, attach, detach. But your product, obviously, is going to have some very common operations. Cycle those as well. We give you a tool in the developer's tools called SleepX that will allow you to do sleep cycling. So that is actually a very powerful tool for checking that transition. And set a goal. I mean this is pretty basic test methodology, right? We usually try to start at 1,000.

Sometimes some of our goals are up over 10,000 cycles. It really just depends on how long it takes to get through a cycle of something. But this is another area to keep records. You know, how many thousands of cycles can this go, and let it go over the weekend, and helps you see if your quality is improving.

Okay, so you've hammered out your kext. You've tested it. You've got it hardened by cycling, and now you're ready to put it into your users hands. What do you need to do to do that? Well we have a tool called Package Maker that will create a .pkg file that the Apple Installer knows about.

And it automatically does all the right things to install a kext, right? So if you don't want to have to worry about setting owners and permissions correctly, you know, you don't want to deal with touching the extensions folder, all of those other things that you might have seen on Apple Tech Notes.

The Package Maker will generate a package that does all the right things for you. And as it turns out there's a nice walkthrough that describes exactly how to go about doing this in the Packaging a Kernel Extension for Distribution and Installation. It's a very straightforward title. It says exactly what it's about. But it walks you through packaging up an example kext.

Now, what's actually going on behind the scenes here is that your kext is being copied into system library extensions, which is the folder for auto loaded kernel extensions. Please do not install kext in library extensions. There have been a few discussions on the mail lists about, you know, why doesn't Apple use library extensions, because it should, because other file system domains include libraries as opposed to just system library. And this is a decision that was made early on. We've revisited it a few times. But we want to keep library extensions for future use. We think we might want to use it in the future.

But if you're installing kernel extensions there now, that limits how we could use it in the future. Plus, there's no reason to install them there now, because it doesn't actually mean anything. It's just a folder. So put them in system library extensions if you want them to auto load.

If they're tied to an application, if you have a kernel extension that really is only there in support of a specific application, then you can put the kext inside the app. It can be inside its resources, or plug-ins folder, and the app can load it programmatically by running kext load.

Thane had mentioned earlier that you can nest kernel extensions, and the reason you might want to do that is to have one top-level kext with a family of supporting kexts. We will support up to one level of nesting for kernel extensions. So inside the plug-ins folder of a kext you can have as many kexts as you need for your suite of drivers, and deliver them as if they were one kext.

It's actually pretty handy if you, say, have an older version of the kernel extension that's supported PPC hardware, and you don't want to re-QA it and redevelop it. You can just shift that as a subkext of your main kext or new shiny Intel 64-bit kernel extension. And that, the subkext will be used on the 64-bit, or excuse me, on PAR PC systems, and the main one will be used on Intel 64 systems.

The talk, last years talk that Thane referenced earlier will describe exactly the ins and outs of how to make all that work. After you've installed a kernel extension, or modified one, or removed one, you need to touch the extensions folder. This changes its modification date and allows the caching system to know that it's time to regenerate the caches. You shouldn't directly manage the caches. The caches for every release of MAC OS X have changed.

We've constantly been refining them and it's difficult for you to try to manage them, because you have to do something different on every release. You should not need to. If you encounter a situation where you really do need to manage the caches to get your product to work correctly, please file a bug that would be a bug on us. Because the caches should always reflect what's in the extensions folder. Assuming you've honored the contract of touching system library extensions.

Now if you want to have a kext immediately available for use without requiring a reboot, there's a tech note for that, a Technical Q&A 1319. What it boils down to is you want to send the SIGHUP to kext D. And the tech note, or the Technical Q&A describes exactly under what OS releases you can do that, and how you go about finding its process ID, and, and, it's pretty straightforward though.

Now if you can't use Package Maker, if you need to install your kext through some other installation utility for whatever reason, you can certainly do that. But there's a few things you need to make sure of. First off, make sure that the installed version of your kext exactly matches what you built.

I've seen installers that modify kernel, or modify files that get installed they send them, because they think they need, don't need all the binaries, or they change the permissions in odd ways. You should go and make sure that not only the data, but the meta data, the, the ownership and permissions and ACLs, [phonetic] and extended attributes, and anything else on those files are correct. And also make sure that it does touch the extensions folder to update the mod date.

And there's a common trap here that I'll warn you about. The first time you install a kext, that effectively creates a new subdirectory in the extensions folder that guarantees the mod date of the extensions folder gets updated. But when you reinstall that kext, it's just overlaying the same files. The folder, the .kext folder that was created during the first install doesn't get reinstalled, right? It's already there. So the extensions folder mod date doesn't change in that case.

So that's why you want to make sure that the installer works after an upgrade as well. The second time after the kext is already present, you want to make sure it's still touching the extensions folder, so the caching can take over. And you also want to make sure that if you are doing an upgrade install that if you have any old resources or old files from your previous kext, until they get removed, obviously you don't want anything bleeding through from a previous version, because that can lead to who knows what.

OK, if you get stuck along the way, if you need some help, there's a large variety of resources available in the Apple Developer Community. And I'm going to start you off at the Hardware and Drivers page off of developer.apple.com. This is probably going to be your hub for doing driver development. Pretty much all the resources that Thane and I have talked about today are referenced through there. There are, there are developer forums, developer.apple.com/devforums, and they haven't been used extensively for device support to date.

We do have a forum on there for 64-bit migration, and it is suitable and appropriate to talk about 64-bit kernel issues there. There's certainly the mail lists that are pretty common, the darwin mail lists: darwin-kernel, darwin-drivers, and development that'll be suitable for any of these topics. And then there's application or domain specific ones like USB, FireWire, ata-scsi-dev, where you can ask questions about those specific areas.

And of course, at any point along the way, if you think that something is not behaving correctly, or not behaving as documented, please file a bug report. This is how we know what we need to do. This is important communication. I know it feels sometimes like a one-way street where you file a bug and you never hear back about it.

But understand that that's a voice that comes in and gets added to a lot of other voices. And it helps us understand maybe collectively where we should be putting our resources. So it does help us understand how to focus on, or what to focus on, I should say.

So please don't be shy about filing bug reports if you think you've encountered a defect. You probably shouldn't be filing bug reports just to get answers, right? If something isn't working, because you don't quite understand it, that's not the same as something isn't working, you understand it, you're following the directions, and it's just not working.

If you're not sure and you want some one-on-one help, you can always request a DTS Incident. These are paid support sessions where an Apple engineer will work with you to try to resolve the problem that you're having. All of these methods have been used to great effect by many of the developers I've worked with. So please use your community.