Good Morning. I'm Jerry Cottingham, I'm an engineer on the Core OS IO team and today we're going to talk about what's new in Kext Development. So in my part of the session, I'm going to talk about Kext Development overview, give you a little bit of overview of the Kext development. The Developer ID program, why that's important for you and your attention please, I'm going to call out some things that are in OS X 10.9 that you need to be aware about.

We're assuming that the target audience for this is somebody that's already developing a kext, you think you're about to develop a kext. And if you're here, you know the top three reasons why you need to avoid writing a kext if at all possible. So if you're thinking about it then here's a couple things that you need to think about before you get into writing a kext. Kexts require wired memory, so all the memory that your kext consumes is not available for the rest of the system.

Mistakes are often fatal, so if you run it writing an application, you make a mistake, you crash the application. You're writing a kext, you make a mistake and you crash, you're crashing the whole system so you affect everybody. Debugging is harder. So today, if you're developing an application with Xcode, debugging is very simple, very, very simple. If you're debugging a kext, you need two machines, you need to setup the machine, you need to configure it correctly. The problems that you're dealing with are often much harder to figure out, so debugging is much harder.

So, if you're still not deterred and you still really think that you need to develop a kext, then if you must there's two things that you need to read before you even start writing your first line of code. The Kernel Extension Programing Topics and the Kernel Programing Guide, so make sure you read through those, and if you still need to develop a kext, make sure you talk to us to see if there's any other way.

And, if you're going to write a kext, you're mantra should always be, "With great power comes great responsibility." Now let's quickly go over a little overview of what kext development is all about. So a kext is just a bundle that extends the kernel. Kext stands for: Kernel Extension, and it's only available for OS X. So if you're trying to write a kext for iOS, we don't allow that. And a kext is structured just like any other CFBundle.

Do you see the binary down in the OS folder? So here the Apple sample PCI kext, you're going to see this through all three of our talks today about kext development. You'll see that the executable is down in the OS folder. The Info.plist describes all the properties of your kext and I'm going to call out two here that we're going to discuss later today. The CFBundleIdentifier, so that identifies your kext in the whole kext name space, so that's a unique identifier across the whole system.

CFBundleVersion, so that allows the kext management system to figure out which of your kexts, if it has the same bundle ID to load. Now this becomes very important and today when we talk about some of the other things that we're changing, that you are bumping that version number as you go along.

Okay, so you're part of the kernel and part of that means that you have to buy into extra protection. So, we require specific File and Folder permissions. Kexts must be owned by root and the group must be wheel. Must only be writable by root, and permissions are deep so this little folder here on the right shows you if you did like an 'ls -alR' all permissions all the way through would look like that. If you get any of those incorrect, your kext will not load.

And you can use kextutil -tn which is test no load to verify your kext as you're developing it. Okay new in OS X 10.9, we are now code signing kexts and checking the signature of your kext. In OS 10.9 all kext signatures are verified. Unsigned or invalid signatures are not fatal with just one exception and I'll talk about that exception in just a little bit.

So if you sign a kext in 10.9 and try to run it on earlier versions, newer than 10.8, it will not load, there was some code signing machinery changes in there that will not allow those to load, so just keep that in mind. Valid code signatures will eventually be mandatory on all kexts. So now in 10.9, there's only one place its mandatory and I'll talk about that, but be prepared, be proactive. Let's get your kexts signed and you'll be ready for the future.

And again, kextutil -tn is good for testing your code signatures. Now I have to mention in the seed that you have today, I didn't quite get around to getting that working, so it's fixed in the next seed they will actually test, so you can just try a kext load to verify your signature.

Okay so now I want to talk a little about kext loading. So there's three types of loads here I want to talk about. Autoload. So autoload is basically your IOKit KEXT when it matches hardware that's attached to your machine, the kext machinery goes through, matches the best driver for that, and loads it for you.

Now, where we find those kexts now is in /System/Library/Extensions. So on-demand loading of kexts, so you can do that by an explicit path, so here we're using kextload with our Apple sample PCI kext. So I just give it a specific path. You can also use your CFBundleIdentifier. So here is kextload -b with a big, long bundle ID that we have for our sample kext, and again, if you supply nothing else to kextload, we'll go search for that bundle ID in System/Library/Extensions.

And the other place that your kext gets loaded is from the kernel cache. So the kernel cache is built to speed up the boot of the system and in there is anything that is required for booting, for rooting, anything that needs to be loaded early can be put in the kernel cache. And again, when we build that, by default, we'll look for those kexts out of System/Library/Extensions. So you see a pattern there that /System/Library/Extensions is very important for us finding kexts automatically.

Okay now in 10.9, autoload, on-demand load by bundle ID and kernel cache builds will look not only in /System/Library/Extensions, but also /Library/Extensions. So, and again, if you touch, the /System/Library/Extensions we will automatically rebuild the kernel cache. /Library/Extensions is the same thing except, again, the seed that you have, that will not work until the next seed. We have a fix that I just put in. Okay, and you must sign your kext if you're going to deploy them in /Library/Extensions. So this is the exception I was talking about earlier. So, if it's not signed, we will not load it from there.

Okay. So we've done some research to find out where your kext is already installed recently. Of course, /System/Library/Extensions and those are needed for autoload or autosearch. /Library/Application Support, that's a fine place, and that's where you guys are. /System/Library/Extensions/File Systems, that's where mainly our file system kexts are. /Library/File Systems are your File System kexts.

Library startup items, in the application folder, inside your app bundles, that's a perfectly good place for your kexts and /Library/Extensions. Again, it did not support autosearch or autoload. In OS 10.9, same setup, these are all perfectly fine places and now we have the autoload and autosearch in /Library/Extensions and, again, the caveat there is that they must be signed.

Okay where we want your kexts installed. So the autoload kext that required rooting, booting, early, you know, early load from the kernel cache and autosearching of kexts, again, we want those in /Library/Extensions and they must be signed. /System/Library/Extensions for compatibility. So if you have shipping kexts out there, which you already do, it's fine to leave those in /System/Library/Extensions.

As you revise and start signing your kexts, move them to /Library/Extensions, and anything on 10.8 and earlier, the unsigned kexts, leave them in /System/Library/Extensions, that's fine, all other kexts though, any other kext that doesn't need to be autoloaded or automatically found by the system. Signed kexts can go into /Library/Extensions, so just because we autosearch and autoload from there, if you sign your kext, it's still fine to put it in there. Even if you don't need to be autoloaded.

Do not install anywhere in /System. So in the future, we are going to tighten down access to the system hierarchy, the whole hierarchy down from /System and everything in there. So, be proactive and stay out of there because changes are coming that will require that you not be there.

So all the other common locations that I talked about, applications in your application bundle, all those are fine. So, you may need to have two kexts for compatibility and in a little bit we'll show you actually how to build those kexts, sign them, and deploy them across multiple OS's.

Okay. So, we're talking about signing your kexts, so how do you go about that? The Developer ID Program is your mechanism to get your kext signed. So what is this? Its part of the Mac Development Program, which you're probably already a member of. It allows distribution of applications outside of the Mac App Store.

So a kext is not going to be delivered through the Mac App Store, so this is actually a great place for you guys. And then you get a special Developer ID Certificate for signing your applications and now in OS 10.9 you're going to get a new certificate for signing applications and kexts.

So how do you go about getting this Developer ID Certificate? Very simple. So you go up to developer.apple.com, Resources/Developer ID, you click on the link to request a Developer ID Certificate, you fill out the form, and then once the form is approved you'll go back to the Member Center and in Certificate/Identifier/Profiles you will see this.

This should be active right now, by the way, so you should be able to go up and get your certificate right now, so click on the Developer ID Application and kernel extensions, and you're ready to go. So for more information you can go back up to developer.apple.com and then, also, Session 702 from last year's WWDC, Perry talked about Gatekeeper and Developer ID, so he talked about the Developer ID Program last year in context for applications.

All that stuff he talked about in that session, he gives a lot of information about code signing, is applicable to you guys too. So go back and look at that, there's a great deal of information there for you. Okay. So in the seed that you have now, in 10.9, I want to call out some things that are going to affect you and our customers. The first thing, as I said before, /Library/Extensions, kexts in /Library/Extensions will not load if they're unsigned or if the signature's invalid.

So what this means is that the user will see this no load alert dialogue. Basically, it says the kext is not going to load and the part that affects you is that very last line in there that says contact the Developer to get updated software. There's a second alert that will come out.

Anything that is a kext outside of /Library/Extensions will load if it's unsigned or if it has a signature validation problem, but the user will see another alert. In that particular case, it will say that yes we're going to load it, but please contact the vendor to get an updated version on this.

Also, what you guys will see as you develop your kext and some of your customers might contact you and say, "Hey, I saw this thing in the System Log," so there's some warnings and error messages, so the warnings would be if it's actually going to load the kext, but it has some kind of problem in verification, and the error messages are when you get that alert that says I'm not going to load it at all.

So the most common code signature error codes that you're going to see is the -67030, which means that something in the bundle has been modified. Something that, the kext signature is no longer valid, so if somebody went in and messed with your Info.plist, then you're going to get this error. The other one is -67062, which is the kext is not code signed, which means there's no signature at all on this kext.

So there's all kinds of error codes in the security framework and you can go to the cs.h and see all those. So here's an example of when I took the Apple sample PCI.kext, built it, had it signed, then I went into the Info.plist file and changed just a line in there, and then I tried to load the kext. And so this is exactly what you're going to see in that particular case. So this is an example of a modified Info.plist, invalidates the signature, and this is the message that you'll get.

Okay, in summary, sign your kext with your Developer ID, it's very simple and easy, get out of /System/Library/Extensions as quickly as possible, in fact, all of /System, and if you think that you need to be in /System for some reason anywhere down in that hierarchy anywhere, please talk to us and let us know so we can provide alternatives to that. And get your kexts into /Library/Extensions. Okay, now Dean Reece is going to come up and actually show you how to build your kext, get it signed, and deployed across many versions of the OS.

Dean Reece:Hi. Thanks for joining us today. My name is Dean Reece and I manage the IOKit Team in Core OS, and I'm just going to run through a couple examples real quick.Let's get started with the most basic project here just to kind of set a base to start from.

We're going to build a kext that follows the new signing rules and installation paths and only loads on OS X 10.9.So you're going to need to use at least OS X 10.8.3 as your minimum build for the build machine and this is necessary to get correct code signing. It was a change to the security framework that's available in that build and later OS releases. You're also going to need to use Xcode 4.6 or later, and for these examples I'm showing you Xcode 5 which is preferred.

And as Jerry described, you're going to need that Developer ID Certificate that allows you to sign kernel extensions, and you're going to install that on your keychain so Xcode can get to it.Okay, so you create a new Xcode project and there's two flavors of kernel extensions: The "Generic Kernel Extension", which is typically used for File Systems and network plug-ins.

An "IOKit Driver", which is typically used for autoload hardware drivers.And you'll go through the normal creation process so we'll give it a name, a unique bundle identifier, and you have a project ready to start putting code into.So, we're going to set a few properties and most of these are going to default this way, but a couple won't so we'll walk through those very quickly.Make sure "Architectures" is set to "Standard Architectures". You only need to building for 64-bit Intel on OS X 10.9.

And also make sure the "Base SDK" is set to "Latest OS X (OS X 10.9)".Under Code Signing, this is new for kernel extensions, so you're going to want to change the "Code Signing Identity" to your Developer ID Certificate that you've installed on your keychain. You can set it to this setting which would match any Developer ID.

If you have multiple Developer ID Certificates, then you might have to pick the specific one that is authorized for signing kernel extensions.In the deployment section, another change here, you're going to have change the installation directory. It will default to /System/Library/Extensions, and of course as we said, that is now /Library/Extensions for OS X 10.9.

And the deployment target should match.We're not going to go into the actual coding of the kernel extension, that's outside the scope of this talk, but there's a few quick reminders that I wanted to give you.When you're developing a kernel extension, you may only use headers that you find in the Kernel.framework.

The Kernel.framework is intended for use by kernel extensions and conversely, it cannot be used by applications. So all the other frameworks are for applications, Kernel.framework is for kexts.If you're writing an autoload kext or IOKit kext and you happen to subclass OSObject or any class that's derived from it, which all the IOKit classes are, then you must use the OSDeclare and OSDefine macros that you'll find in the OSMetaClass.h header. This is important for our binary compatibility.

This includes additional symbols that our run-time system picks up and uses so that we can patch up your kext and make it binary compatible with whatever OS you're loading it on. So, for example, if you build a kernel extension using the OS X 10.8 SDK and you fail to use these macros, that kext will only ever load on OS X 10.8; it won't load on OS X 10.9, you'll get symbol errors.And the last thing, is you need to not strip those binary compatibility symbols out of the binary. You can use "strip -S -x". That's safe for stripping a kernel extension binary, but anything more restrictive might rip out those symbols that we need.The project we just described builds one kext.

It will load on OS X 10.9 with no errors or warnings and you're going to debug it using lldb. In a few minutes Brian's going to join us and describe how to do that, but you'll need to be running OS X 10.9 on both the target system and your host system for that lldb connection.And of course after you package it, part of your Q.A. process should be to verify that it is installing in Library/Extensions.

I think that might have come up once or twice.Now let's expand this project a little bit and we're going to add support for OS X 10.8. To do that, we're going to create a new target, and we'll have visit those same settings that we talked about before.But before we do that, we're going to check our version numbers.

Now, this is very important; this calls back to what Jerry told you earlier. If you have multiple copies of your kexts installed on the system, and the system encounters both of them, it's going to pick the one with the highest version number. Because of the way that this project is going to install its kexts, we're going to say that the target intended for Mavericks, for OS X 10.9, will have a higher version number.

And that the one intended for OS X 10.8 will have the lower version number. The reason this works is because the newer kext is going to be installed in /Library/Extensions, and older OS's don't look there so they won't see it by default, they won't know about the newer version and they'll just ignore it.

Whereas Mavericks will see the newer one and load it by preference.Okay, so let's go and visit those same build settings we talked about last time through. Your new OS X 10.8 target is going to have the same architecture setting, again you only need 64-bit Intel there, but your "Base SDK" will be "OS X 10.8" in this case.And of course "Code Signing Identity" will default to "Don't Code Sign", but you should verify that is the case, because you don't want it signed.And the "Installation Directory" in this case will need to be /System/Library/Extensions, and the "OS X Deployment Target" needs to match the SDK.This project now builds two kernel extensions. The signed one is exactly the same as the one we built in the first example.

And the unsigned kext that is being built by the new target will load on OS X 10.8.Now you're going to use gdb to debug that one. That's the divide between 10.8 and 10.9: 10.8 and earlier is gdb, 10.9 and later is lldb.And you're going to verify that that did, in fact, go to /System/Library/Extensions.Since you'll be installing two kernel extensions, it might be a good idea if you were to look at the target OS in your install scripts, see what OS you're installing into and avoid installing the legacy kext if you're installing onto a Maverick system, because it will never need it.And another warning I'll throw out here is in the future as we start to lock down the /System folder, you might actually get write errors. So when you try to install a kernel extension into the /System folder, the write itself may fail.

Your installer script needs to be smart enough to know if it tries to write something there, as long as the one that went into /Library/Extensions made it okay, that's good enough.Alright.So let's expand one more OS release back. In this case we're going to modify the project we just created for OS X 10.7, but one difference is we're going to use Xcode 4.6. You can't use Xcode 5 for this because Xcode 5 doesn't have the 10.7 SDK.As before, you're still going to need that certificate for signing the Maverick's kext.We're going to start with the same project as last time, but we're going to show the screenshots in the older Xcode.

So the second target is going to have slightly different settings, particularly to target OS X 10.7, we have to add the 32-bit Intel architecture for that, for full compatibility. And the resulting kext from that target will load on both OS X 10.7 and 10.8.As before, you're going to make sure that the module identifier, the CFBundleIdentifier that Jerry talked about, is the same for both targets.

And you're going to make sure that the legacy kext, the one that's installing in the old path, /System/Library/Extensions, has the older version number.Let's go and look at these build settings. Again, "Architectures", you need to make sure that its standard for this SDK. So the "Base SDK" is going to be "OS X 10.7" and the "Architecture" is going to be both 32 and 64-bit Intel.And as before, this target needs to not sign the kext.In deployment, again, /System/Library/Extensions is where it has to go for OS X 10.7 and 10.8 to see it.

And the "OS X Deployment Target" needs to match the SDK as before.This project will also build two kexts. The signed kext is unchanged, it's exactly as the previous two examples. But the unsigned kext should actually load on both 10.7, all flavors of 10.7 and all flavors of 10.8, and you will, again, use gdb to debug that.

Oh yeah, and as before, you're going to want to install only the kexts that are necessary. If you're installing to a 10.7 or 10.8 system, you'll install both kexts, but if you're installing into an OS X 10.9 system, you only need to install the signed kext.What happens if you want to target releases even further back than 10.7? It is possible.

You can't do it from a single Xcode project, because there is no single Xcode that spans that many releases. So probably the best approach for you is to take your existing kernel extension project that already runs on OS's up through 10.8 and just use that kext as the unsigned kext that you install in /System/Library/Extensions.

And then use the simple OS X 10.9 project that I described first, and just import the source code and build a second kext that way and that will install in /Library/Extensions.So effectively you're kind of building a line in the sand between Mavericks and older operating systems that didn't use signed kernel extensions, and so it's kind of a clean break in the new kext project and focus on Mavericks and the future.

So, when you're building for older OS's, I couldn't go into all the different permutations, it really depends on what OS's you're targeting, what families you are using within IOKit and so on as to exactly what compatibility you would have. But you need to follow the compatibility guidance that we've released for previous operating systems when targeting those, and again, you're kind of separating out the 10.9 kext anyway.

So, the compatibility guidance we've given you in the past still applies for the legacy kext.And always install only the kexts that are particularly necessary for the target system and also make sure that you're robust against write failures to the /System folder for future.So this is your handy pocket guide for compatibility. The first column here says, OS X 10.7 and beyond. Those are all the various settings you're going to need and each column further to the right takes it further 1 OS release.

This is mostly for the at-home audience that can freeze frame on this and very quickly get all those build settings that they need.All right, and with that, I would like to turn the stage over to Brian Bechtel who's going to talk to you about lldb.Thanks for your attention.

Hello everybody. I'm Brian Bechtel. I work in Core OS doing Panic Triage. I look at panics all day and we have, I know it's a wonderful job [laughter]. We have a saying in our group. [Laughter] And it really is true that after about 10,000 panics, there pretty straightforward. In Mac OS X 10.9 we have a new debugger, lldb. It is a very powerful, very fast, very nice debugger.

It is slightly different. I'll be talking about some of those differences. The big things we get out of lldb are tighter integration with llvm and the rest of the development tool chain. We have a much, much faster debugger. I will be demoing that later. And we're supporting it very actively and we're very much listening to developer reports of issues and fixing them, which is a very good thing.

lldb includes Python as its scripting language. That means you can do almost anything that you can do in Python inside lldb, it gives you way more power than the old macro language in gdb. There is a gdb to lldb command map up on the llvm.org website, which helps you make this transition, but much of the commands are exactly the same for the things you typically do. lldb is the debugger for Mac OS X 10.9 and beyond. gdb is dead.

[Laughter] gdb is gone in Xcode 5.0. There is no gdb binary there. There is no kgmacros file in the kernel debug kit for Mavericks. We have no plans to fix any remaining issues. It is still the debugger of choice for Snow Leopard, the cats basically, if you have a cat, gdb is your friend.

If you have a giant wave or some other California feature, lldb is your friend. This is a demo of how fast gdb is, versus how fast lldb is. I'm just going to issue one command in the debugger while connected via gdb to a 10.8 system and lldb to a 10.9 system.

I've set up two terminal windows here, the light brown one is running gdb and the white one is running lldb. I'm even going to let gdb start first so you're going to see a cursor there that touches that window, I hit Return, then go down and hit the other window. And what you see is lldb is much, much faster at just printing the zone list.

Every other command in lldb has this kind of speed improvement. It's really quite, quite noticeable, and wonderful if you're doing 10,000 panics. [Laughter] All of the standard kgmacros are still there, I said there's no kgmacros file, but the kgmacros themselves, reimplemented in Python, are in the mach kernel.dSYM file in a directory, which I'll discuss later. These are all the standard macros that we use when we're debugging to just get the generic information we need for a panic. They're still there, there are some minor changes in the output format, but everything works great.

For adding kext symbols, on startup, lldb will automatically scan for kexts installed in System/Library/Extensions, the new Library/Extensions, and in the directory /Volumes/Kernel/DebugKit because that's the kext symbols that we give you in all of the symbols for IOKit kexts. You can specify additional locations in the .lldbinit file in your home directory. There's the command line for it. And of course while you're debugging, you can add kext symbols using the addkext macro. I'll have some examples of that later.

Every time you change something you'll have little gotchas, these are some of the ones that hit me that I thought you guys would like to know. lldb likes to pretend it doesn't know how to do addition. Backtick is your friend. Backtick, if you get a, if you try to display something and it says, "I don't know what that is that you're trying to display," put the expression in backticks.

It sometimes fixes that. gdb was very, very sloppy about describing a structure or a pointer to a structure or fields in a structure. lldb is very pedantic about that because it gets very pedantic information from llvm. So if you're saying this is a pointer to a structure, you have to say it's a pointer to a structure, it'll painfully warn you that you probably meant to say that and let you reenter your command.

And you may need to cast a variable very explicitly. We've had issues where you know that this class is called, "MyClass," and you say, "Print MyClass," and instead it says, "I don't know what MyClass is," but if you say, "Class/MyClass," or "Struct/MyStruct," it'll say, "Yeah sure I can do that." Anything you encounter like that, we want to hear about it, so please go to bugreporter.apple.com, file any issues you're having, we really want to hear those things, they're going to the engineers, they will repair them or at least have a reason why.

So one of the great things in lldb is the mixed mode disassembly. A lot of times when you're trying to figure out what happened, it's difficult to translate in your head what is going on in the assembly versus what the compiler did. The disassemble -m option gives you mixed mode of whatever you're asking for.

You can say, for instance, "disassemble -f -m," which will disassemble the entire current frame with both source and assembly, intermixed so you can see where everything is. If you just want to see the current line that you're stopped on you can say, "disassemble -l -m or you can give it a start and address. And you can abbreviate that, the minimum command abbreviation for disassemble is di. So you can say di -f -m.

Here's an example of mixed mode disassembly. The stupidest, simplest example I could come up with. There's a routine in the kernel called, fseventsf-write, which just returns EIO, an error. So this is the example of that routine disassembled with mixed disassembly. Here you see the prologue of the routine. EIO turns out to be the value of 5, so here it's moving 5 into the return register. And here you see the epilogue. So in a much more complex example, this is really, really helpful. Here, you could probably figure that out in a day or two. [Laughter]

The kgmacros now take an additional argument, or series of arguments. You can now redirect output from a macro directly to a file with -o. If you get a macro that has a lot of output and you don't want all that output, you can grep inside the output with -s and you can add additional verbosity to the macros to see additional details about the macro with -v and it, it is cumulative so you can say -v -v -v, you can be anal and see how far out you can go.

So here's an example, if you want to find the load information for just our sample PCI kext, you can say, 'showallkmods," which shows all of the kext information, and then say -s AppleSamplePCI to just get the information on one line rather than have to scroll through a lot of lines.

For security purposes, we do not automatically load the lldb macros in lldb by default. The reason we do this is if you're doing application debugging or you have a malicious kext, and they've put some bad lldb macros in a hidden symbol file or something, bad things could happen. You know? It's the internet nowadays.

So to overcome this because you trust your development environment and your setup, you can add that particular setting line to your .lldbinit file and then it will automatically load the script from the symbol files. You can also create custom Python and put it in the, in the dSYM file in the Contents/Resources/Python directory, and then that will be automatically loaded assuming that you have your settings set correctly.

For faster loading, a lot of times you only want to see your particular kexts, you don't care about Apple's kexts, you don't care about all those other things. Add this line before you do the connection or put that in your .lldbinit file and it won't load all the other things and you just come right back up to Ready To Go. And then at that point if you want to automatically load, or manually load symbols for your kext, you can use any of these four examples of the addkext command.

You can either pass it the UUID, pass it the name and if the name is found in one of the places that lldb searches, it will load it. You can give it an explicit path, an address, or you can say, "Hey, I changed my mind I want them all loaded," that will take a lot longer than it would if you'd just let lldb do it at the beginning. nvram is needed, I'm going to now be talking about two machine debugging.

We use nvram to enable debugging at all. If you have no debug boot argument set, your system will reboot on panic, that's a new behavior starting in 10.8. If you set boot-args using the nvram command line tool as root and reboot to let those settings take effect, you can change that behavior so you'll halt at panic and let you connect from another machine.

So in the old days, back when we had just iMac's or something, we had built-in Ethernet and built-in FireWire. Nowadays, probably most of you out there, have a MacBook Air or something that no longer has built-in Ethernet nor built-in FireWire, so we've added new ways to connect. You can now connect for two machine debugging using a Thunderbolt Ethernet Adaptor, using a Thunderbolt to FireWire Adaptor, using an Apple Thunderbolt display in either Ethernet or FireWire on the back of that monitor. We do not support wireless and we do not support USB Ethernet, so you're going to have to get one of those other connections.

When you connect, two-machine debugging assumes a network port of en0, that's the old behavior from the original iMac, MacBook Pro, Mac Pro, whatever. If your network port isn't that, you have to add, as part of your boot arguments, kdp-match-name=, and then some value. That some value will depend on your connection.

For a network connection, ifconfig, will tell you which port you should use. So if you have a Thunderbolt to Ethernet connector you would use ifconfig to find the connection and then kdp-match-name=en whatever that number is. Here's an example of that. This is the output from the ifconfig command just cut down to a single port.

I'm looking for a port with an inet value that makes sense. So, for instance, here at the conference, we're running off the 10.0 net or 10.1 net, I'm not sure which one. I look for a connection that's 10. whatever, the en1 tells me that the port that I will be using in kdp-match-name=.

So for built-in debugging on a machine like a MacBook Pro with built-in Ethernet, you would just set your boot args as this: debug=0x144, the 100-bit means don't put up the graphic panic screen, just halt. The 4-bit means let me connect to the machine, and the 40-bit means let me connect over a different subnet if necessary. So you can simplify this to 104 instead if you wanted.

For Thunderbolt to Ethernet, you'll notice that same command line has added kdp-match-name= and then an en value. And for Thunderbolt Display, you would do the same thing. I'm giving examples here with en2 and en4, but you actually need the output from ifconfig to figure out what the proper en value is, and you would put that in. Many times this value may coincidentally be there, and it may be the right value, but you have to use ifconfig, you can't just trust a slide at a conference.

[Laughing] If you're using native FireWire, you set kdp-match-name to FireWire. If you're using Thunderbolt FireWire Connector or Apple Thunderbolt Display to FireWire, you have to add an additional command, which is fwkdp=8000hex. That tells the Thunderbolt System not to turn off power to the FireWire chip, which would be rather inconvenient when you're trying to connect. And as always, you need to reboot to let the settings take effect.

So for FireWire debugging it's very simple, you open up two terminal windows. In one terminal window, you enter fwkdp, in the other terminal window you invoke lldb, and then connect to the local host. The fwkdp is a redirecting program that makes it look as if your local machine is panicked, but it actually redirects the commands out the FireWire port to the panicked machine.

It's important to note I used xcrun there. xcrun guarantees that you're going to be running the lldb that's in your current Xcode. You want that because who knows what horrible thing you've got installed in /usr/bin or /usr/local/bin that might accidentally be grabbed. If you use xcrun lldb it will always find the proper lldb for your Xcode.

And I'm giving a couple examples of how you connect to a machine. If you're connecting live, you can either give a name or an IP address. You can use a Bonjour name although there is a warning that with Bonjour you may have a reassignment of an address as you're going along, so that's a little shaky. I tend to either use a fully-qualified domain name or a dotted IP address. And if you're reading a coredump file, the command is "file -- c" and then a path to the file.

In fact, talking about coredumps, I'm going to talk about several things that we've changed recently that are helpful for you. The first thing that you'll notice is we've changed how you do a NMI in Mac OS X 10.9. With boot-args set so that the 0x4 bit or the DB-NMI bit is set, in 10.8 or earlier, you'd just hit the Power key.

Anybody who's done kernel debugging and hit the Power key will discover that the Power key on a MacBook Air is directly next to the Delete key. And then you curse and reboot your machine because you didn't have it setup for debugging. So we changed that in Mavericks. Now it is Left and Right command keys and the Power key. This will make your day-to-day life a lot easier. If you really like the old behavior, you can revert to that by OR'ing [phonetic] in the hex 8000-bit, so instead of debug=0x144, you'd say debug=8144.

You can automatically save coredumps. This works far back in the system, but I'm documenting it to make sure everybody has it because it's a pretty easy process. As root on a host machine, the machine where you're going to save the coredump, you need to create a directory called PanicDumps, change it to permissions, and then launch the kdump.plist, which will load a demon that will sit there waiting to see if anybody wants to send it a coredump to save.

On the client machine, on the machine that you want to panic and save a core from, you can set your boot-args where debug is 0xd44 panicd ip= and then the dotted IP address of the host machine. It has to be a dotted IP address because at the time of a panic we can't do DNS resolution.

We've added some new protection boot-args, which will help you track down buffer overruns, use after free errors, mismatched frees, you know, freeing from a zone that you didn't actually allocate from, and some timing-induced races. -zc, the allocate code inside the kernel for zone allocations, checks the free-list pointers of all the zones for correctness every time you do an allocation. -zp, the free routine, will overwrite freed memory with deadbeefdeadbeef. That will catch any attempt to execute or access freed data. And here's an example of how you would set that: debug=144, for the two-machine debugging, -zp -zc, and of course reboot to let it take effect.

We also have a guard mode kernel zone allocator. This will let you target any zone with a certain size or any zone above a certain size, below a certain size, or in a range of sizes. You set, specify one of gzalloc-size, gzalloc-min, gzalloc-max, or min and max, and then gzalloc-wp. And that will write-protect the allocations rather than unmapping them. So the allocation will still be around, but it will be write-protected.

Zp and zc aren't very much of an impact on performance, I wouldn't do your benchmarking with them set. But they don't really take up too much memory and they don't really take up too much time in the allocation and free. The gzalloc-wp, gzalloc-size stuff, every allocation is going to be put on its own 4k page with an additional guard page, measurably slower, you'll notice it when you're using the machine, but it catches all kinds of bugs, it's a very valuable tool.

Another point I wanted to make is that starting in Mac OS X 10.8, we changed graphics to support multiple buffers. We draw the console output tool buffer in instead of directly to the screen. This was an enhancement needed in order to support the retina display Macs, but it applies across all the machines.

Unfortunately, that means you have unreliable console output to the screen if you panic. So if you have your boot-args set to display text instead of a panic screen, you may not see that text, it will appear to hang. The only way to tell is to attempt to connect to the machine.

For more information about all of this, we have documentation online, we have a contact here from the evangelist group, and the developer forums are actually get pretty good coverage from people inside of Apple. I recommend understanding and debugging kernel panics. The Kernel Core Dumps, I specified all of the stuff you need from Kernel Core Dumps on one slide. So, it's not very difficult.

And there's a new revision to the generating a non-maskable interrupt, which is in the Mavericks section of the developer website, the original version of that is referenced here. We have some related sessions later this week, on Friday in the Pacific Heights Conference Room, they'll be an Advanced Debugging with LLDB, I recommend you go.

So in summary, sign your kexts in 10.9. We've had two people tell you that, it's the wave of the future. lldb is actually pretty cool. I like it. I prefer it over gdb now that I'm used to it. Like any new technology, it takes a bit to get used, but it's very powerful and very nice. We have new ways of debugging for machines that do not have built-in FireWire or built-in Ethernet, and we have new boot-args that help you track down bugs. There you go.