Ladies and gentlemen, please welcome the manager-- OK, now. As I was saying, please welcome the manager of Mac OS X file systems engineering, Clark Warner. Thank you. The stage police are watching me. And I've heard from my spies that when I turn around to jump on the stage, something bad's going to happen to me. So I have to be careful. I've noticed that the stage is higher this year than last, so I think I'm ready.

Thank you. Welcome. So we have, we hope, some really good information for you folks whether you're just a user of Mac OS X, whether you're a developer on top of Mac OS X, or whether or not you actually want to implement a file system or some kernel extension of the file system.

And we hope that by the end of the session you'll have some vague ideas of what you need to do and whether or not you need to do some of these things. So we've got information on the interfaces, we've got some information on differences between Mac OS X and Mac OS 9 that come from our BSD heritage, and we'll also have a little description of some of the differences between the different volume formats that we support in Mac OS X.

We have a relatively significant architectural change to tell you about in Mac OS X involving our buffer cache. And so we're going to spend some time on that at the end of the session, hopefully enough to give you a feel for what we've done and why we've done it. And for those folks who have actually been to a file system session before, then you're going to see a fair amount of review. And we think that's needed for a couple of reasons. One is, as Steve said in the whole keynote on Monday, Mac OS X is becoming very real.

And there are probably, we expect, a lot more folks who have not actually been introduced to any of the technology in Mac OS X who are going to need to hear some of the stuff we've actually talked about in previous sessions. And also, given that it's much more real now, I imagine there may be a few of you who perhaps weren't as concerned about what you were hearing in years past and will want to hear some of that stuff over again. With that having been said, there will be some new stuff.

Let me start by giving you a brief overview of the Core OS. If you went to the Darwin overview session, then you've seen this picture already. In fact, My tech lead has supplied me with this really handy laser pointer so I can point out the coolest part of the Darwin system.

And it goes like this. We want to tell you a little bit about the internal architecture of the file system as well. And this diagram is something you're going to want to keep in your head as we move along in the presentation. So, several key things I need you to remember.

The application environments, Classic, Carbon, Cocoa, they all have their own way of interfacing with the file system. And we're going to talk about those interfaces a little bit at a later time in the presentation. The actual stuff that my group works on is here inside of Darwin, inside of the BSD kernel. And we have two interfaces of particular significance, which we'll talk about in some detail today also, as well as a collection of volume formats that we support.

So, our basic interface is a BSD Unix interface. But BSD traditionally doesn't support in its volume formats some of the sophisticated features that have been available in Mac OS X. So, namely things like resource forks, catalog information, and that sort of stuff. And so we've extended the BSD interfaces to allow for retrieval of that and establishment of that sort of information.

Likewise, inside of BSD, there is a relatively clean abstract layer called the virtual file system layer. And that basically separates the file system dependent from file system independent layer inside of the BSD kernel. And if you wanted to develop a file system yourself because you had a new device type, or if you wanted to develop a new system yourself because you had a new device type, or if you wanted to develop a new system yourself because you had a new device type, or if you wanted to develop a new system yourself because you had a new system yourself because you had a new system yourself, or if you wanted to develop a file system plug-in which will intercept all the file system calls and then deliver information to file systems beneath, this is the interface that you would actually write to. or if you wanted to develop a file system plug-in which will intercept all the file system calls and then deliver information to file systems beneath, this is the interface that you would actually write to.

We're not going to talk in great detail about the volume formats, but suffice it to say that we support a number of them in Mac OS X. We've got support for the BSD fast file system, which was our implementation of the Unix file system, HFS and HFS+, the network file system, AFS by virtue of the Apple share server team, they've done an AFS client implementation, and more to come. Come.

So here's how we're going to do it. Here's our session outline. We're going to do a little status update to let you know what's changed in Mac OS X. We're going to talk about some of the new features of the system since last we got together at last year's Worldwide Developers Conference, focusing on some of the stuff we've done for DP4 and for DP3. And we're going to, as I mentioned, talk about some of the interfaces, the various ways of getting at file system functionality in Mac OS X.

And we're going to talk about some of those differences I mentioned between different volume formats and arising out of the fact that we have a BSD system and a Mac OS X system, a Mac OS system coexisting. And we'll talk, we'll do a review of the virtual file system interfaces themselves to give you an idea of what they look like and how you might make use of them.

And we're going to talk in some detail about the changes in our buffer caching architecture so that, again, if you're one of those Darwin folks that is building a new file system, you'll have an idea of what sort of things you'll need to do to change to work with our new unified buffer cache. The next new feature could be yours. We're going to talk a little bit about Darwin and maybe some ideas about what things could happen next.

So here's our status update. Back at this time last year, we demonstrated to you, for the first time ever in public, a system, a Unix-based system, in that case Mac OS X, booted off of an HFS+ volume. And it was the first time ever in public. And this year, we're happy to report that this is just a standard humdrum, boring old part of the system now. At DP4, and even back in DP3, the primary volume format is HFS+.

The server team has added support for the Apple FileShare protocol, as I've mentioned, and it's actually in DP4. If you look in your demos folder, you'll find an application called TestAFPClient, and you can use that application to mount an Apple Share IP-based volume on your Mac OS X system. Now, it's still a little early and still a little new. You may run into some bugs, but it does work. I have used it.

We also are well underway with the development of a new version of the Unix-based system. We also are well underway with the development of a new version of the Unix-based system. with our universal disk format implementation. It's not in the release yet. It turns out that UDF is very much a kitchen sink of file systems. It has just about every feature any file system person could ever imagine and so it's quite a challenge to build support for into the system. But we're well underway with that. We also have added support for large files.

It's not exposed yet for your use because as a result of the new caching architecture we're actually relying on the virtual memory system for IO in some cases and the VM interfaces are still 32 bit. But all of the file system interfaces now are 64 bit so that we'll be able to support very large files in Mac OS X and the virtual memory system is going to change to accommodate that.

Also important to note is that all of the interfaces into the core file system at the BSD level are UTF-8, take UTF-8 strings. The important thing about UTF-8, it is a single byte character encoding of Unicode characters. Unicode, for those who don't know, is an international standard that allows for just about every character in any language that's in popular use today.

Up to 65,000 character spaces are reserved in the Unicode set, generally two bytes. But a single byte encoding of Unicode was invented and it's called UTF-8. The key feature of UTF-8 is that you can pass it through normal C string interfaces. It guarantees that there won't be any null termination where there isn't supposed to be.

And the first 128 bits, sorry, the first 128 characters in UTF-8 are just regular ASCII characters. And so for people that are just dealing with a regular ASCII system, it's going to look the same to you even though we've dictated throughout the kernel that all file names are going to be UTF-8. It's not until you start seeing shifting of the file names that you can see that it's going to look the same to you even though we've dictated throughout the kernel that all file names are going to be UTF-8.

It's not until you start seeing shift gist and so forth kanji representations that the byte stream will start to look different. But the beauty is, again, you can pass it through car star interfaces. We also have a number of new features besides what we've given you on the status update and we're going to talk about those in more detail. Actually, I'm going to have my tech lead talk about those in more detail as we move along.

So I have to apologize a little bit. Our radar screen hasn't changed much. If you don't remember, our definition of the radar screen is things that I think we're going to have at some point, but I'm not sure and I'm unwilling to tell you. So, you know, you can ask the question, but that's the answer you'll get. But we do recognize that a lot of people are really interested in a directory change notification feature. And quite frankly, we had sort of hoped to have it by now, but there were other things that took priority. It is, however, something we're still considering.

Another thing we'd like to do is allow NFS access to HFS+ volumes. It turns out this is actually quite a bit trickier than it would seem on the surface. And HFS after all, HFS+ after all rather, has pretty much all the features now that a UFS or a Berkeley Fast File System has. And so it seems like it would be trivial to do the same things that UFS does to allow NFS access.

But the problem is there are a number of architectural issues with Mac OS X overall that make this rather tricky. And in a nutshell, the Carbon application environment, when it runs across a volume that doesn't support some of the features that are associated with HFS+, like resource forks and catalog information, access by ID and so forth, well, it mimics that for you.

It actually builds its own environment for creating its own separate resource fork and its own separate catalog information, does its own ID to path name mapping, so that as an application, you don't have to care if you're on a UFS volume or if you're talking over NFS. You can make the same Carbon calls you've always made and they'll just work.

Well, the problem with that is, on HFS+, we have local support for all of those features. And so if you were to export an HFS+ volume over NFS, locally, the clients would use the native resource fork and catalog information. Remotely, because the NFS protocol doesn't have support for this sort of thing, Carbon is going to see that it's NFS and start mimicking it. And so the remote clients will have a different view of the resource fork data and of the catalog information than the local clients will do.

So that's actually quite a problem for us. We're still figuring out how exactly to solve that. In the short term, we're going to have to figure out how exactly to solve that. In the long term, we'll probably have some configurations where you can NFS export HFS+ volumes, even though those problems aren't solved. And for people that just want to use it in a server configuration and don't worry, aren't so concerned about how the local access works and are able to step around the issues. We also would like to add some new file systems, specifically the Joliet extensions to ISO.

At some point, we'd like to have an NFS implementation and even more importantly, an implementation that will allow us to read files from DOS and Windows. So the FAT16 or FAT32, some, one of the Windows format file systems. So now, we actually do have a demo. And I'm going to call Conrad Minchel up to bring it to you. I'm always a little bit jealous of the QuickTime guys because they get to do these big, fancy multimedia demos. And they have movies and it's really exciting. And file systems, we never get to do that stuff. And they just won't let us do that stuff.

So we're going to sort of skirt the edge a little bit and try to get in at least a picture for you. And with that, I'm going to introduce Conrad Minchel, who is one of the file system engineers engineers on my team. So this actually got working about 20 minutes ago. And hopefully it's still working. A DVD RAM disk installed in this system down below and it's already mounted. And it's mounted on /data here.

So I can see, oops, yeah. So I can see I have one movie already there. So last year at this time we gave you the first ever public demonstration of booting off of HFS and HFS+. You can imagine what might happen next year when you all pay your money again and come back to see us at the Worldwide Developers Conference in 2001. So with that, I'd like to bring up Pat Dirks, my tech lead, and he's going to walk you through some of the cool and awesome new features that we've added to the file system in Mac OS X.

I'll do without that. Yeah, it's pretty amazing to think that last year around this time we just very carefully demoed the first booting and rooting off an HFS+ file system. It's been quite a year. I could take a moment and talk about some of the new features that we've added.

Prime amongst them probably, once you realize that you are running on HFS+ file systems is Mac OS X is a multi-user system and at its core is a BSD base. Every file and every directory in the system is going to have permissions assigned to it. Those permissions are a bit like what you may have seen on AFP file servers, but they're not quite like it.

There's a similar division between owners of files and directories and sort of everyone or guest access class of people and then there's a group that you can assign to every object. You can assign separate read/write or execute permissions to each of these groups. Underneath, although you'll see names in most of the interfaces, these are mapped to numeric identifiers that are mapped out of a database. What access you have to any given file or directory is determined by the user ID of the running process.

That's all fine and well if you're running on a single big Unix system where things get mounted and they stay mounted all the time, but it's a different story if you're running in a Mac OS type environment where you've got zip disks that you're carrying around that you're popping up and dropping in and out of machines all the time.

You have a problem where the numeric IDs that are assigned on a disk that may be created on your neighbor's system may not mean anything in the system that you have created or they may map to entirely different names. Simply plugging in a disk and using the permissions on there, even though HFS+ has provisions for storing the user ID and group ID and permissions information is not sufficient, you have to worry about what to do when there are permissions on there that make no sense.

We wanted to make that as transparent as possible, and to that end we decided to tag every volume with a 64-bit ID that we make up the first time that volume is initialized. The system actually maintains a list of IDs of volumes that it knows about, and it only uses the privileges on those disks whose ID is in the list.

If the ID is not in the list, we fall back to a mode where we ignore all the privileges on the volume and we make the person who's logged in on the machine the owner of the disk. That's the first step. The second step is to make sure that the disk is in the list. The third step is to make sure that the disk is in the list. The fourth step is to make sure that the disk is in the list.

The fifth step is to make sure that the disk is in the list. The sixth step is to make sure that the disk is in the list. be the owner of all the files and all the directories on that system. The only thing to be aware of is that if a disk is used in that mode and you create new things on there, those will also be tagged with unknown. When the disk comes back to the originating system where the permissions are used, they'll see unknown on there. They'll get the same treatment.

The person logged in on that machine will be the full owner of it, but that's kind of a little subtlety to be aware of. Basically, what we strove to do was to approach as much as possible the regular Mac OS experience of being able to take a disk, put it in your machine, and use it in any way you see fit. We don't want permissions to get in the way of that.

Something else we did is an extension of something that is an API that's already provided for AFP URLs, allowing you to manipulate server references as URLs and have the system mount those for you. So this is currently supported only for AFP, but the idea is that we'll extend it over time.

And it's the perfect implementation, in fact it is the implementation, for the connect to dialog in the finder, you just type in a URL and the system takes that URL, figures out what file system uses that, finds a place in the file system to make that appear, creates that directory, mounts it, makes the server connection and makes it available to you.

So it's a very nice API. We're working on that, it's still evolving. If you think you have URL schemes that would make sense to tie into that, we're still discussing whether we want to make this dynamically pluggable or a small static list, come talk to us afterwards. But that's how we envision it. providing access to URLs over time through the file system.

We've added a number of features to the system to help support the HFS+ APIs. Long file names are prime amongst them. On Mac OS, using the Unicode APIs, you can now create and manipulate files with 255 Unicode characters. And we've extended the BSD APIs to do the same thing. As Clark mentioned, UTF-8 is now the standard interface for all files and directories in the file system.

And so we allow you to pass a UTF-8 encoding of a string of up to 255 Unicode characters into the kernel that way. So everything is done through UTF-8 in the kernel. And last but not least, we've supplied new system calls that allow us to support the bulk enumeration calls. And we'll get into some more detail about that in a moment.

I want to talk both a little bit in review and a little bit to bring you up to date on the interfaces and talk about the differences between the behavior of possibly the same code on Mac OS X and Mac OS. So when you're writing your application, you've basically got your choice of one of three environments, and they all make sense at sort of different times.

They're all layered on top of a common BSD layer, though, and as Clark mentioned, that's where we really do our work. And inside BSD, there is a virtual file system or VFS layer that all file system calls get translated to, and that takes care of doing the file system specific implementation of some particular feature or operation.

So Carbon is the natural way to port your application. If you have a Mac OS application, it should be a cinch to Carbonize it. And the Carbon interfaces are, in most ways, exactly like the existing Mac OS interfaces. If you're writing a brand new application, you may want to consider writing to the Cocoa APIs. They're a completely different object-oriented set of APIs. They were designed-- they were designed from the outset to be very portable to a number of different environments. And so they don't support some of the features that you may expect on HFS or on Mac OS.

So there are no resource forks. There's no support for Finder info, those sorts of things. But you can have a very uniform API to general file storage. Finally, of course, there's Classic. If you do nothing to your application, it will run in Classic. And obviously, all the calls are supported there, too. And as I said, all of those are implemented in the PSD layer that we've extended with an actually surprisingly small number of new system calls.

I guess the first thing to be aware of, when Mac OS 9 came out, we introduced a number of new APIs to let you operate on files and directories with Unicode and with FSRefs, a new form of file reference. And all those calls are supported in Carbon. So I think the message is clear. Think Unicode, think FSRefs. It'll work great in Carbon.

The full range of calls is supported. Cat search is supported. It's not something that's in traditional BSD, but we've added system calls to support that. And so anything you can do in 9.0 Carbon, it's fully supported. The new Get Catalog Info Bulk Call that I refer to is a great call.

It's a bit like the AFP Enumerate call, which lets you get directory information on a number of objects in the directory in one single call. And it has a very flexible method of specifying what it is you're interested in about each particular object. So you can say, I want to get just the name and the finder info, for instance, for all the objects in the system. And it will return just that to you. You'll see that actually appear in a number of different calls and some of the new system calls as well.

There are methods using a bitmap type of specification to let you specify which attributes or which pieces of information about files and directories it is you're interested in. And it's great because the other thing that we've done is we've added a bitmap to the system. So that way, the underlying file systems don't have to go and possibly make up information that you're not even interested in.

I mean, for the longest time on Mac OS, we've carried around this number of directories in the root field just because it was there in MFS when the Mac was first introduced. And we had no way to tell whether somebody who called GetVolInfo actually needed that field or not. And so this will help us move away from that.

So the BSD extensions. Um... Get Adder List and Set Adder List are new ways of getting information out about individual files or directories. They basically let you specify a buffer that your information will be filled into and a bitmap with a couple of fields, very much like the AFP Get Fault Airpods call again. You can say, "I want the finder info about files. I want the fork links," and that's it, and you'll get exactly that returned.

SearchFS is a very direct implementation of the CatSearch functionality, and all the things you can do with CatSearch you can do in this system, and it's all implemented through SearchFS. ExchangeData is an implementation of the file exchange functionality that lets you do save saves of data. These are all things that we needed to do to support Carbon's functionality, but there were no equivalent BSD calls to do it.

Most other things actually map pretty straightforwardly to existing BSD calls. All the various variants of open and opendf and whatnot all just map to the straight old BSD open call. For those areas where we didn't have any system calls to support them, we made up these new ones. F-control is an existing BSD system call that lets you do various operations on open files, and we added an option to let you do file allocation, for instance. The allocate call is not something that is known in the BSD system.

Get-Dir Entries Adder is the system call that was added to do the bulk enumeration. And if you are at all familiar with the attribute list, get-adder list, and set-adder list API, you'll see this is very familiar. There is an attribute list that is passed in and a buffer and a size that you specify and a few parameters that let you say which entries in the directory you're interested in, how many items you are prepared to take back, and various options that you can pass. And with that one call, you can get information on a whole number of directory entries all at once, possibly the whole directory in one single call. So it's a very efficient way to enumerate file data.

The VFS layer is something we'll get into in a little bit more detail later, but as I said, that is the layer inside the kernel that separates the file system independent system call layer from the file system specific implementation of certain calls. and it basically manipulates these abstract data structures called V nodes that are specific to each file system and there's a dispatch table that's associated with each V node that lets the system dispatch to the file system specific implementation of some call. If you're writing file systems, you'll get very familiar with that layer of interface.

Now, there are some differences in the way that code works even if the code looks the same or similar. Unix for instance, various flavors of Unix and BSD among them, lets you delete open files which was always a big no-no under Mac OS, you get FBC error if you try to do that.

We support both semantics. If you're calling the delete call in Carbon, you'll call a variant of unlink that actually will return an error if it's open. But if you are a BSD or a Cocoa application calling unlink to remove a file from the directory, that will succeed even if it's open. So it's possible that files you have open will get deleted from the catalog.

That's not as wild of an adjustment as you might think because even under Mac OS 9, it's always possible that something would get deleted or renamed out from under you and no longer appear where you thought it appeared. And Don may get an interview. Okay. So I'm going to go ahead and jump in a moment into the kinds of games we play to make this happen.

But it's something to be mindful of. Hard links are a strictly Unixism. It's a way for two entries in the catalog to refer to the same exact data on disk. And this is not an alias that you resolve to get to the data. These are two equivalent names to get to the same data on the disk. So again, that may be something to be mindful of.

One thing in the Unix file, there's a lot of things that you can do to get to the same data on disk. So you can do that by using the file system APIs. It's very careful not to return you data that you have not yourself written. And if you're just streaming out a file, that's no issue. You're writing everything before you'll ever read it.

But you should be careful of some things that may have made perfect sense at times on Mac OS, like calling setEof to extend the file to its maximum length before you start writing the data. Now, to prepare for the possibility that you might actually read that data since the logical end of file has been moved out. That whole interview is a very important part of the process. And the intervening gap between the current EOF and where you're setting it is zero-filled.

So you may be doing gigantic IOs where before you were doing absolutely none. Calls like allocate are fully supported and they're a much better way of making sure that the data is available on disk before you make your write calls, for instance, than setEof is. One sort of quirk in the BSD APIs is that there is no notion of a create date the way there is on HFS+. HFS+ volumes have them and we support them. But if somebody uses the BSD APIs to manipulate the modification dates of files, the create dates are actually untouched.

And until we decide to do something about this, you may actually see modification dates that are prior to the creation date of a file. It's a little weird, but again, something to be mindful of. And then, of course, the path separators, Unix, BSD, all uses slash to separate path names.

And HFS, Mac OS uses colons. And then, if you have a path that's not a path name, you should be aware that the path names that you use in your Carbon calls are not the same path names that you should use if you make BSD calls. The system underneath actually translates between colon and slash for you.

And the results may be surprising. If you call BSD APIs to enumerate a directory, you may see files with slashes in them, but they will appear in the BSD APIs. So they will appear to you with colons inside the name, because we can't pass up a path name through the BSD layer that has a slash in it.

We can safely pass one with a colon in it. So between Carbon and BSD in the system, there's all kinds of colon and slash translation going on. And if you are spanning these two layers, if you're doing some BSD calls and some Carbon calls, you should be careful about path name separators.

Finally, we support a range of different file systems, and they all have their own implementation, and they all have their own quirks. Unix systems, BSD, no exception. The UFS file system has long been case sensitive. We decided not to make a case sensitive implementation of HFS or HFS+. So HFS is the same case insensitive but case preserving file system.

But you may be running into other file systems unbeknownst to you because you're just traversing the catalog hierarchy and run into UFS volumes that are mounted there, which are case sensitive. So if your application counts on being able to find the same name under two different case representations of it, you're going to be surprised. And that's actually very common in code, where you refer to a header file first by one name, then by another. If you do it on UFS. UFS. You may run into trouble like that.

Sparse files are something that UFS has long supported that there's been no precedent for on HFS+, although we've had some requests for it on occasion. Basically, sparse files are a way for the file system to store your data files without storing the gaps in the data blocks that you've written. So you can seek out to a location two gigabytes into your file and write 10 bytes of data there, and on UFS that will actually only take about 10 bytes on disk.

That's not something that's supported on HFS+. So if you try that because you're porting a BSD application or something, you may be surprised to find that step one is to allocate two gig of data and then go write your ten bytes of data. Hard links are supported on HFS+, but not on HFS. They're supported on UFS and NFS as well.

But, again, if this is something you count on, you better make sure you know what file system it is you're talking about. And so there are a number of subtle gotchas, even though the basic application that you have, if you ported the Carbon and run it on HFS+ volume, will run remarkably well unchanged. And you may be unaware of the difference altogether. At this point, I'd like to bring up Don to talk about some of the differences between the way HFS+ volumes are used in the system. Don? Thank you.

Okay, as Pat said, I'm going to talk about some of the changes we made to the HFS+ volume format. In general, almost all these changes should be transparent to all of you out there, but some people like to hard code things, and so for that reason, you're going to have to suffer through all these boring details here.

The first set of changes involve what we do during initialization in Mac OS X for HFS+ volume formats. Primarily we made some changes for performance and for some of the features like hard links that we added. The allocation block size, which used to be

And like I said, we have the startup file now where we use that to boot for Mac OS X. However, I should note that that startup file is not in the DP4 release. That will be in a follow-on release.

The last mounted version up until now had been set at 8.1. Now that we're using a different code base, we've upped that to 10.0. As Pat mentioned, there's some BSD fields that we now are used. The permissions data and the last access date. We now enforce the permissions and we initialize them under Mac OS X. And on every read and write access, we set the last access date.

Finder info is mentioned here. It's not really Finder, it's not really file system data, but people tended to use it in file system calls. So I mentioned it in here. Typically, like the type and creator and the invisible bit are often used. However, you should be aware that in Mac OS X, often they'll be uninitialized. So if you're doing searches for criteria on type and creator, be aware that they might not be initialized.

One interesting thing we discovered when we were debugging is that there's occasionally in Mac OS 9 you'll get file names with embedded nulls, mainly due because they had a Pascal interface and you could actually do that. It's not a good thing to do, but so to support those and not trip up some of the stacks and the file system layers, we allow them to go through, but you can no longer create them.

In the past, we really didn't have any reserved file names, but a long time ago, pre-System 7, we discouraged the use of dot prefixes in files, mainly because the file system and the device manager shared a common open call, and we needed to differentiate between opening a file and a device.

That support is no longer there or needed, so we no longer discourage it. In fact, given our VSD underpinnings, we have lots of files that start with a dot, but that's not a bad thing anymore. Of course, you should avoid naming any file dot or dot dot, because we use those in the VSD layer.

There is some pseudo metadata, additional. In the past, we had Apple Share and the desktop DB. Now we have symbolic links and the hard link and the deleted files. All this stuff, in terms of the pseudo metadata, should be completely transparent. In Carbon, you'll see some links as aliases. And in the POSIX and the Cocoa layers, they'll actually automatically resolve, so you won't even see them. But we just tell you about that so you know about them.

We updated the disk first aid and the FSDK to be able to fix hard links and we came any of the files that were We're hidden for the delete on busy. And if you're doing any type of repair utility, we encourage you to also add that minimal support into your application. Back to you, Clark.

Thanks, John. If you attended Steve's overview on Monday, you know that there are a million different ways that QuickTime technology is distributed on CD-ROMs of famous artists, sorry, audio CDs of famous artists. He also actually had a cereal box with a CD in it that had QuickTime technology. Well, I have great news.

As part of the whole new era of respect for file systems that got us this large hall to get together in, my boss just informs me that we have made a deal with the makers of generic cereal. for the distribution of file system technology in the lesser stores near you.

Alright, let's talk about the virtual file system layer. So, to begin, this is the layer in the file system that you would implement to if you wanted to develop a new system. There are two fundamental uses of it: a new file system for a new device type or a new protocol, or what we call a file system stack.

The way file system stacks work is they basically implement a set of functions to the VFS interfaces and they sit on top of an existing file system and at the bottom they call that same VFS interface for a lower file system to process. So, for example, if you wanted to build an encryption system, you could build a file system stack, you would support the VFS interfaces, most of your VFS calls would simply call the file system underneath you.

But if you wanted to do I/O, then you would look at the buffer, encrypt it if it was coming in, decrypt it if it was coming out, and so you would call the file system below you to get the data, make your manipulation, and then return to the caller. Or you would get the information from the caller, do your manipulation, and then call the file system below you to finally do the write. So that's roughly what the VFS is for.

Two key data structures you need to know about if you are really seriously thinking about implementing a file system or a file system stack. And the first is the VNode, which stands for virtual node. And this is the key data structure that represents a file in our system. If you programmed in Mac OS, it's similar to a file control block, an FCB, but there is one key difference. VNodes are created whenever a file is referenced for any reason, not just when a file is opened.

So if you just wanted to get file system statistics, if you did the equivalent of a getcatinfo, we're going to allocate a VNode inside of our system and we're going to initialize it. And it's going to stay around for a while. We have a cache of these in the system and up to thousands of them that just stay in case you want to reference that file again so we don't have to recreate it. And we also have another key data structure, a mount structure, which is kind of similar to a volume control block.

So these are the two key data structures that you'll need to work with if you're wanting to implement a file system. There are also two key segments of the VFS interface. There are what we call VFS calls, and I apologize for the name confusion, but we inherited that.

And there are also VNode operations or VOP calls. And the reason they're called this is because if you actually look in our code, if you look in the file system independent layer of system call processing, you'll see some work go on and then you'll see a call to something called VFS underbar something or VOP underbar something. And what those are are macros that look for the dispatch table for the right operation to call.

Given the type of a VNode that it's about to make the call on or the type of file system it's about to make the call on. If it's a VFS operation, you'll see VFS underbar operation and it will call the VFS routine for that file system. If it's a file based operation, you'll see a VOP underbar operation, VOP remove, VOP open, VOP close and so forth. And it will make that call to the underlying routine in that file system.

So while you see all these VOP macros scattered about, if you look at the implementation for any given file system, you'll see that they have routines called in the HFS, for example, HFS close, HFS open, HFS remove. Those are the routines that will actually be called when you see VOP calls inside of the code.

So let me give you a walkthrough of some of the VFS operations. Actually, I think we have the full list of these. One quick note: to instantiate a file system in BSD, and this is the model we've inherited in Mac OS X, you first create a directory and then you mount the device, which corresponds basically to our petition on a hard disk drive or some protocol that alludes you to a hard disk over on a server. You actually mount the device on top of the existing directory. If the directory was not empty, all the files and anything that was in that directory disappears from your system until you unmount it again.

So usually we do this on empty directories. So you'll create a directory, you'll mount the device on top of the directory, and from that point on, that directory will be a window into all of the files on that partition that you've mounted there in that place. VFS mount and VFS unmount are the calls you use to manipulate that. So VFS mount is going to create and fill in the root V node, the V node for the root directory that you've just mounted on that directory, and it'll initialize the mount structure.

VFS unmount is going to clean up the mount structure and remove that device from that file system you create, from the directory that you've created, and from that point on, you'll no longer be able to access the files on that volume or on that partition. But you will be able to access any files that were left in that directory when you mount it on top of it.

VFS Root's job is to give you back the root V node given a mount structure. So if you're trying to do path name processing, you're starting at a mount point, this will get you to the root V node. StatFS is just a get statistics call. It's kind of like a get ball info.

It is give me some information about this overall volume, things like the overall size or the amount of free space available, that sort of thing. VFS VGet, we won't get into detail, but this is the call we use to support access by identifier. So if you want to implement a file system that allows access by directory ID name the way HFS+ does, then you'll need to support VFS VGet.

VFS init is just your basic file system initialization routine. You may or may not need to support VFS init. It depends on whether or not you have some global file system-wide data that you need initialized at mount time, or I mean when your system is first initialized. VFS FH to VP, this stands for File Handle to VNode Pointer, and VP to FH stands for VNode Pointer to File Handle, and these calls are specifically for supporting NFS access to your volume. These are the two key guys for getting file handles sent across NFS for remote access. And sys control and quota apple, we won't go into detail about those, but those are for supporting system-level controls of your file system and for quotas.

We're not going to give you an exhaustive list of all of the VNode operations, but I will tell you about some of the important ones. And these are the ones you'll actually be tested on later when we talk about our caching architecture. So, VOP lookup. This is the most important and most complicated of all the VNode operations.

And its job is to handle path name resolution. So basically, you give it a path name that starts from a directory that belongs to your file system. And you give it a string that contains a bunch of path name elements separated by a slash. And then this routine will give you back a VNode.

And it's generally called iteratively as we do the lookup process in the system. So, you'll get a component pointer, you'll get a VNode that corresponds to a directory through your lookup call. You'll return the VNode for the next thing in the path. You'll get called again if that's one of yours and it will go on and on and on until lookup finally reaches the end of the path and gives you back the VNode for the end file that you wished to target.

Probably worth mentioning that Mac OS X has a singly rooted hierarchical file system layout, much like BSD. The Carbon interfaces will abstract that for you. They'll know where all the various mount points are. And you can ask the Carbon interfaces for a mount list and work like you did in Mac OS 9. But in Mac OS X, there will be actually a complete hierarchical file system list. And so BSD calls will work. And the Cocoa environment which relies on that will also work. And so lookup becomes a very central routine in Mac OS X.

Open and close are what you're expecting. They're the two routines that enable you to start doing access to a file and rip out that ability later. Read and write. Basic read and write. Read and write routines. Truncate, which is used to both grow and shrink a file. The name is a little unfortunate. But you can truncate a file and send its valid space way out to the end, move its EOF. Or you can also do a truncate and reduce the size of a file, cut it all the way to zero if you like.

Fsync is file synchronization. This is the call that takes all of the dirty buffers you may have associated with your file in that system and starts IO to the disk on those so that we can get all the data flushed that we have cached in memory out to the disk. And you can make sure that your file is in the disk. You can make sure that your file data is consistent. It's a lot like flush files in Mac OS 9.

And vop remove, that's basically the delete call. As we mentioned before, in order to give the full range of abilities of HFS, HFS+, things traditionally associated with Mac OS, we've extended the BSD interfaces to add things like get adder list and set adder list that will get you to things like the catalog information, allow you to do searching.

Well, all of those calls are also supported by underlying VNode operations. And so if you wanted to implement the file system that's supported by the get adder list, you implement vop get adder list the way we did in HFS+, and then those abilities will be available to your callers.

So now we're at the real hard stuff. We want to talk to you about this big change that we've just made in our caching architecture. And anybody who's actually thinking of implementing a file system will need to know this. And we'll also tell you about why we did it.

The changes in the caching architecture were actually done by the BSD team with some assistance from the file systems group. But this is something that spans both file system development and the virtual memory system. And so we're fortunate. I'm going to talk a little bit about the beginnings of this and then later I'm going to bring up Ramesh, one of the BSD engineers, who will tell you the details about how that system worked.

This is a rough picture of what our old cache looked like. And up on the top, top of your right, that big box that says VFS there, that's the VFS interfaces we've been talking to you about. And then that's our underlying support for all of the different file systems.

And this is, I'm going to walk you through how the buffering used to work when a read call would come through. Basically, the overall BSD call read would come in and it would go to the VFS layer, it would dispatch to the appropriate file system that was going to service it. And then that file system would most likely call the buffer cache underneath.

The buffer cache might have that data already read in memory because you've read it before or because you were doing sequential access and we anticipated that that page was going to be read and we put it in memory for you in advance. And it will satisfy the request out of the buffer cache. And if not, then the buffer cache will call back to one of your routines to actually read the data off of the device or get it off the network.

So that's the basic, that's the basic deal. Come in with a read, go to the file system that supports it, talk to the buffer cache, try to satisfy the page. And it generally works pretty well and the buffer cache can be used to optimize performance. Well, there's this other way of doing I/O in our system that we call file mapping. And some of you, if you've worked on BSD, may be familiar with this. But the idea is you can open up a file and then you can decide that a certain range of address, by the way, I appreciate the applause over there.

A certain range of address will be backed by that particular file. So you'll say, "Hey, I'd like you to make addresses 1000 through 5000 valid for me, and I want you to use as their backing store this particular file that I'm telling you about." And then if you read anything in that array of addresses, you'll get the data that's actually in that file that you've decided is mapped into that array of address.

So it's a very handy way of doing I/O, but it does cause some interesting problems for file systems. The first one being the cache for the virtual memory system is different from the cache for the file system, or at least it was in earlier versions of Mac OS X. And so to walk you through this, you have a process. He comes along, maps a range of data, says he would like it backed by this particular file, and then starts reading the memory.

But what happens is the VM cache will start doing I/O via new calls called Page In and Page Out. They'll eventually call the file system underneath, they'll get the data, and then they'll put it into the VM cache. And the VM cache will then satisfy those requests. But what can also happen is one process can open the file via read and write, another process can map the file, and either of those processes can now write to it.

The read and write process is going to write to it via the normal file system read/write APIs, and the data is going to be cached in the buffer cache. The mapped file is just going to write into an array of memory, and it's going to believe that that maps into that file, but in the meantime it's going to be stored in the VM cache until the VM system decides to flush it.

So if two processes open up the same file, it's going to be cached in the buffer cache. So if two processes open up the same file via these different mechanisms, they can actually wind up with inconsistent data that corresponds to the same address in the file for that page of memory. That's kind of actually diagrammed here. You'll see a page of memory here in our buffer cache that's supposed to correspond to a page of data in the file, another page in the VM cache, and you can see how these guys can become inconsistent.

So that's the big problem. We have these two caches and the data can become inconsistent in these two caches because ReadWrite is using the file system buffer cache and the VM system is using a separate cache. And as long as these things were totally distinct, you could get yourself into trouble.

Now to try to avoid some of those problems in DP3, there were a number of synchronization points in the file system where we would check with the VM system to see if it had data and try to get flushing to happen and so forth. But there were still some holes in there and while it was unlikely, you might have stumbled across some cases where your file system data looks sort of inconsistent.

So what we've done-- let me tell you what we did in Mac OS X Server. We recognized that this was a potential problem, so we had a solution for this issue in Mac OS X Server. What we did was we implemented something called the map file system. And what it did was turn all regular I/O into mapped I/O.

And so the VM cache became supreme in a Mac OS X Server system. You would do a read and you would do a write, and actually ultimately it would turn into a map call, and we would map it into a spot in the kernel's address space as opposed to the process's address space.

Well, that worked pretty well, and it did get rid of the potential data inconsistencies, but the buffer cache was a fixed size and rather tiny underneath. That was a bit of an issue. The VM system, again, was reigning supreme, and so some of that read ahead stuff might not be happening. There still were two copies of data in the system.

Now while the VM cache was sort of superimposed on the buffer cache so that you would never get the data inconsistent, you could still replicate the data, because the VM system would ask the file system-- the file system would get out of the buffer cache and it would go back up. They were never different data. for different data, but there were still two copies, and that was inefficient.

Only part of the file could actually be mapped in the kernel. We don't have the full range of process address spaces. We have one kernel address space. And so that caused us to have to do some funny things like potentially throw out mappings for different processes as new processes would come and go and wish to do I/O because we only had a limited address space.

That meant that not all currently mapped files could necessarily be mapped at the same time and you might not be able to map the entire file at once because we had to deal with this limited address space. So we're throwing mappings in and out of the kernel all the time causing potential performance issues.

And it also wasn't very friendly to the file system because reads and writes would come in and the file system wouldn't get them right away. They would be redirected to the VM system. The VM system had its caching going and it would only call the file system when it decided to flush its cache or bring in new data.

And so if you wanted to have a counter in your file system of all the reads and writes that were taking place, you wouldn't have them at the time the user asked. You'd only see them when the VM system was ready to communicate. So that was an issue too.

So what we've done is we've just unified these caches. Sounds simple, it was a lot of work, but it's into DP4 and so we got rid of all of these issues. And now there's just one cache for both the VM system and the file system and all the pages that are associated with the file are satisfied out of the same cache. So to talk about some of the details of that, we've brought up Ramesh, who's one of the BSD engineers, and he'll walk you through some of the implementation of the unified buffer cache.

Hi, good afternoon. My name is Ramesh. Basically, as you saw that Clark mentioned that we had two kinds of caches in the system. When we had the VM page cache, which was a big thing, and we had a tiny buffer cache, which is much smaller, which we were using to buffer cache, we had a data inconsistency.

The way we used to construct buffer cache was when a Mac OS machine boots very early on, the secondary loader then transfers the control to the kernel. When the kernel comes up, initially it has a pool of pages. Actually, this pool consists of all the physical pages in the system except for the pages that are consumed by the kernel text and data itself. What we used to do is carve out a small portion of the pages from there and then reserve them to use exclusively for buffer cache.

And this is where the fixed size of the buffer cache is coming because we used to take a number of pages and we were trying to use that for the buffer cache. As Clark mentioned before, we have one huge honking VM page cache on one side and we had a small buffer cache caching the two. What we did basically was To stop using the buffer cache for the file data, what we did instead was to turn all the file data to the VM cache itself.

Now that we turn all the file pages into the VM page cache, we had to implement a mechanism to track all the pages corresponding to a particular file or a single page corresponding to a particular file or a particular offset in the VM page cache. For this, at Apple, we implemented a mechanism called Universal Page Lists. What Universal Page Lists does for us is to get at a page, a specific page in the system.

Universal Page Lists is a mechanism to get at a page in a memory object. And it actually also allows us the physical access to the page so that we can use that to do DMA directly without ever mapping inside the kernel address space. If you want to know more about the page lists, the kernel session on the Mac OS X is a very simple process. On 106 on Thursday morning, 9:00, they will discuss about this. If you want to know more about this, they will talk about it.

What we did was to use these UPLs, Universal Page Lists, and develop an infrastructure underneath the buffer cache APIs. The buffer cache APIs were like B-read, B-write, get block, B-rails, and things like that. So not all the colors of the-- The colors of the buffer cache APIs never have to change because we maintain the same APIs. We changed all the plumbing underneath Thursday to use unified buffer cache.

We also wrote a set of routines that we need to sprinkle all along the different file systems to work with the unified buffer cache. The best way for me to explain all this is to take different scenarios in a lifecycle of a file and then see how unified buffer cache works there.

First, let me take up the open. Let's assume that we were opening a file in HFS volume, so that all the vNode operations that I'm going to be talking about are basically HFS plus vNode operations. And let's also assume that we're opening the file for the first time in the system.

As part of the open system called implementation in the kernel, the first thing it tries to do, as Clark mentioned before, is to convert the path name from path name to a file representation inside the file system. So basically that, for our example, is going to be HFS lookup. This HFS lookup is a vNodeOperation lookup vNodeOperation HFS. One of the things that it will do is going to see whether the file already exists. And see whether it needs to create a file if you pass the correct flags for the open.

And then, what it does, whether the file exists or not, whether it created or the file was already there, it creates a VNode. It creates a VNode for that particular file. All future references to the file, all manipulations to the file from now onwards goes through this VNode.

In the lookup, we added our first support routines that was talking about file system support routines in the unified buffer cache called UBC-Infinite. What this UBC info does basically is create a memory object and create an association between that memory object we just created and the vNode we just created. It's a one-to-one association. For example, if we're looking at a 2000 offset inside this vNode, it translates to a 2000 offset inside the memory object that we created.

Now let me take another scenario where We're just going to map the file. Let's say we take first 32K of the file and we map in the current address space. Suppose a user, this usually basically comes to the kernel as an M-Map system call. What we do in the M-Map system call is basically set up a pager and then allocate a 32K range because my example is 32K. We set a 32K address range in the user process for this particular file mapping.

And then we return the address of this where we allocated in the user space back to the user through the M-Map system call. Note that we haven't done any page I/O, any file I/O, nothing yet. Nothing has happened yet. When the user first accesses this address space, let's take it as, you know, for example, he's reading the first byte of this allocated range.

What happens at that point of time, since there's no page backing this address range, it's going to generate a page fault. The VM system goes and tries to resolve the page fault. The page is not there in the system, so then it allocates a clean page and then it tries to fill the data from the file system for this page.

In our case, since we're taking an example of HFS+, it ultimately goes to the VNodePager and falls into HFS Paging Call, which's responsibility is to copy the data from the disk to this particular page, which VM gave us, and then it returns the page back to the VM. And this page stays in the VM page cache and all future modifications that users would play with this particular page would happen without any knowledge in the file system or anywhere else.

Let's say for our example we access some more pages. I just put like sparse, it's not always contiguous, so 0 to 12K and then 20 to 32K range of files we have touched. Then, now I'm going to take a scenario where we just want to do a write.

On, at an offset 15, just to make it a little bit more interesting, like offset 15, and then we'll just write only 10 bytes of data into the, we want it to the file. So we use a write system call to do that. As part of the write system call, implementations are the kernel, which will then translate to the particular BNode operation for that file system, which happens to be HFS write for us.

What we use, we then use a universal page list routine called VMFault List Request, and we ask the VMFault List Request to return us the page to starting at an offset zero, and the size which is equal to the page size. What it does is it gives us the exclusive access to that page which was residing in the VM page cache.

Once the page has been obtained, we then copy the data from the user buffer which he had to this page. Once we copy the data back into the system, then depending on what file systems want to, they either can write up the data immediately to the disk or they can delay, postpone writing to the disk. In HFS we actually delay the writes.

Then, all we do then is basically written the page back to the VM page cache that we used to by using a kernel UPL commit call, which is also coming from the universal page list. So basically, we did a copy of 10 bars into the system, which the user who has mapped this can also see the modifications that were already done.

Okay, let's say I just took another example where he just dirtied another page as well. I'm going to now move to take another scenario where we're going to talk about the WinRAR operation called Fsync, WAP Fsync. In our example, it's going to be HFS Fsync. Basically, the HFS Fsync's responsibility is to push all the modified data from the buffers back to the disk.

Since we have modified data in the VM page cache, we have a new routine which the Fsync also has to do, apart from whatever it used to do before, called UBC Push30. What UBC Push30 does is goes and looks at all the modified data corresponding to this particular file that was there in the VM page cache and writes it back to the disk. That's how all the data goes to the system.

I forgot to mention that this Fsync, you know, operation can come in two ways. Once the user program can call a Fsync system call to make sure that all the modified data is pushed back to the disk. Or it can also come from the system which periodically runs a system called called sync, which goes through and looks at all the modified data in the system for all files and pushes them to disk. So even if you don't explicitly call a Fsync, the data is synced back to the disk.

Now I'm going to take an example of the truncate. We just want to truncate the file back to 16K. Basically what I mean is the logical end of the file is set to 16K. So we have another support routine called UBC set size. Basically what it does is it invalidates all the pages in the VM page cache which are beyond the logical end of the file. So as you can see in this diagram, all the pages that were marked 20K, 24K, 38K, they're all gone. They're all invalidated, so they don't exist in the VM page cache.

Now, I'm going to go to the last scenario that I want to talk about where the file gets deleted. So basically, everything that we set up for so far, we just have to tear everything down. So we do that in three steps. First, we call the UBC set size routine, which I mentioned as part of the truncate, which basically takes away all the pages that were there in the VM page cache. Basically, we set it to zero as the logical end of the size.

And then we also call another function called UBC release, which basically disassociates the memory object association with the VNode. And then we also need to call another routine called UBC NCache, which basically tells VM systems don't ever cache that memory object, because VM does cache all the memory objects that it has.

So this basically gives us a whole lot of overview like how we have implemented the unified buffer cache and what kind of set of proteins that we did and things like that. Now I'm just going to talk next two slides about like what each file system has to do and what kind of changes each file system had to go through to work with unified buffer cache.

While implementing the unified buffer cache, we introduced five new vNode operations. Actually two of them are already there, three of them are new. And in this file, four are mandatory. You need to have them to work with UBC and the last one is optional. The first part is a vNode operation called block to offset.

What it basically does is given a logical block, it converts and gives us back the file offset. If you know already that all the buffer cache is in the file set, buffer cache routines like get block, bread, everything works in logical block numbers. You tell bread, logical number, blah, and then it returns the data back. So since all the memory objects work with the offsets, this basically tells us what the offset conversion for that block number is. Because every file system can have their own logical block size.

and the next one, it does exactly opposite. It actually translates from the offset to a logical block number, which we use it in the page to shoot any buffers that are there in the cache. We also have two next two on operations, page and page. They actually existed before Unified Buffer Cache 2.

There's slight differences in there, like we passed Unified Page List to that, Universal Page List to that, other than that there's nothing changed there. And then the last one is used for the file system clustering. It's called Cmap. It's very similar to BMap. What the BMap feed-on operation does is given a logical block number, it goes and computes the actual disk block number so that we can do an I/O on it.

What Cmap does is, It takes a file offset instead of the logical block number, and it also returns the disk block number corresponding so that we can do an I/O. And it also tells us how much continuous blocks are there from that disk block onward so that we can do a big chunk of I/O. If anybody wants to use file system clustering, then they need to implement that too.

And as I already mentioned in all my examples, like the all file systems have to sprinkle these five, one, two, three, four, five, yes, five support routines that we wrote. So for example, just to recap them, UBC in point eight, we need to insert them whenever we create a VNode.

And then we need to call UBC push dirty, which basically whenever you want to send the modified data back from the VM page cache to the disk, you need to implement that. And then you need to call UBC set size whenever your file size changes. The file size can change, for example, when you're growing a file by write system call, it changes the size. That's the time when you need to call them. And also when you're shrinking the file, we're using a truncate or making a truncate to the far end. In any case, you need to call UBC set size.

And then when you actually delete a file, most of the file system actually does, but they don't do a truncate because they need to get rid of all the disk blocks that are allocated and things like that. If you already don't do that, then you need to do a UBC set size. Otherwise, you need to do UBC release and UBC uncache. We're going to be having a -- we're going to be putting a document saying how to write file systems to UBC.

It will be there in Darwin sometime in the near future. That's a big resource. And all the code for the unified buffer cache and all the file systems are there on the system. It's already in the Darwin, and it's already there in the Darwin server now. If you have any specific questions on which corresponds to your file system, then we can talk about it. Come see me. We can talk about it in the end. Now I'll get Clark back on the stage.

Thanks. Thanks, Ramesh. All right, so that's it. The only key thing left to tell you about is that the next new file system feature could be yours. This is our Darwin slide. If you're really interested in the stuff we've talked about with UBC, VNodeOps, and so forth, check out Darwin.

Take a look. Take our code, please. some of the new things, some of the ideas we've just sort of tossed around. An NTFS implementation, because we haven't gotten to it yet. ProDOS, the stuff that used to ship with Apple II machines. We don't have a logical volume manager, so Venom might be an interesting thing to port. We don't have a Linux file system support yet, so EX2FS might be an interesting thing to port. And, you know, of course, there's always URFS, something we haven't thought of yet.

Some stuff we'd like to have you take away with you. Choose your interface carefully. Remember that there are some subtle differences that happen because of the volume format, and there are some subtle differences in what the interface is doing to not allow. If you're doing a new object-oriented application, you want to use Cocoa.

If you need access to the resource for it, you may need a new Carbon, et cetera. Be aware of those behavior differences that Pat talked about between Mac OS X and Mac OS 9 and also among the different volume formats we support. If you do want to implement a new file system or a file system stack, be sure it's necessary because you are, as Dean Reese mentioned in the IOCAD session, going to be in the kernel, and if there's something that goes wrong, you'll crash the whole system.

Be sure you need to. PhoneFS is probably not the best way to do modem access, for example. And be aware of the unified buffer cache because it has changed the way file systems interact with caching in the system. And finally, make sure you use Darwin as a resource. And I'll mention now, even before the roadmap slide, that today at 6.30 in Hall J1, there'll be a birds of a feather session for the folks that are interested in Darwin. And if you have questions about Darwin, you'll want to come to that Hall J1, 6.30 tonight.

Other sessions you might be interested in, there is a Mac OS X kernel session tomorrow at 9 o'clock in the Civic Center. And if you're curious about universal page list, there'll be some talk about that there. There's also a Mac OS X application packaging and document typing session that may be important to go to if you want to find out about how we're doing, like, mapping launch bindings and icon decisions and stuff like that. That's going to be tomorrow at 2 here in the big hall. Mac OS X BSD support, it's going to be in Hall A2 Friday at 3.30, a crucial session if you're curious about the BSD infrastructure inside of Mac OS X. Thank you. Thank you. Thank you.

And the feedback form for Mac OS X overall, I'll be there as well as all the other Core OS managers. That'll be in Room J1 Friday at 2 o'clock p.m. If you have questions, you want to get in contact with us about, you know, closer relationships or get on a mailing list or whatever, give a shout to John Signa at Cigna at Apple.com.

He's the technology manager for Mac OS X in Apple Worldwide Developer Relations. I see we only have a couple minutes left, and so we're not going to be able to do questions here on the stage, but my team will be in the back of the hall over there if you want to come by.

And ask us questions, and we'll also be at the Birds, I will also be at the Birds of a Feather, and I'll also be on campus tomorrow from 7 to 9.30 for the big campus event. So thanks a lot for your time. We appreciate your coming. Thank you.