Good morning, everyone. Thank you all for coming to session 536, Developing Xsan-Friendly Applications. My name is Jason Thorpe. I'm the engineering manager for Xsan. So here's a little bit about what you're going to learn today. First of all, you're going to learn what XAN is and how it works. Raise your hand if you're an XAN user today. You have one deployed. All right.

That's a good amount of you. So for you guys, this is going to be a review, because I'm sure you're all familiar with it. How many of you are considering deploying an XAN? And how many of you are developing applications that you would like to work with XAN? OK, so this background information, you guys are really going to pay attention to this, especially application developers. Secondly, we're going to talk about some of the XAN volume characteristics. Again, application developers pay careful attention to this. This is really targeted at you guys. And finally, we're going to talk about tips on making those applications both compatible and how to make them work better.

So let's just jump right into the basics. So for those of you who don't know, Xsan is the SAN file system for Mac OS X. SAN stands for Storage Area Network. Network -- that's a key concept to keep in mind as we go throughout this discussion. It actually describes a lot of the behaviors -- explains a lot of the behaviors of Xsan. What Xsan does is it allows multiple computers to have high-speed access to a shared pool of storage. And it differs from the traditional file server in a number of ways, which we'll go into shortly.

And let's just take a look at what some customers are saying about Xsan. So, JW2, they are the in-house production facility for the award-winning JWT ad agency. And Xsan has allowed them and their creative staff to work in a really organic way, to collaborate in ways they couldn't do before, because they can share all their assets quite easily.

So, Bunamuri Productions, they're a television production company. They make some TV shows you may have seen or heard of, including MTV's "The Real World" and Fox's "Simple Life." The combination of Xsan and Final Cut Pro has allowed them to do all their TV production editing really in real time. You know, they're on really tight schedules, and it allows them to do this very quickly at a fraction of the cost of other solutions that are out there.

So how many of you watched the Olympics? Raise your hand. Did you enjoy some of the stuff that they were able to do? So NBC actually had a very large Xsan. It was over 20 terabyte. And they used a completely Mac-based workflow on that Xsan to take the content from the cameras that are out at the actual events, edit it, add motion graphics and commentary and whatnot. And they actually had three Power Mac G5s that were connected to the Xsan that were able to play that content out directly to air.

And of course, these are just a few examples of the many organizations that are using Xsan today. So the benefits that Xsan brings to its customers has really enabled -- has really made Xsan one of, if not the fastest-growing SAN solutions out there in terms of the number of deployments.

So what does this mean to you as application developers? Well, first and foremost, it means that if your application is compatible with, you know, works well with Xsan, it's going to have an advantage in the marketplace up against another, you know, competing product that does not. It also means that as the number of Xsan deployments grow, you know, one of the things that we hear a lot from customers is, "Hey, you know, does this work with Xsan? Does that work with Xsan?" And, you know, your application doesn't want to be left out as more places adopt Xsan.

So that's what Xsan is. Let's talk about some of how it works. So what we see here is a very basic Xsan setup. What we're going to do is kind of just walk through, highlight each piece, and describe a little bit about what it does. So first of all, we have the Xsan controller.

So the Xsan controller is really -- it's kind of the key focal piece of the SAN. It actually controls all aspects of file access within the volume. It's not involved in the data transfer, but it actually controls where the data is located on the disk, how blocks are allocated, the policy for block allocation, and also manages all the volume metadata. So this is not to be confused with Spotlight metadata.

We're talking about very basic things like file attributes such as the owner group and permissions. And also it manages all the directory entries. So anything revolving around file creation, deletion, and renaming, all that is handled by the Xsan controller. And the Xsan controller also manages the byte range locks.

So unlike some other SAN solutions that are out there, Xsan provides file-based locking that has the granularity of individual ranges of bytes within a file, whereas other SAN solutions out there might lock out all other clients from updating files, writing files, while one other computer is operating. So it's a very simple setup. Jason Thorpe So let's talk about the Xsan controller. is updating them.

So the next piece -- there's no real surprise here -- is the XServe RAID. It basically provides the backend storage. So it stores your volume metadata and your file data. Next, we have the clients. These are the workstations, the servers that actually connect to the SAN and use the SAN to do work.

It might be a Final Cut Pro editing station, it might be a cluster node in an HPC environment, or it might be a server that's providing file services to other clients on the network. The clients actually access the data directly over Fibre Channel. They don't actually have to go through the controller to get that data, so that makes it unlike a sort of traditional file server. However, they do have to send messages to the controller to gain access to that file data, which we'll go over in just a minute.

So next here we have the Fiber Channel Switch. This is really kind of the glue piece that binds everything together. It allows you to connect your clients, your controller, and all the RAIDs together to form what we call a fabric. Without the switch, you'd be able to only do really a direct-attach scenario where you have one computer attached to a single RAID. And it's useful in some environments, but not in others.

So we can't forget the cables, of course. And then finally, we have a private Ethernet network that connects all the clients and the controllers together. A lot of people ask me, "Well, why do we have this private Ethernet network?" The reason we have this is because the messages that are sent between the clients and the controller, they're very small.

And for those of you who have been network administrators for a long time, you know that small packets are kind of the pessimal workload for a network. So by keeping the SAN traffic away from your normal local area network traffic, you're kind of doing two things. First of all, you're preventing the SAN traffic from slowing down your email reading and your web surfing and all the other things that you do on your local area network.

At the same time, you're also preventing the traffic on your local area network from kind of competing for the available network resources that the SAN network is providing. So you can see here that the SAN network is actually using the same network that the SAN uses, which actually improves the overall performance of the SAN, because you need to have that responsiveness between the client and the controller.

So, unlike other volume formats supported by Mac OS X, Xsan has this concept of storage pools. It allows you to actually structure your data around on the disks in kind of different ways that match your workload. Now, you always have to have one storage pool that's dedicated to holding your volume metadata. Here we've highlighted one of the RAID sets there.

We actually strongly recommend that you put this on its own dedicated RAID controller. The reason for this is that the transactions involved in the volume metadata are -- again, they're small -- and you don't want those to really interfere with the streaming of data. You know, say you have a large video clip to read. You don't want these individual metadata transactions to be interfering with being able to stream the data off the disk. So, again, we strongly recommend that you have a dedicated RAID controller for this.

Next, in our little example here, what we've done is we've actually pooled the other RAID sets together to form a single large storage pool. Another thing that Xsan allows you to do is to create what we call "affinities" -- it allows you to assign an affinity to a folder in the volume.

What that does is it allows you to steer data to individual storage pools just simply by copying it to that folder. Now, in this example here, it doesn't make much sense -- we only have one user data pool, but as we go along the slide, you'll see how this makes a little more sense as we go along.

And, of course, one of the things that's really nice about Xsan is that the controller really puts us all together and shows it just as one volume on the desktop. You don't have to put your users through dealing with 12 different drive icons in a desktop just to know where to store their data. It's all one volume. Now, in this example, we're talking about just one volume. Of course, you're not limited to just one. You can actually have up to 20 of these.

So what we're going to do now is we're going to just kind of step through a simplified write operation and highlight each of the components that's involved, just to kind of illustrate better a little how this works. So first of all, the application on our client here decides it wants to write some data to the volume. So it uses the write system call, and in response, the Xsan client software on that computer sends a message to the controller, Basically asking where that file is located on disk.

So next, the controller consults the volume metadata that we have on our first RAID set there and basically it traverses its data structures to determine where that file is located. If the data blocks for that file are not allocated, it goes ahead and allocates them. And then it responds back to the client with what we call an "extent list," which basically describes the layout of that file.

Each time the client has to write a file, let's say for example that this client wanted to write the same range of the file over and over and over again. It does not actually have to contact the controller each time. It is able to cache the information, and then the controller will send a notification to the client when that information becomes out of date.

So, armed with that extent list information, the client is then able to go over Fibre Channel directly to write that data to disk. Now, note here that the controller actually allocated the storage for that file and spread it across each of the three RAID sets in our User Data Storage Pool. And by doing so, it actually allowed the client to write to those three RAIDs in parallel to increase the throughput.

So that's how Xsan works, basically. And this little example isn't all that interesting. There's just one computer. This is kind of glorified direct-attached storage. But one of the beauties of Xsan, of course, is because it's a network, it allows you to connect multiple clients to the SAN. They have access to the SAN, first-class access to that storage on a kind of a peer-to-peer level.

Now, let's say you've been using your SAN for a while and you suddenly realize that you need some new space to store your audio clips. You've run out of space or your audio clips are kind of interfering with being able to stream video off of your other storage pool.

Xsan actually would then just simply allow you to plop on another storage pool, expand the volume without having to recreate it, and you could basically add another storage pool for this different kind of workload. You could tune that second storage pool for the type of data you want to put on it.

And this is where affinities would become very useful. You could create a folder at the top level of your volume, say this is my audio folder, assign it to that pool. Any content that went in there would be automatically assigned to those drives, and it wouldn't interfere with being able to read and write data quickly from the other storage pool.

Now, let's say, for example, again, that you've gone from shooting static to video. And you've got a lot of data that's standard definition video to moving to high definition video, and suddenly you realize that you need more throughput. Well, again, XAN would simply allow you to add another RAID. It would allow you to assign that RAID to your existing storage pool and to expand the throughput available to that storage pool. Xsan also allows you to add redundant volume controllers.

This basically protects you in the event that your primary controller were to fail for some reason, for any of the number of reasons that computers sometimes fail. So, that's how it works. Let's talk a little bit about the characteristics of Xsan volumes. So, the next point I'm about to make is probably the most important one I can say to all you application developers out there. If this is the one thing that you were to come away with, I would want it to be this: Xsan is not HFS+. It has a much different heritage than HFS+. The technological underpinnings are very different.

In a lot of ways, it does not have some of the same capabilities or features that HFS+ has. And so, we find that when applications depend on some of those features of HFS+, the applications either do not perform well on Xsan or they'll fail to work at all.

So it's not like HFS+. Well, what is it like? Well, in many ways, it is like any direct attached volume that you have on your system. You have high speed access to that data. You can depend on that sort of performance profile. However, remember that XAN has to communicate with a controller to do things like change file attributes or even look up a file, because it has to do path name resolution through that controller. So in many ways, it behaves a lot like a network volume. And with all the sort of latency and whatnot that go along with those type of operations on a network volume.

So, Xsan is also a POSIX volume format. This is where most of the differences of HFS+ really lie. In a lot of ways, Xsan is very much like UFS in the way it behaves in terms of its POSIX-ness. And in fact, one of the first things that the Xsan engineering team does is when we see a compatibility problem with an application, the first thing we do is attempt to reproduce that problem with the UFS file system because they're so alike in many ways.

So, first of all, there's no resource fork. So, it's a single-fork file system. Any application that uses a resource fork using the resource manager, basically that is going to be emulated by the file manager, and you get those little dot-on-a-bar Apple double files littered all over your file system.

So, the emulation of the resource fork, you know, works pretty well, but it's not perfect. So, just keep that in mind. So, there's also no exchange data operation. There are a number of applications out there that are based on Carbon that use exchange data in order to implement atomic save operations. Those did not work on Xsan.

Next, there is no catalog search operation. Catalog search APIs that are available in the file manager are very tied to the way the HFS+ catalog file works. Doing the same types of queries with the same type of API is not really feasible on Xsan, so we don't provide that underlying file system hook that the file manager uses to do those queries. And finally, Xsan is always case-sensitive. Jason Thorpe So, HFS+ also has the capability of being a case-sensitive file system.

However, it is not the default for HFS+. And this might seem like kind of a trivial or minor issue, but believe it or not, when we were doing our qualification phase with Xsan 1.4 release, we actually found a test suite for an application that failed specifically because it encoded file names inconsistently with respect to their case.

So, with those characteristics in mind, let's talk about some of the important do's and don'ts. So, first of all, every single one of you, when you're writing an application that saves a file to disk, you should be using one of the pre-allocation APIs that are available to you.

You can do this on any file system -- Xsan, HFS -- any file system that doesn't support it will just give you back an error, which you can just ignore. There's no reason not to do this, and there's plenty of reasons to do this, which we'll cover in just a second.

There's basically two APIs available to you for this. There's the FSAllocate fork API and the file manager, and there's also an F-control operation for applications that are written at the BSD layer, or if you're a Cocoa application, you can use some of the NSFileManager and FileHandle APIs to get file descriptors for those files and then use F-control.

So, there are really two reasons that you would want to pre-allocate files. One is to minimize file fragmentation, which is what I have here on my slide. Now, when you're writing a file out to disk, the whole thing doesn't magically appear all at once. It gets written out in pieces. When that happens, Xsan has to extend the file every time you write to it.

Now, in many circumstances, Xsan is able to make good decisions about where to allocate those blocks in order to keep them as contiguous on disk as possible, which helps with reading them back. You get higher performance reading them back. But especially in cases where you have a large number of clients that are writing concurrently, Xsan cannot always make the best choices. So, you know, we really strongly recommend that your application, which, after all, has the high-level semantic knowledge about what's really going on, we really recommend that you tell the file system what you want to do instead of making it guess.

So, this can actually have a huge impact on performance, and what I'm going to do here is kind of walk through an example, of a scenario where an application was not using preallocation, performed badly, and they used preallocation, and they worked great. It's an application I'm sure you're all familiar with. It's the iTunes Music Store.

So the iTunes Music Store, they actually use Xsan as kind of a key component of their workflow that converts the MPEG-2 content that they get from content providers into the iPod H.264 format that you all know and love. Back when there were only a couple of shows on the iTunes Music Store, they had kind of a human-driven workflow that wasn't perfect, but it was workable. You know, content came in, person sat down at a desk and dragged that content to the SAN and started marshalling it through the process.

Well, pretty soon a couple of shows became a couple of dozen, and this sort of human-driven workflow wasn't working so well anymore. People were staying late, and they started to look and smell funny because they never left the office. And, you know, they realized that they needed to do something a little different. So they realized they needed automation in this process. So they wrote some tools that automated the process of, you know, content gets uploaded, and it gets pushed through the system, and out comes an iPod video.

Well, I was actually standing out in front of a Walgreens near my house one night and my cell phone rang and I got this kind of panic phone call from one of the music store engineers who told me about this horrible performance problem they were seeing and they were under the gun to get some content up before a deadline and could I help them.

So I told them I'd be happy to help them. I took a look, and with some drilling down, we actually discovered that the files that they were processing, they were having so many problems, were heavily fragmented. We're talking thousands, and in some cases tens of thousands, of individual chunks on the disk for each file.

Now, as a result, what was happening is when those files were read back by their post-processing tools, they -- So, really a couple of things. First of all, there's, I don't know, 20, 30 machines on this SAN that are actually doing the post-processing. And the load of those computers asking for the extent list from the controller each time was actually causing the Xsan controller to be just pegged in terms of CPU usage.

It was really busy. Now, kind of a secondary effect of the files being fragmented were that they were spread out all over the disk, right? They were, you know, a chunk here and a chunk here and a chunk there. And the clients were not able to stream that data off disk in an efficient manner. The actual just I/O was slow because the RAIDs couldn't, you know, prefetch the data on disk and stream it out.

Now, what was curious is that there was some content that they had not yet moved over to this new workflow that was still being marshaled through by human beings. And all that was working just fine. You know, their sort of expected performance because of the amount of processing that they have to do -- it was about two times real-time. So a 30-minute show takes about an hour to process. And the humans were basically beating the computers, and that's sort of counterintuitive. Well, it turns out that what was happening is the finder in Mac OS X actually pre-allocates the space for files when it copies into an Xsan volume.

So there were a couple things that we did to kind of, you know, get them over the hump, right? We used some tools to defragment their file system. And then we provided them with a little teeny five-line patch they were able to apply to a Python module that they were using. And they went from the six times real-time that was just killing them all the way back to two times real-time. They gained all of their time. performance back with just a simple five line change to their source code.

So the next thing that all your applications should be doing is using large I/O sizes. Large I/O sizes are much more efficient in a SAN environment. Actually, they're much more efficient everywhere, but the consequences of not using them are a little more readily apparent on a SAN. So there's a couple of reasons why this is.

First of all, when you use large I/O sizes, you're sort of -- Amortizing the overhead of the disk transactions over the wire. There's commands and headers back in the responses. And the more data that you ship over the wire with each one of those, the more efficiency you're going to get out of the actual storage network.

But secondly, so one of the things that operating systems do is they go to great lengths to try to make I/O that goes out to the file system sequential. So when you're talking about a single computer, talking to its internal drives or to a FireWire drive, This is no problem. The operating system lines up all I/O, spits it out, the disk is very happy because sequential I/O is much more efficient for a disk drive.

But think about what happens when you have a lot of computers on a SAN talking to the same pool of disks. Suddenly, even though each of those computers might be doing sequential I/O, by the time it gets to the disk, it effectively looks random because they're all reading different files.

So the bigger the I/O size is that you actually issue out to those drives, the more efficiently the drive is able to schedule that I/O internally, and then it can do sequential I/O within the context of that one operation -- single operation. So it can just perform more efficiently.

So again, we're going to take a look at the iTunes Music Store, because they actually had a little performance problem with an application that they used to upload content from the content providers up to the SAN to start processing. So again, we're going to take a look at the iTunes Music Store, because they actually had a little performance problem with an application that they used to upload content from the content providers up to the SAN to start processing.

So I went on over and we discussed what was going on. They were having just kind of bad performance. There was lots of idle time on the CPU and made a couple of suggestions to this application developer. The first one, of course, was make sure to pre-allocate your space on disk. But as we talked a little more, we came to discover that they actually had some code in this application that would buffer up the I/O to 64 kilobyte or 1 megabyte chunks and spit it out to disk.

in those big chunks. But that code was not enabled on a Mac OS X platform. Instead, what they were doing is when a packet came in from the network, they were extracting data out of that packet and then just spitting it out to the right place on disk. So remember, the Ethernet MTU is 1,500 bytes, and so for every three packets that they were getting from the network, three packets or so, they were actually having to read a file system block, modify part of it, and write it back out. So it's basically three times the amount of I/O that you would normally need to do.

So I made a little suggestion to them that they actually enable this buffer at I/O for Mac OS X as well. And kind of for two reasons. One, just for all the benefits of using a large I/O size, but also, you know, if you're buffering it up in a nice, you know, rounded power of two increment, you get to write it out on block boundaries, which means you eliminate all those read-modify-write cycles in the file system.

And as you can see, the results were absolutely outstanding. They got a 23 times improvement in performance by simply turning on buffering in their code. It went from 10 megabytes per second to an aggregate throughput of 203 megabytes per second. Now, how many of you watch the Daily Show or the Colbert Report or anything like that? Yeah? Do you watch it on iTunes? Sometimes, maybe? Okay. It would not be possible to get the content uploaded on a daily basis without this change. So every time you get your fix, you're going to get a new one. just remember that this made it possible.

So I expect this is going to be kind of a controversial suggestion, to use the setAddrList system call for a couple of reasons. One, it's not a POSIX system call. It's available at the BSD layer, but it's only available on Mac OS X. And second of all, it's not really the most straightforward API to use. However, if you are an application that, you know, say you're a file service application, You can change the attributes of files frequently.

This enables you to reduce the number of messages that you send to the Xsan controller. Remember, the Xsan controller is connected to a network, and the communication latency is network-like. It's not just like modifying a data structure in memory. Jason Thorpe So, by using the setAddrList system call, instead of discrete chown and chmod and chgroup and utimes and all those other things to change file attributes, you can collapse all that stuff from the end messages into one and get better performance.

The example I'm going to turn to here is the Mac OS X AFP server. Some time ago we were investigating an escalation we got from a customer. They were having some bad performance when they were copying a folder that had tens of thousands of small files in it to an AFP share that was backed by Xsan volume.

In the end, we actually made several changes to the file system that improved the performance for the customer. But one of the things we noticed during our investigation was that we saw a lot of what seemed to be very redundant set attribute operations going from the client to the controller.

So we drilled down in this a little more, and one of the places we looked was inside the Carbon File Manager. So that's the framework on top of which the AFP server is built. It turned out that there are two code paths in File Manager: one for HFS+ and one for basically everything else, for positive volumes.

It turned out that there are two code paths in File Manager: one for HFS+ and one for basically everything else, for positive volumes. So what we did is we modified the file manager to use the same code path for file set attributes for POSIX files as it did for HFS+.

And just with that simple change, we actually squeezed another 10% out of the little test that we were using, which was to copy a large folder of Keynote presentations that had a bunch of small files in it. And we gained 10% on that benchmark, which if you're doing performance tuning, 10% is not really anything to sneeze at.

So the next thing that you're going to want to avail yourself of is the Disk Arbitration Framework. We actually recently ran into an application that started early in the boot process, and it was sometimes failing in strange ways, and sort of more to the point, it was actually causing Xsan to fail in strange ways because it was making bad assumptions about whether or not the Xsan volume that it expected to be mounted was actually mounted. It was testing certain conditions, and when those things evaluated to true, it assumed that the Xsan volume was there, even though those conditions really had nothing to do with whether or not the volume is actually mounted.

So we suggested to that application developer, "You really ought to be using disk arbitration, because it can do a couple of things for you." One, you can ask it if a certain volume is mounted, and it will give you a definitive yes or no. You can also register for a notification from the framework whenever a volume is mounted, or also when an existing volume goes away.

So, I guess my broader point here is that, as of Tiger, we all live in this world of deferred initialization. We are moving away from startup items and going more toward launch-D. And this overall is a very, very good thing. There's a lot of benefits to that. Your systems boot up much faster now than they did in Panther, and that's largely due to the fact that we have deferred initialization.

However, it can sometimes break assumptions that applications make that are bad assumptions in the world of deferred initialization. So, please, make use of all the different types of notification facilities that are available in Mac OS X, not just disk arbitration, because it will make your lives a lot easier in the long run.

So next, the resource fork. Do not use this. Please, please, please. As I said before, we're a single fork file system. All that resource fork stuff is emulated. And as I said before, it's not always perfect. And it leaves these other little files that Mac OS X knows about and knows how to process.

But other systems connected to the SAN may not know anything about. And remember, XSAN is actually compatible with some third party SAN products. And those Linux or Windows systems don't know about this Apple double file. So if you rename your file, suddenly you've lost your resource fork information. And that's not good.

So the new way to do this in Mac OS X is to use bundles. So I realize that for a lot of you who write applications that use a resource fork, this is a file format change. But in your next major release, I strongly encourage you to consider making this change because you're just going to gain kind of broader compatibility everywhere. And there are both C and Objective-C APIs, the CFBundle and S bundle APIs. So it's available to a wide array of applications.

Okay, FS Exchange Objects. This is something that you should not be using either. This is an API that's in the file manager, and as I described before, Xsan doesn't support this. What Exchange Objects does is it allows you to take one file and another file and atomically swap the data forks of them.

And what applications use this for is to implement a safe-save operation. That is, basically, when a user saves a file, they expect either the new contents to be there or the old contents to be there, and not a mixture of both. And that's what this API call allows applications to implement.

So, it's not available on Xsan. It's also not available on UFS or FAT or anything like that. So, what can you do instead? Well, what you can do instead is you can create a temporary file, and then you can rename the temporary file to the original file name. And on all the file systems that we support, that is also an atomic operation and gives you the same semantics. There are some caveats, however.

When you use Exchange Objects, you're only exchanging the data forks. So all the other attributes of the original file, like the owner and permissions and all that stuff, remain intact. However, when you create a temporary file, the owner and the group and the permissions and the timestamps and all those other things are going to be different.

So your application has to be aware of that and actually copy the attributes to the temporary file that you want to preserve from the original. It's a little tedious. There's not really another way to do this, especially in a Carbon application right now. However, there are APIs in Cocoa that do this for you, so maybe you can consider switching to those. Any of the classes that provide sort of write-to-file atomically methods actually do this work for you. There are other classes that don't provide this exact method but that implement the methods that they use on top of this one. And this document is one such example.

So those are the do's and don'ts. Those are really the main key points that you as application developers need to keep in mind when you're developing applications and you want to maximize both your performance and your compatibility with Xsan.

So, how many of you are excited about those cool new Macs that were introduced earlier this week? You like those? Raise your hand, yeah? Guess what? Xsan now works with them.

So we actually are shipping universal binaries in Xsan 1.4. And in addition to Xsan being ready for those new machines, those new machines are basically ready for Xsan. They actually -- all of them come with dual-gigabit Ethernet networks, so you have -- you don't need to buy another network card in order to have that private Ethernet network. Really, the only thing you need to buy for them is the PCI Express Fibre Channel card.

Last year at WWDC, I got a lot of questions about this next one. A lot of people were asking me, "So there's this new snazzy access control list architecture in Tiger. Does Xsan work with it?" And unfortunately, my answer last year was kind of a meek "no." But I have good news. This year, I can say that Xsan 1.4 supports access control lists.

Not only does it work just like Access Control Lists work on every other file system in Mac OS X, it also interoperates with ADIC's StoreNext file system and the Access Control Lists that they support on their Windows client. So, and like always, there's lots and lots of bug fixes. So all told, we made over 100 software changes in Xsan 1.4. And a lot of those were actually in direct response to you guys filing bug reports through Bug Reporter or participating in customer seeding, so I'd like to just take a minute to thank you for providing your feedback.

Jason Thorpe And for those of you who don't participate in that, I'm going to encourage you to participate in both our customer seeding program and our developer seeding program for Xsan. You get early access to Xsan releases, and we like to hear feedback, and of course, we'd like you to test your applications with Xsan.

So we're going to go ahead and wrap up here, just to kind of review some of the points we've touched on. First of all, Xsan allows for shared volume access at high speed directly over a fiber channel. And it also allows you to configure your storage in a flexible manner that works for the type of work that you need to do. Of course, applications need to be aware of Xsan's somewhat unique characteristics. But more importantly, they can be tuned for better performance on Xsan once you're aware of those characteristics. And then finally, of course, Xsan now supports the new Intel Macs and Access Control Lists.

So finally, again, I just want to reiterate that ADC Select and Premier members, you have early access to these Xsan seeds. I really, really, really encourage you to participate in this. The more feedback we get, the better. We love it when you guys use this stuff early. And one of the things that I'm often asked, even within Apple, by other teams in the company, We don't have an Xsan. We don't know how to test our stuff with Xsan. We don't know how to fix this bug that people are seeing because we don't have a SAN. Well, you can actually use Firewire for basic Xsan compatibility testing.

This is especially good for smaller shops. You don't necessarily have to break the bank to buy a bunch of stuff in order to just do basic compatibility testing. I'm going to be working with Developer Technical Support to get an article written. You can just have one place you can reference for the steps to do this. It's very simple. It's basically exactly the same as setting it up with a regular fiber channel, except there's less stuff to hook together.

That's really about all I have for you today. Just to kind of reference you to some more information, I really strongly encourage all of you who are either running Xsan or thinking about switching to Xsan to go to the Xsan Best Practices session that is in this room this afternoon at 3:30.

JD is going to be presenting that. He's got a lot of experience in doing Xsan deployments for customers. Bring your questions. He's got a lot of good info for you guys. And I'll actually be on the Q&A panel for that as well. So any questions that we don't get to today or if you have something you think of later, I'll be available at that time.

And then I just want to point you to just a couple of places online. First of all, the Xsan webpages at www.apple.com/xsan. In addition to basic product information, there are links to the support pages, and also there are frequently updated links to new customer deployment scenarios that make for an interesting read.

Also, there's a great mailing list out there. If you're a Xsan user, you can find a lot of Xsan users at list.apple.com. There's quite a vibrant Xsan user community out there. For those of you who are using or deploying Xsan in your IT environment, this is just a great place to meet other Xsan users and bounce ideas and problems off of them. And for those of you who are developing applications for Xsan, it's a great potential place to find testers or people to buy the stuff.