All right. Good morning everyone. Thank you very much for showing up at a 10:30 session on Friday morning. I know between the bash and plane flights out of here it might be a challenge for some. But thank you, we really appreciate it. My name is Jason Thorpe, I'm the Manager of the Directory and Sharing Technologies Team for Mac OS X server, and that includes Xsan.

I've seen a lot of you at, at previous WWDC, and I see familiar faces and a lot of new faces here, so welcome. We've got a great session here for you today talking about Xsan configuration, optimization, and integration best practices. And so without further ado we've got a lot of content to cover. I'd like to introduce JD Mankovsky.

Thank you Jason. Good morning.

So my name is JD Mankovsky, I manage Apples Professional Services. We've been doing a lot of Xsan deployments for, for many, many years. And with me today we have Jason Deraleau and Ken Holden who are on the Professional Services Team who actually do a lot of implementations across the country.

And they're going to share a lot of the best practices. This session is something that we've been doing for many, many years, so we try to, you know, every year kind of go through some new best practices, as well as for those of you who are new to Xsan make sure that you've got some of the fundamentals. So, first thing is to make sure you're paying attention. I know it's early in the morning.

I have 3 questions for the people in the room here. First of all, how many of you are looking to deploy Xsan in the next 6 to 12 months. Wow, so there's a lot of new people in the audience. How many of you actually are consultants that are, you know, deploying Xsan or focused on deploying Xsan? OK. And how many of you already have Xsan deployed today? Great. So I think we've got a good mix of a session today to cover, you know, a lot of interesting tidbits and share some best practices with you.

So we appreciate you coming today. So, you know, this is kind of a high level agenda what you're going to learn today. We're going to talk a little bit about where Xsan is being deployed. You know, we're going to talk about, a little bit about Xsan 2.2. So as developers you guys have access to the upcoming version, and Jason will talk about that. We'll talk about some best practices, some backups, solutions that are available for Xsan, obviously. Some tuning.

Some debugging. And we'll talk about Promise, who's the storage vendor of choice that plugs in, that is supported by Apple. And some updates from them as well. And then we'll finish with the Q&A's. Our session should be about an hour. And we'll have at least 50 minutes for Q&A.

And there'll be a lab this afternoon, you guys can come and talk to us. So the first thing I wanted to mention is, you know, Xsan has been around for, for many years now. And at the beginning, you know, we were very focused on, you know, the video customers.

And I'm sure a lot of you that, you know, are looking at deploying, you know, Xsan for, you know, video deployments. But it's been an interesting, it's been an interesting 2 years, because we have a lot of other customers that are coming to us wanting to, you know, to deploy Xsan and other, you know, areas. And one of them is forensic.

So, you know, you might be familiar with forensic, you know, government agencies do a lot of forensic work. They might be capturing, you know, laptops, and wanting to do a, you know, forensics analysis on the hard drive. A lot of, you know, commercial companies who have possibly disgruntled employees, or one needs to do analysis on computers, they all use, you know, they all do forensic work some way or another through their security groups. And what's interesting is that obviously requires a lot of storage.

Right? So look at the hard drives today, you get a new Mac Book Pro with a 500 gig hard drive, if you need to do any analysis on that, that takes a significant amount of storage. And so, you know, I wish we could actually show you a picture, a real picture of what this looked like.

Well you can imagine like a, you know, 4 racks with about, you know, 32 Promise units. This is a 3 letter agency in the,D.C. area. But they basically deployed Xsan for forensic use, and it's about 250 terabytes of usable storage, so we don't really like talking about raw storage, but it's usable, 250, one single volume, 250 terabytes.

And when deploying this solution they saw improvements in terms of throughput to do their work in the order of, you know, 20 times the speed of what they were accustomed to before using, you know, a NAS type, you know, solution, with, you know, over a gigabit Ethernet. So they really sought tremendous improvement.

And, and so we've been deploying quite a bit of forensics fans in the past, you know, year or so. And then obviously there's some really new interesting technologies, you know, Podcast Producer is going to be what we, you know, think huge with PCP2. And I'm sure a lot of you are looking at deploying Xsan as the backend to Podcast Producer.

But also final, Final Cut server has also really picked up since its introduction. And that really is a great, those are two great solutions that integrate well with, with Xsan. So with that I want to turn it over to Jason to kind of give you a high level overview of Xsan. Thank you.

Good morning everybody. So amongst all the hands there I saw we did have some newbies to Xsan. And so I want to just kind of start out with a bit of an introduction to Xsan for those of you who are not quite as familiar with the platform.

Just kind of a quick review here in the idea of how we've come to use clustered file systems. If you think back to the classic example of a local file system, you know, you have a hard drive inside your workstation, and then that's going to be formatted into some kind of format, such as HSS Plus, or NTFS, you know, even FAT32.

But basically the idea is that I have a storage device that's going to be most likely directly attached, you know maybe a hard drive over setup. Some kind of local bus like FireWire or USB. Or this could even be a, you know, like an array of disks, like a, arrayed fiber ray that is set up as the logical unit of LUNs.

And basically the way that this is going to work is that when my operating system wants to request data from the storage, it's first going to go over that local bus and take a look at the metadata, right? So the metadata and the data are stored on the same physical device. Just a quick side note, no metadata's going to be your information like what particular blocks on the disk contain that data? Who owns it? What are the permissions? You know, that kind of thing.

So once it locates the metadata on the disk that's going to again be piped backup to the kernel, which will then go back down to the device and request the raw blocks. Now when you, to have this kind of scenario the problem is this doesn't really scale well for multiple use, right? It's great when I'm using it within the confines of a single machine, but when I want to share this data with other users I'm then going to have to find some other method of distributing it. So the next step in this evolution was network files systems.

And the idea here is now I'm using some sort of protocol, such as NT, I'm sorry, NFS or AFP, or SMB, and we're kind of just abstracting this out one level. So instead of having, you know, this, still this hard drive is kept within the confines of a single server, but now that I'm kind of dividing it out and making it available over these different protocols, I can share that with multiple clients.

So in this case, now we're going to have the client initiate the connection over AFB again, and that's going to cross over the Ethernet. Right? And now the server is going to perform those same kind of operations that we saw in the previous slide. So the idea is that it's going to go over the local bus, connect into the storage to locate the metadata, and then from there find the actual blocks and pipe those backup to the client. Now again this is great when you want to share this information over a connection such as Ethernet. But the problem is again this isn't going to get you the, you're still kind of limiting yourself in that bottleneck.

So the next step along the evolution was to a clustered file system. And in this case what we're going to have is a storage system that is going to use most likely Fibre Channel. We have an integration of a couple of different networks, so for example, the, Ethernet network for metadata, an Ethernet network for the front side of the house. And because of these different systems working in concert we're able to share this storage to even more devices.

So in the case of a clustered file system what's going to happen here is, again, you know I'm going to have my client initiate a request. But this time it's going to go over Ethernet, over the metadata network. And the idea is I have these two servers, or, well, at least two servers in your best deployments.

But basically those are going to be responsible for locating the data on the storage. So the metadata controllers will go over Fibre Channel at that point and access the metadata LUN to locate the metadata that's related to the particular request. Once that's occurred that information's going to be sent back to the client, which will then go over Fibre Channel to access the data directly. So the idea here is I have a little bit of a slow down as far as when I'm going to look for that data initially, but once I know where the data is on the storage I can over Fibre Channel and access it over fiber.

The other advantage of this, of course, is that when I have this, these larger rays of disks, I'm much more able to expand that storage down the road, I'm able to allow concurrent access to the storage. And really the critical part of this that might differentiate it from other SAN products is that multiple clients can access the storage at the same time. This is phenomenal for when you're working with large data files and you want to be able to distribute that load across different machines. So one of the things we introduced in Xsan 2 is this concept of volume presets.

And for those of you who kind of worked with the product as you have come up through the years, you may remember a lot of this was sitting down with a calculator and figuring out, you know, what's the optimal striping, or the optimal block size for my particular application? In the earlier days this took, like I said, a good amount of work to figure out what was going to be best, especially when you're working with Xserve RAID and figuring out what the optimal settings were for your buffers.

But with Xsan 2 we've actually done a lot of that work for you back in the lab. And the idea is that when you're initially configuring your storage and setting up the volume, you have the ability go to in and choose a preset that kind of fits the type of deployment that you're doing.

Now what are these things really working with? There's several different variables related to an Xsan volume that are paramount in how the storage is going to be used. And a great example of this is storage pool sizing. And what we're doing here basically is determining what is the best ratio of LUNs, which are again like a a set of disks working together as one logical unit, and the number of pools that we put these LUNs into to allocate the storage. And basically the key concepts are that we have more LUNs, there's going to be more spindles working at that data.

So if I need a lot of throughput I'm going to want to use a larger ratio of LUNs per pool. While the other factor that we're kind of working against is that pools, in order to work best, should be uniform in size. So if I have, for example, 4 LUNs per pool and I want to grow my storage down the road, I'm going to have to, again, buy another 4 LUNs worth of storage.

So it's kind of a balance between the cost of upgrading and expanding your storage versus the throughput you need to support the type of access you're going to be supporting for your clients. Another variable we're taking a look at is going to be the block and stripe sizing of the storage.

And basically the idea here is how are we allocating chunks of data across these LUNs as we stripe them? And the thing we want to look at here is, again larger amounts of data are going to help with throughput. So if I'm using a larger block size and stripe size I'm going to be able to read larger chucks of data at a time.

This is great when I'm working with large continuous files where I need a lot of throughput to read in all that data. However, if I'm using an application, such as, you know, maybe file services or something where my files are smaller, you know I'm going to kind of steer towards a smaller block size to conserve space, and I think everybody here can remember the old days when you used to have huge blocks, and you just, you know, you allocate a file and it would chew up a lot more space than was really required for that data. So again, these same kind of concepts apply here, but we're just applying them to large amounts of hard drives at a time.

One last variable we're taking a look at is the allocation strategy of the volume. And this basically comes down to the way that files are allocated to different pools. And depending on the allocation strategy, you can actually make different use and make sure you're getting the optimal amount of space out of the, I'm sorry, the optimal amount of use out of the storage. But you also want to make sure that as you allocate files to the storage devices that you kind of balance them between different pools of storage.

The idea is that, you know, as my clients go to request data, if I have multiple clients accessing that storage, they're all going to kinda be vying for that same set of spindles. So if I can distribute the data across different pools, then different clients can access different storage simultaneously.

And by the same token, when I'm working with applications, such as video, where I have multiple streams of HD I can have my client kind of rotate between these different pools of storage. The most common one you're going to be seeing with this deployment is going to be a Round-Robin distribution.

And basically the idea with Round-Robin is that as the files are getting written out to the storage, each file will be alternated amongst a different pool. So if I'm, you know, ingesting different streams of video content, that first stream might go into the first pool, and then the second stream into the second pool, and so forth.

This is again, great to balance out the storage when you're working with it, but you kind of run into a little bit of a headache when you go to expand the size of the storage, and this is because at that point the new storage you're bringing into the environment is going to be not, you know, not as full as the existing storage. So what we've seen is that when you're working with the allocation strategy, you can actually change this after you've set up the volume. So I add that new storage, and I could choose the balance strategy.

And what the balance strategy, what's that going to do is fill up that, that new storage until it's about as even as the other ones. And at that point I could switch it back to Round-Robin to again, rotate the files across that storage. So once you've picked out an allocation strategy, and you have the right stripe size, and you've got the preset that you think is going to work best for you, there's some things you can do to kind of check on how this is really working out, right? And basically the idea here is that I want to go in and determine, have I set this volume up right? Is it supporting the application I need? And for that there's a couple of different tools you can use.

And for example, you can go in and the real simple one is, you can just grep the cvlog file. And in that file you just look for the term SUMMARY. And every hour or so the FSM daemons that host the volume will log some information in there. Specifically what you're looking for is the amount of time it takes to write to metadata. In most Xsan deployments your bottleneck really becomes metadata. You're looking at the Ethernet connection to get to that metadata network, that's really the slowest part of this entire system.

Everything else is going to be done over 4 gigabit, 8 gigabit fiber and if you have the new stuff. So basically what you can do is grep that log, look for these values, and you'll see a minimum average and maximum time it's taking to read. And in our testing we've seen that, you know, the best values are going to be under 500 milliseconds or so.

So if you see anything beyond that range you might want to be concerned, because the degradation of metadata performance will have a profound effect upon the rest of your access to the volume. Some other simple stuff you could do, you know, monitor copy performance. Sit there and use the stopwatch on your iPhone, see how long it takes to copy that file over to the storage and the finder. Remember, what you're really doing here is testing what your user experience is going to be like.

So in order to tune for the application you want to make sure that the storage is set up in a way that allows your users to have a pleasant experience. Another similar thing you can do, you know, do some UNIX copies and use the time command. And just see how long it's taking to work with that storage.

Make sure it's as fast as you can get it, because otherwise you're going to hear about it when the users are waiting for that work to get done. A couple of other places you can look, you know, don't exclude your devices. A lot of the newer Fibre Channel switches have some tools in there.

You can use SNMP monitoring. A lot of them have like a web interface you can go into and take a look at. But basically look for information on, maybe, errors; know how long they might get some latency information about the fiber performance. Whatever you can find from the switch vendor, and they'd probably know better than we would. As far as storage devices go, you know there's some great stuff in the different storage arrays that are coming out in the market right now.

We recently partnered up with Promise and JD will get in this a little bit later. But in the new Promise firmware there's actually some options in there to monitor performance of the storage device, so I can see information about how long is it taking to seek across the disk, how long is it taking to fill those buffers and make sure that I'm getting the throughput I need for my application. So let's just kind of briefly go over some of these different volume scenarios and give an idea of what they do. They can kind of be divided into a couple major categories.

The first ones are going to be video solutions. So for this what you'll tend to see is that you're going to again use more disks, because you need to support that throughput for these high bandwidths applications. So in a video deployment with a video preset you'll tend to see a volume allocation strategy that's going to be again Round-Robin. You're going to see some presets that are along the lines of using 4 LUNs per pool, so that I can have many, many spindles working on that data.

And again, we're going to tune for a larger number of, for large files for these operations. You know? Anybody who's worked with a video workflow can tell you that HD video, and especially when you're getting into the higher resolutions, takes a lot of space. And basically in order to tune properly for those types of client applications, you want to make sure that you're using an Xsan preset that will tune for large files and high throughput.

So some of the workloads, I'm sorry, some of the presets you'll see that work with this will be something along the lines of Final Cut Studio, Final Cut Server, Podcast Producer. You can actually tune it for a particular type of codec, so there's options in there for uncompressed HD, or standard definition, or whatever's appropriate for your application.

Another major category of the types of presets that you're using is for IT solutions. And basically what these, the focus tends to be on a smaller file size. Typically with an IT solution your bottlenecks not going to be really the fiber access, but it's going to end up being Ethernet.

So for solutions such as mail servers, you know, PHDs, this kind of thing is, the limiting factor's going to be that Ethernet on the other side of the server. So with these types of volumes what actually ends up happening is the most, the most performance, the, I'm sorry, the key point of performance to monitor is going to be metadata performance.

If you think about a video workflow you tend to have a small number of large files, and that's not going to incur as much data, metadata use as using a large number of small files. So with an IT solution what you're looking at is, again, optimizing that metadata performance. The metadata is going to be the bottleneck in your deployment. A couple of things we've found pretty good luck with, with this is, we've actually started starting using multiple metadata LUNs.

This is something's we've been testing, and really the key benefit of this is that when you're using more than one metadata LUN, the Xsan software can divide up the file system journal, and the file system metadata. And what that allows us to do is as it's working with this data it can kind of split the operation between those two different pools.

So that allows the different spindles that are working on that data to be able to work on it simultaneously. The other big focus when you're using IT solutions is going to be conserving space. So what you'll tend to see are smaller block and stripe sizes, and you'll also see that, that are going to use fewer number of LUNs per pool.

So that the, first of all your expansion of the storage is a little bit cheaper for you, and also again it's going to make more, more conservative use of that storage as it goes through the system. As JD was talking about earlier, you know, a new area we've been looking at is computer forensics.

And initially you might this is real similar to the file server solution, right? But really the thing here is these files are big. If you're working applications like NKFDK you're taking images of entire hard drives. And, you know, in the newest machines are coming with 500 gig drives, people are, it's not unusual to see RAID arrays, especially in some of the newer, newer gamer rigs and stuff like that.

So you need a lot of storage to be able to keep all this forensic information. Because it is a newer area of our system we aren't really, we don't have a preset necessarily for it. But you do want to use a larger number of LUNs per pool. Basically this allows it again to put more spindles at it, and also going to be using a more throughput to support that application as it goes to read these large files from the storage. So you definitely want to tune for a larger file size.

And make sure that the storage is going to be supporting that application in the best way possible. So once you've gotten everything all set up the key point you want to do is backing up your configuration information. I know I, we were actually, before the session, we were just talking about an email came through where one of our clients had a problem where they accidentally overwrote some information on their Xsan metadata controller and after that they could no longer find the data on the SAN. And really the information's still there, right? Because they didn't erase the files necessarily, but because they didn't have the geometry information, the Xsan metadata controller could no longer locate, say, the metadata LUN.

So these files are critical. And you want to make sure that anytime you make any kind of change to your SAN, whether it be expanding the size of the storage, changing allocation strategy, adding clients or controllers, backup this file. Easiest way to do it, run a CD gather on all the metadata controllers, and basically that, you know, take the resulting file and send it to your MobileMe account, email it, print it out on the wall, do something to keep it. And basically this is going to preserve all of the geometry information related to your SAN.

Again, run it on all of the metadata controllers, so that if you have a problem with any of them you can restore it back at a later time. If you're particularly paranoid, you know, go ahead and use a launchd job to back it up weekly Maybe you're using Time Machine, and Time Machine's going to preserve the files.

But no matter what you're doing, you want to make sure that you have this information for down the road in case you have a problem with the metadata controllers. And with that I'd like at this point welcome Jason Thorp back on the stage, and he's going to give us some information about Xsan 2.2.

Jason Thorpe: All right. Great. OK. So you may have heard us talking about Xsan 2.2 in some of the various labs. This is some actual details here. And this is actually available to you on the developer website, the seed is available now. Basically what we've done is we've fixed a lot of bugs, a lot in Xsan Admin. But the primary driver for this, of course, was compatibility with Snow Leopard, the new version of the operating system. That's kind of the main feature.

However, we didn't stop there. We took this opportunity to, since we have a sort of better foundation in the operating system, took this opportunity to make some improvements in the actual file system itself. And the main one there is we now have support for Native Finder info and resource fork information, and extended attributes, and all that kind of other good stuff that makes files systems sort of first class in Mac OS X environment.

So the big thing there is no more Apple double files. Yeah. Yeah.

Nothing like cutting the number of files in your volume in half, right? So that feature is available for systems that have sort of all new clients. We do support legacy clients in the SAN. And there'll be a migration path for that, so you can start out with your existing volumes and actually upgrade them to the sort of native way of doing things. Or if you have older clients, understand you can keep the old way.

So this will be an Intel-only release. We are not going to support Power PC in this next update. However, Xsan 2.1.1 clients on Power PC will still be supported. And we are removing support for Xsan 1 clients. So everyone should be up on Xsan 2 by now. If you're not I really strongly encourage you to do so, because we fixed a whole lot of bugs in Xsan 2. And that pretty much is it. I will be happy to answer questions about Xsan 2.2 during the Q and A, and also in the Xsan lab later this afternoon. And with that I'm going to bring up Ken.

Thank you Jason. So I'm here to talk about best practices today. As many of you know if you've deployed Xsan in the past, this is simply just installing the Xsan software, configuring it, and moving forward. As there is many third party components that make up your Xsan, each with their own configurations and best practices. So as JD mentioned earlier, we've deployed, you know, hundreds of SANs since this initial release. And through a lot of trial and error, and field testing to come up with a set of best practices that really encompass the third part devices as well as the Xsan software.

So if you're going to be deploying Xsan we highly recommend that you follow some of these Apple best practices in order to have a solid Xsan deployment. So the first thing I want to talk about is networking, and specifically I want to talk about the metadata Ethernet network. And this is something that a lot of times is overlooked.

Each time your Xsan clients try to communicate with the Xsan volume, the communication is handled by your metadata controllers over this metadata gigabyte Ethernet switch, or excuse me, just regular network. So we recommend to have the fastest switch available with the lowest latency. And therefore we do recommend gigabyte switches. It's not a requirement, but we have seen clients that have used 100 megabit switches and it has some latency.

So we definitely prefer them. And also with the switch we recommend that it be a dumb switch So that's good, because you don't have to go out and buy a $10,000, you know, Cisco gigabyte switch. You want something that has less management capabilities, or little to no management capabilities.

As any of those advanced features can actually cause latency on your switches. Also you want to be sure that plugged into this switch is only your metadata clients and your, your, your Xsan clients and your Xsan metadata controllers. So avoid VLANing you switch and using it for your public's networks as well.

When you're IPing your Xsan clients and metadata controller, stay away from the ACP. The ACP with static is supported. But just general the ACP we want to avoid. And also you want to make sure that the network used for metadata is unique. So when you're deploying it the best thing to do is before you configure your metadata interface with your, your private network that you're going to use for metadata, be sure to try to ping the broadcast address to that address space that you're using. So if you're using 192168.1, with a, you know, 524, just ping the 192168.1.255. Make sure it's not pinging out the public interface and getting any responses. If it is you want to choose a different network.

You want to make sure that that network is not routed anywhere else. Also if you switch has Port Fast capabilities you want to be sure that you disable, or excuse me, enable port fast. If you don't and the switch supports port fast, what can happen is your operating system is going to boot prior to having a network link established, and this is going to cause issues or the bind not to mount at boot. So next thing, DNS.

If you heard in many other slides, or many other presentations here this week is, you know, DNS is critical. Xsan is no different. Actually Xsan is, I find its even more finicky with DNS. So we definitely want and require forward or reverse records for every metadata and public interface of every Xsan client and metadata controller. You want to make sure that this is configured properly prior to even installing Xsan.

So, so definitely important. When you're deploying that and you're pulling your own DNS you can use one forward domain. So if you have example.com, you can create records for your servers as server.example.com,. pointing to your public IP address. And for your metadata you can create records like server-Xsan.example.com pointing to your metadata IP address. So for there you have a public zone, and, a forward zone, excuse me, and then two reverse zones on your DNS server.

And your DNS server doesn't need to be on the metadata controllers. I mean it can be in smaller networks that's fine. But in your larger environments may have active directory DNS, you may have 2 IP or another BIND solution, or DNS for your open directory. So you can utilize that. We just care about forward and reverse records for each metadata and public interface of every client and metadata controller.

In certain situations you may find that your DNS admins don't want to host a reverse records, a reverse zone for your metadata IP addressing. We see this a lot in the federal space where the AD admins don't want to host a 192168.X or whateverh in their DNS. So in these certain environments where they don't allow us to run DNS on our own servers and we're not allowed to have that second metadata reverse domain, we're able to use host files for the metadata.

So we don't recommend, we always recommend using DNS, but in the situations where you need to use host files, just be sure that the host files contains, you know, metadata IP to fully qualified name of every Xsan client in metadata, and make sure that their metadata controller, and make sure that that host file is copied to every client and metadata controller prior to installing Xsan.

So once you've established proper DNS you want to make sure that each client in metadata controller can query the names, or query the DNS, for both forward and reverse. So you can use commands like dig, or nslookup, or host to, just to run a check to make sure that the IP translates hostname, and hostname translates IP.

The second thing you want to do once you've verified your DNS is you want to make sure that the metadata controllers fix the servers and the clients, that what they think that their local hostname is matches what DNS thinks it is. So for servers you can use the change IP command -checkhostname.

And this will actually forward the name servers for the, what DNS believes what your hostname is, and then it also asks the server what it thinks its hostname is and tell you if it's valid or not. If it's not you definitely want to run change IP to change the server to match the hostname to the DNS name.

For clients you can use the scutil command to either get or set these settings. So convenient over ARD you can just, if you have a lot of clients just push that out to all your clients. So next directory services. Xsan uses a file level locking to allow multiple clients, to simultaneously access your Xsan and read and write to it.

Read, write features are if your client's writing to a file that's the only client that can write to that file while he's writing to it. Once it's done writing to it other users can read and write to it. But this is achieved by using directory services. The directory services are manual, or excuse me, mandatory for, for Xsan. You can use Open Directory, you can use Active Directory, or you can use another open, or L.

based directory services. So for smaller Xsan deployments where you don't want to, you don't have active directory or you don't want to manage open directory on your own, during the Xsan set up you can actually choose to have Xsan manage users in groups for you. What it actually does is it creates an open directory master on your metadata controller. And it gives you the option in Xsan admin where you actually see users in groups. So you can manage users and groups and quotas from within Xsan Admin, rather than having to go to Workgroup Manage or Active Directory.

So for smaller environments that's totally acceptable. Access control lists very important for SAN. It used to be that your Xsan admins would modify your umask settings to allow read and write access for all your users to be able to access their, each others work. This is no longer necessary.

With access control lists you're going to be able to define your read and writes for your user polices. And this is also can be done within the Xsan admin. You no longer have to go out to server admin or workgroup manager as you used to manage access control lists. And the other thing that makes you sure you plan out your access control list problems before you deploy your Xsan.

It can also be tricky once you've deployed it and you've, didn't spend a lot of time thinking about it. So you want to make sure that you know who your users are, you know your groups, you know what access policies you're going to need for them. Another thing you want to think about is the ACL inheritance.

They're extremely useful, especially as your volume grows, because you don't have to continuously have to change ACLs to match what your users and groups need. However, it can be really annoying if you haven't planned it out from the beginning, going back and having to fix this on a production SAN.

So just spend a lot of time beforehand mapping out your ACLs. So out of the Xsan users out there, how many are actually backing up their Xsan volume? OK. So again we see a lot of clients that they're not. It actually still astounds me that when we go out to clients and they're not backing up Xsan.

And that professional services a lot of times we're going into worst case scenarios, you know where an air conditioning unit when offline for 20 minutes on a summer day, or they had a water damage from a floor above, and their entire Xsan's lost. And I've seen federal clients; I've seen private clients with 20, 30 terabytes gone, completely unrecoverable. So, backup your SAN.

When you're deploying your SAN that should be the first thing that you're planning when you're ordering your equipment is also a proper backup solution. And also as Jason mentioned, backup your config files. When I'm changing, or when I'm doing anything with Xsan admin, prior to making any change I run the CD Gather, because I want to be sure that I have those files.

And as Jason said when we just saw that client yesterday complain about they now cannot recover their SAN, because they do not have a current, current backup. And we've had to rewrite some of these files before, by memory and manually. And sometimes we've had some luck doing it, but it's sometimes very difficult. And a lot of times without the configuration files you're really hosed, so.

And this slide shows five backup vendors, three of which have backup software servers that run on OS X. So that's going to be TIMEnavigator by Atempo, ARCHIWARE PresSTORE, and BakBones NetVault. All these vendors on the slide states that they have Xsan compatibility and they will backup Xsan volumes. No problem, we use them in the enterprise regularly. Also if you have existing backup architecture in your enterprise environment, two very popular clients Tivoli TSM and Veritas have clients for OS X that actually backup Xsan. So very, very important.

If you're setting up a fiber based tape backup library, a couple of things you want to keep in mind. You don't just want to plug your library into your, to your fiber switch and go. Cause actually it can, it can blip your SAN. You want to be sure that you zone your fiber, your fiber tape library accordingly. And the way you do this is you have to have a backup server that's got multiple fiber ports. You want to make sure that you zone only one of those ports from the backup server to your tape backup drive and library.

This is, this is to prevent fiber loops. Because the tape drives only have one fiber strand, where as your cards have 2 or 4. So it can actually cause a loop. And you also want to make sure that you turn off RSCN suppression on your fiber switch for your tape drive, specifically.

And I'll go into that in a second. Oh, one thing I also want to mention about tape backup. If you want to avoid having to plug your tape library into your fiber switch, you can just sometimes buy an extra fiber card or a 4 port fiber card, and use 2 ports for Xsan and 2 ports directly attached to your library. It's another way of configuring a smaller library based solution. Oops. Going to go back here. OK. Next thing we want to talk about is file manager.

You have your file, you have your Xsan configured for the type of SAN that you're going to be deploying, but there's some best practices in just about how to set up the actual file structure on the SAN for better optimization and performance. So we're going to go into that. The first thing we'll talk about is your video type SAN, specifically in this slide Final Cut.

What we generally would do with any Xsan volume is you want to keep your top level of the volume simple. So you want to have a small amount of top level folders that you use ACLs to prevent users from modifying or deleting these top level folders, and you've also prevented your users from writing to the top level volume in the SAN.

And I'm going to get into that in a second, but the reason why you want to avoid that. But for here at Xsan, you know, create a media folder on the top level, and you know, have your, and then underneath that media folder create a folder for each edit workstation.

And then have your users set their capture settings to that, their corresponding edit folder. And this makes Final Cut Server integration a lot easier, because you can point the watch folder Final Server to your media folder and it will dig down into that file structure. So for general file servers, again you want to keep the top level simple, have a few small, have a small number of folders on the top level to prevent users from writing to the top levels, and preventing your users from deleting those top level folders.

And this example on the screen is what you don't want to do. You don't want to have your, your top of your volume in just a drop zone for anything. And the reason why we've seen is, we've had clients that have folder and file structures that, they go inside a folder that's got thousands of entries.

What we've seen is that when a user traverses into a folder like that it actually can slow down the SAN for every other user until that folder has been redrawn, and all that information is gathered. Because you're metadata controllers are taxed getting the thumbnails, getting the information about the folder, the modification times. So until that, that process is done, the SAN is actually slowed down for your users.

So you want to kind of file, if you look at OS X's general operating system, this is the way we've laid our file structure it, it's a very small amount on the top level, it digs down in. And you want to kind of maintain that level of file systems for your Xsan. Same thing with the general directive tasks, but just Xsan kind of exacerbates it, because your metadata controllers are separate, a separate entity.

So use a lot of ACLs too for, for users and groups access. And move the folder. Oh, also with resharing, if you're going to be resharing your Xsan volume, don't just reshare the actual Xsan volume itself, reshare top-level folders or reshare folders. Make sure your administration of your ACLs a lot easier to handle.

And you can prevent some issues, so. For collaborative services, such as Mail or Podcast Producer, again you want to follow the same laws, small amount of top-level folders. The only thing I really want to say about this slide is, if you have a volume that is optimized for a collaborative service, don't use that volume for file services or file, you know, home directories, or Final Cut. Because what happens is you have a Mail server Monday morning at 8AM, everybody's logging in.

That Mail server needs to have as maximum amount of IO for that, to that Xsan. If you also have your users home directories on there you're going to cause some major bottlenecks. As well as that volume is tuned specifically for that purpose, so create a separate volume for your other needs. So next we're going to talk about fiber fabric and interconnects.

There's a lot of different switches out there, a lot of different third party products, each with their, you know, their own configurations, their own naming of things. But generally the fundamentals are the same. So in this solution we're showing 2 object switches, but it could be Cisco, it could be Brocade, it could be other, other switches.

You want to be sure that you don't connect each client to your fabric, to your fiber switches. And you want to set one port to one switch, one port to another switch to provide best redundancy and multipathing. Leopard brought with us multipathing. So we really want to utilize that.

And with Promise we now have storage multipathing, 'cause you have 4 ports per Promise RAID. So you can truly have a very bolt tolerance environment, such as what's shown here. In this environment if I were to unplug a switch while writing, I would actually not loose a beat.

My performance may go down a little bit, because I've lost multipathing and I may have lost half of my fiber, but the thing is I'm still writing to it, I haven't crashed my volume. I can also shut down the metadata controller. I can also yank out a fiber strand from a client, or even loose a controller in a Promise RAID. So it truly with Promise and multipathing Leopard this design really causes a fault tolerance up SAN.

Now this environment shows 2 fiber switches. But when you're in an environment where you have fiber switches that have blades, or you have multiple fiber switches, you want to be sure that you just isolate your metadata controllers and your metadata LUNs on the same fiber switches. This prevents a lot of cross talk, unnecessary cross talk between your switches.

So for Fibre Channel interconnects and every fiber optic card that Apple ships comes with our 2 or 4 copper fiber cables. They're totally fine to use with your SAN. However, you really want to, they, they have a length limitation is 3 meters. So the reason why you're going to go to optical fiber cable is when you need to go above 3 meters. So for your longer runs, not only do you see multimode LCNs to LCNs, LC connection type, excuse me, LC to LC.

And you're going to make sure that you need to purchase small form factor fiber transceivers for your switch side and your, and your fiber card side. Depending on what type of fiber is going to be depending on how long the lengths, the runs are going to be. As well as you're going to need more powerful transceivers to handle longer, longer, longer hauls and different types of fiber cables. And another thing if you can is avoid mismatch of fiber transceivers, the brands. You don't want to have a QLogic on one, on your server side, and a ProOptic, or Finasar on the other side. Because we have seen sometimes it causes issues and errors in your switch.

So if you can avoid it, try to avoid that. So in this example we're showing some QLogic switches stacked together. And if you're just using other brands of switches you can do stacking as well. What we're showing here is we're interconnecting your fiber switches using 10 gigabit stacking tables.

And the first example, it's showing that we're using 2. And this provides better performance as well as redundancy. And this is, of course, the minimum recommended. You can use 4 if you wanted, or you can use 2 or 4 gig standard fiber cables still also be inner switch lengths.

But we recommend the 10 gigabyte stacking tables. In the second example there's 3 switches. And this one we're not cascading. We're not going from switch 1 to switch 2, and then from switch 2 to switch 3. We're having that switch 1 to switch 3 link as well, to provide multipathing and better performance and redundancy. Same thing with when you go into 4. And QLogic switches, like these actually support up to 6.

And so. So next thing we're going to talk about is fiber zoning. What we recommend is that we create a fiber zone for your storage. You create a fiber zone for each client and metadata controller. And this storage is, it is our zone to only contain the fiber ports of that specific client or metadata controller and the storage.

Therefore the client and metadata controller can only see the storage, but can't see one another. And this can prevent interference from clients and tape backup devices, and also people plugging in things to your switch without having Xsan software installed. This is definitely something that is strongly recommend and it prevents client interference.

And one other thing to note, if you're plugging in Windows clients into your SAN and you're going to use another, say a product to allow them to join the Xsan, if they don't have that software installed and you haven't zoned you switch, Windows clients can actually format and wipe out your entire SAN on the fly while in the middle of using it. So if you zone your switch accordingly, and also another good recommendation is to turn off your unused switch ports so that the users that are not familiar with what they're doing, don't bring down your SAN. Next thing we want to talk about is fiber port section.

I spoke about RSCN suppression earlier. And what this actually does is, RSCN is a notification that when a device comes online or offline to the rest of the switch ports, and that can actually blip your SAN. So for initiators or clients, want to be able to be sure that you suppress those events. For storage, however, you want to make sure that that's enabled.

And also the best practices are to configure the ports to be auto-detect and for speed and type. Unless you're using your older 2 gig card, like in your Xserve G5's or your power Mac, where on those you want to set them statically to 2 gig on this, on the client side, not the switch side, as well as point-to-point for, for a switch type.

But for your, your clients you want to set them on the switch to be the generic, the default, which is the GL-Port. And that'll negotiate to F-Port, which is a fiber port, is you want it to be a set at. Also Device Scan. We used to say, state to not enable this, especially with some of the RAID firmware.

But now with Promise you can enable your device scanning for your clients and storage. Next I want to talk about power and cooling. This is something that you really want to spend time on prior to even turning on your, your, your equipment. You want to make sure that your data center or server room has enough adequate power and cooling.

We've seen a lot of times where, you know, clients haven't planned for this, we just deployed that 250 terabyte SAN, actually the one was pictured earlier, in November. And one wall of their server room was windows. And in November it was fine, it had adequate cooling. And when we visited in March it was starting to get really warm in there. And now they're actually having to buy external, you know, portable cooling units just to be able to provide enough cooling for that SAN. And the problem is that your servers will start auto shutting down when they reach to a certain heat.

But at that point the damage to your drives is already done. So if you want to avoid, you know, unexpected drive crashes and issues with your SAN, you really want to make sure that adequate cooling and power. And a lot of times it would also power, you know, it's a terrible situation when we're in clients and we're plugging in things, and all of a sudden we're blowing a circuit, the server room, or the entire building.

And all it is, all of which has happened. So, these are just a couple of examples of our output of power and thermal. You can reach; you can find a lot more of this stuff on [email protected], as well as the vendor sites have most of that information posted.

So for UPS integration, the first thing to talk about is you want the smaller SANs. Don't just buy one UPS. You want to make sure you have at least 2. Cause you have the redundancy throughout the rest of your SAN, don't leave the single point of failure with the UPS.

Your batteries, 1, 2, 3 years, those lead batteries are going to start to fail, and you really want to make sure you have redundancy in your, in your power. Another thing is when you're purchasing your equipment you want to make sure that you purchase as much of your devices that have dual power redundancy as possible. Xserves now do with the Intel Xserves.

Promise RAIDs, QLogic switches, so all that, you really want to look at that from the beginning. You want to be able to also make sure that you can shut down one of your UPS's and be sure that the other UPS can handle the load. So if you just distribute it equally you don't want to be frying your UPS when one of the UPS's goes down.

And also for every UPS you buy you want to be sure that you buy a management card. I'm going to go into that in a second here. Yeah, buy your management card. With the management cards and UPS's you can do testing to show how much power can be, how, how long the batteries can run when you're on power, when you're off of power, excuse me.

So the it allows you to actually plan accordingly automated shutdown, so that when UPS's are on battery it actually can contact your servers and shut down the servers. And with an upcoming Promise release, that JD's going to speak to you in a few minute about, you can actually now shut down your Promise RAID as well, so that, that's pretty excellent.

And also with the management cards you want them to email you notifications, they do self tests regularly, the batteries dying and so forth. So be sure, be sure to configure those. So shut down procedures, this is something important, not just to know, but also you want to print it out, you want to, you want to paste it on your SAN rack, so that when you're not there and they tell you that, they tell you that the other people that are, that are at your office that they need to shut down the power that day, you don't want somebody just powering things off.

Because that, if you do it in the wrong order it can really cause some major, major issues with your SAN. So, so document this and, and you know, let everybody know about it. So the first thing you want to do is, what I generally do is you want to unmount your, your volumes from your SAN.

This isn't a necessary step, but I recommend it. This way this actually tells the clients now to automount backup when they, when they start up. Because if I'm doing a controlled shutdown I want to make sure that my controlled start up is controlled as well where the clients aren't mounting the SAN at will. I want to make sure I mount them for them. So once you do that you can stop the volume. That's another process that I do. It's not mandatory, because as you start shutting down your client's they're going to shut down just fine.

Once your clients and your servers are shut down, go ahead and shut down your secondary metadata controller. If you're not sure what is the primary and what's the secondary, you can look in Xsan Admin select your volume, it's going to tell you what's actually hosting the volume at that time. So once your secondary metadata controller's shut down, go ahead and shut down your primary metadata controller.

And make sure it's offline. Make sure it's powered off. And at that point you can begin to shut down your storage. And then if you have any additional servers that are not part of your SAN, but they provide services like open directory or DNF, you can shut those down as well.

And then lastly shut down your switches, your Ethernet and your fiber switches. So on the start up procedure the first thing you want to do is bring on your Ethernet switches. Make sure they come online first. And then bring on your QLogic switches. The reason you want your Ethernets to come on first is, not QLogic, but just fiber switches in general. But fiber switches actually talk to each other over Ethernet, a lot of times, so you want to make sure the Ethernet's on first. Once your fiber switches are online, and take note they can take like 10 minutes to come up depending on what brand.

So be sure that they're online. You can go ahead and bring on your other services that weren't Xsan related if you shut those down-- your OD, your DNS, and so forth. And once those are online go ahead and bring on your RAIDs. When you're using Promise and you're using EJ Pair's, you want to make sure that you bring on the J Pair first, so that the J Box first. Give it about 30 seconds to st0art up, then power on your, your, your Promise E Class, which may take about 3 to 5 minutes to come up.

So once all your storage has come up and you can verify that you actually see link likes on your, your fiber switches, go ahead and shut on your, turn on your, your primary metadata controller. And when the primary metadata controller comes up it's going to actually start your Xsan volume for you. So I want to make sure that that volume is started. And I also like to make sure that I can actually mount it on my metadata controller just to verify the data's there.

Because I want to know that if there's anything wrong with my SAN, at this point there's no other clients affected, or there's no other clients turned on trying to write to it or do anything wrong with it. So I want to make sure that that's up and running fine.

So secondly you want to start up your, your secondary metadata controller. And once that comes online it's a good thing to verify that both you're getting a green light in Xsan admin, so both metadata primary and secondary. So next you want to turn on your SAN clients. You don't need to, you don't need to stagger these necessarily, you can bring them all online at the same time. And, as I said, if you unmounted them in the shut down procedures you're going to have to mount them again at the startup procedures.

So you're going to mount the clients and that's and just check to make sure that they can see and write to the storage. So with that I'm going to turn it back over to JD, who's going to talk about getting support as well as some of the new stuff up and coming for Promise RAID's. Thank you. Thank you.

So let's talk a little bit about, about support. And, you know, the first thing I wanted to mention is for those of you who are, you're going to be purchasing Xsan in the near future, it does come with 90 days of support with the, with the software. But obviously what we recommend is that you also purchase the extended support, which is a kind of a yearly renewal.

So for those of you who are already using Xsan today, make sure you do renew your support. And that's for all, and that includes, you know, all the clients as well as the metadata server. So each machine that has Xsan loaded on there should also, should have a support contract tied to it. There's some good support articles up there.

You know, if you're running, if you ever run into an issue where you cannot mount your SAN or you run into a problem, just, you know, we put the KBase article up there. But we always recommend that you call AppleCare and that they help you know, run through some of the trouble shooting steps.

IT's better to do it with someone, a technician on the phone, who might be aware of, you know, an issue, and might be able to help you and assist you. If there is, if, you know, you can, the first thing we would recommend if you can't get to an AppleCare tech, support engineer, would be to run a cvfsck in the read-only mode. And then if there's a problem detected you can then basically do, do the right mode.

And again, that's all in the support article. As Ken mentioned, you know, Xsan is basically not just Apple products, there's, you know, Fibre Channel switches, there's storage that's involved. So from an Apple perspective you want to make sure that you get the right AppleCare support contract for the hardware, for the server, and obviously when you, when you deploy storage, you know, our recommendation is always make sure that you've got, you know, spare parts kits.

Right? So the server, the Xserve, there's a spare parts kit that's available online. So typically what we recommend is one spare parts kit for about, for each 5 servers. And the reason for that is, yes you could have, you know, if you call in for support we've got the 4 hour, you know, onsite, where an engineer, a technician come in and replace it, but you don't often have that 4 hours.

You've got people who need to do work and editing, and you know if you have a spare parts kit available its usually very easy for, you know, an IT person to go in and just swap the part. And then call it in and then, you know, send the defective part back and ship it back.

And the same thing applies with Promise. If you do purchase Promise E Class and J Class, they do have different parts kits, so you want to make sure you have one parts kit for each of the units. And both on the Xserve and on the Promise, those devices do not come with extra drives.

Right? Because you could buy an Xserve with 100 and, you know, 80 gigabyte drive, or a 1 terabyte drive. So you want to make sure that you also buy a few extra, you know, SATA drives and the same thing applies with Promise units. So if you have a bad drive, you have a spare on hand. The other thing you want to look at is, you know, when you, when you get a spare parts kits, especially around storage, you've got a spare controller.

Right? And as you update firmware on those controllers, on your production unit, you want to make sure you update the firmware on those spare parts kits. So that's really important. So you got to think about that and make sure that you plan that. So that when you do run into a problem that spare kit, that spare controller is already up to date with the latest firmware.

OK? So that, that's kind of essential. We talked about the Xsan support. There's also for those of you who deploy much larger, you know, Apple, backend type environments, AppleCare also has, you know, alliance level contracts, which are much higher end that will give you know, UNIX command line support as well as even some patches if need be.

And finally, if you are only going to be deploying one Xsan, right, it's probably best to rely on, you know, a consultant. And we saw quite a few raise their hands in the crowd today. But, you know, just get in touch with you and we'll put you in touch with the right consultants out there to help assist you with the deployment.

Again, I mean this is, this is probably going to be critical to your deployment. So by the time you learn how to deploy Xsan and make sure everything is all set properly, it's probably best to just have someone come in and help deploy for you. And then kind of hand you the keys at the end and make sure that you've got a really solid deployment.

So let's talk about Promise. So Promise has been, you know, our, our vendor of choice in terms of, you know, that is the supportive source device that plugs into Xsan. And obviously there is the upcoming, you know, future release of the Promise firmware that will, that has quite a few new features.

And the first one is UPS support. And as Ken mentioned, that's a, that's pretty big I think for, for all of us. So they also support NTP and SSH. They've done some, some nice performance tuning to both ingest and playback. So, so basically you, it's the same controller, but they've tweaked the software to get some, some more, some more juice out of those controllers and more performance out of them, so that's kind of neat. And you also now have the ability to export a configuration. So if you configure the Promise unit to your liking with the right tune settings, you can now export those, those settings, and them import them into the, you know, to another Promise unit.

So that's very neat. And then finally they have a graphical performance monitoring system. So for those of you who remember Xsan Tuner, that is a product that, you know, with for power PC machines, and there's no Intel version of that. And so you can actually use this to look at some of the performances from your LUNs. And as Jason mentioned, you can also do the same thing on the QLogic, some of the Fibre Channel switches as well. So we'll show you some UI on that as well. You can keep your applauses.

And then serviceability enhancements as well are, are available. So here's the UPS support and the user interface. So you can see that the way it will work is you basically you have IP addresses. And those IP addresses tie into your SNMP card. And that's why it's so important that each UPS device that you have in your environment has that SNMP card. Not only so that the, all the UPS devices can copy each other, right? So what happens is, is as we mentioned, those lead batteries, you know, they have a certain, you know, lifetime.

And if you have multiple UPS's and one of those UPS's has a battery that's pretty much, that's going to die, it needs to alert all the other UPS's that they need to shut down the system, right? Typically, because you've low balanced across multiple UPS's and you probably want to shut down your whole environment and replace, you know, fix your UPS. So the SNMP Cards are essential in any deployment. And that will just plug into, the Promise will just plug into that.

So when you've noticed 2 IP addresses, again for redundancy. The configuration export script, right there, that's what the user interface will look like. So again, very simple to export. And then one thing I wanted to mention is Apple has scripts that are posted on our support, you know, website.

Those are, those are good, you know, example scripts. There are, we definitely support you tuning some of those scripts for your needs, right? So that's really important. If you, if you tweak some of those scripts, it's not because you've tweaked them that we're not going to support you if you call in for escalation.

So I wanted to clarify that. And then here's the graphical performance monitoring. It's very, very neat. And you can see that you can actually monitor LUNs. Right? So you can actually say, OK, what's the throughput coming out of my metadata LUN? What's the throughput coming out of, you know, XYZ data LUN? So, and you can also look at the, some latency as well.

Both on drives, and again on, on LUN that need to be. So very neat. So a big round of applause for those kind of new features. And then, you know, we've been deploying a lot of SANs over the years, and most of them have been in the RAID 5 environment.

And Promise supports RAID 6. So we've, we're beginning a lot of questions on, as far as like, you know, well should we go to, should we switch to RAID 6? And what kind of performance, you know, are we going to get out of it? And what we found is reality is you're not going to get a lot of degraded performance when going from RAID 5 to RAID 6.

So, and we've had quite a few customers now deploy RAID 6. So we're fine with you actually deploying RAID 6 if need be. You know the other thing we wanted to mention is, you know, when you're deploying Promise, and especially in a file server environments, we have the, you guys remember there's an option called read ahead? Right, so there's a read ahead check box in the Promise units. Everyone familiar with the read ahead? OK, we should all be familiar with a read ahead, because that's, that's something you need to be very careful of.

So if you're using, you know, an Xsan backend with let's say Portable Home, right? And you've got a, you've got the Portable Home thinking going on, if you had that read ahead turned on your home, network home directory, your portable home thinking could probably take up to 20 minutes.s tylY By upchecking that box it'll take about 2 minutes.

So you want to make sure you know, and again, some of the scripts that we have on the website might have the read ahead check, you know, enabled. So that's again that's some of those settings that are, that you can dynamically turn on and off and play with and see what fits best to your environment.

And it's the same thing applies to video, where if you have a small Xsan and a small video environment, the read ahead is actually very neat to have turned on. But if you're deploying a very large Xsan, you know, environment with 10 plus machines, client machines, that are editing multiple video tracks at the same time, that read ahead is not going to buy you much, because you've got so much data going into your SAN at the same time that you're not going to get any significant improvement by having the read ahead. You might actually slow it down. So again, that's definitely something that I wanted to mention that, that you should be, you know, play or pay careful attention to.

And then, and then as we mentioned the scripts are, are definitely there, and you can tweak them somewhat. And as far as, you know, another question we get a lot is, you know, what about SAS versus SATA? And what we found is SATA pretty much, SATA drive pretty much works for, for video. And possibly you can get a little bit of improvements when you use SAS drives for like a, you know, in a file server kind of environment, you know, backed up by an Xsan.

So you can potentially get some extra SAS drive, input those as your metadata LUN for optimize tuning for smaller files. And that's it. So as a wrap up, you know, we want to thank you again for using Xsan, for those of you who have it deployed. For those of you who are going to be deploying this in the future we appreciate your business.

As we mentioned, you know, Xsan is a very flexible solution. A very high throughput, and, you know, the biggest thing that I can, you mentioned here, is planning is everything. So when Ken talked about cooling and power, I mean that's really essential for you to do that ahead of deploying your SAN, and making sure you've got the right environmentals in place before you even start racking your equipment.

And we've saved quite a few customers by making sure that we did all those calculations ahead of time, where they were going to put us in a very small room, and you know, obviously with just, the cooling was not going to be adequate. So make sure you plan everything. Make sure you've got good networking.

Work with your networking folks, and don't cut corners. I mean if you don't cut corners you'll have a system that is very robust, that will give you know, that will allow you to run, you know, 24 by 7 and, and, you know, have you as an IT person, you know, focused on other things. And the SAN will just keep on humming. And then again, as we mentioned, backup is very important. So for those of you who are backing up today that's good.

For those of you who aren't, you should. For more information, way to get in touch with us, just [email protected], that's the best way to get in touch with us. And now I'm going to bring in the rest of the team up for, for Q and A. So thank you very much.