My name is JD Mankovsky and I manage Apple's professional services. We're kind of like the geniuses that actually go on site and, and deploy real world enterprise solutions. So what I wanted to do today is kind of follow up on, we had a session last year for those of you who were here to talk about Xsan best practices. And kind of share with you some of the next best practices that we've uncovered over the last 10 months or so. And first I wanted to get a few show of hands on a few questions. So I'd like audience participation.

How many of you guys actually are consultants that deploy Xsan? Okay. How many of you are actually using Xsan or are customers that have an Xsan in place today? Wow. And how many of you guys are looking to deploy Xsan in the near future? Right, that's, that's really good.

So I think we have a lot of information for pretty much all of you guys in the room. So for those of you who have Xsans in place today and for the ones that are looking to deploy Xsan in the near future. And make sure that you deploy that kind of like in a best practice way.

So first I just quickly wanted to review for those of you who are not maybe as familiar with Xsan. Take a few minutes on what is Xsan. Well Xsan is a file system. It's a SAN-based file system. And you know so we'll talk about that shortly. Also we'll talk about you know best practices around how to set up your volumes. Some of the integration and optimization pieces you want to talk about, we'll talk about affinities.

We'll talk about optimizing your fiber channel deployment to make sure that you'll get the best throughput and best performance out of your set up. And then we'll talk about some back up stuff. And then we're going to shift to kind of like a second phase which is you know high availability. For those of you who are in a broadcast world who really need high availability in a SAN deployment, what are some of the solutions that are out there that will give you that.

And then the last topic I wanted to cover is kind of like you know there's a lot of people who do, you know who are IT datacenter people. And they love our storage because it's cheap. But it doesn't offer what some of the tier 1 vendors offer in terms of you know snapshotting, replication, you name it.

and we'll talk about a third party solution that we've been working with that allows you to take our storage and bring it into that, that tier 1 level but yet offer you some of the redundancy capabilities that you're accustom to from other vendors. So that sounds good? Awesome. And we have time for Q and A this year. So I promise you all that it's going to be 58 minutes. And, and that if we go over 58 minutes, we'll buy beers to everyone here at the beer bash on Thursday night.

So, so how does Xsan work? Well, as you guys know, you know you guys are very familiar with direct attach disks. We use that all, you know, everyday. Here with at a SAN file based system, you know the catalog is stored on its own separate you know disk. Right? So you've got a dedicated set of disks for your catalog.

And all that information is stored on usually a RAID-1, you know two disks with you know hot spare. And you know every time you need to access data, to either read and write it to the SAN, you have to basically go through the metadata server and get the information, ask, query the server who will query that metadata LUN and ask where is, do I need to write that data or where do I need to read the data from? And then it sends a response back to, to the client.

And then the client goes and reads those LUNs directly and grabs the data it needs. Okay, so its really important to understand that, you know for people who want to deploy Xsan for you know potentially non video environments, that you are going through several hoops to actually access where the data is and then actually read that data.

So it's not like direct attach where its just you know direct throughput. You know you have to basically ask for permission every time, because there's you know Xsan is a file locking based file system. So it has to make sure that you know every time you access the file, you actually have permission to access into it, it's not someone else, some other individual, some other server who's you know trying to read or write to the file at the same time.

So just quickly to make sure people understand, this is a basic Xsan diagram, okay? So most of you guys will I'm sure have much larger deployments. In this situation we have you know a single PowerMac, you know or MacPro machine and, and we have the metadata switch. We have the two controllers.

And we have you know a metadata LUN which is that LUN1. Which is a RAID-1 with a hot spare. And then we have three what I would call data LUNs, which are usually you know RAID-5 and there's a hot spare on each of those LUNs as well. And we have two controllers, always two controllers for redundancy.

Okay? Because you want to make sure that if, if for some reason the first controller fails that it automatically fails over to the second controller and that your users can still you know work on editing their content and not be down. So that's just to make sure that people understand, this is a you know very basic Xsan you know diagram. But it'll help for further down in the discussion. Everything goes, all the, the metadata switch is really for your metadata traffic.

So again every time the machine that MacPro needs to read or write the data to the LUNs, it needs to ask for permission again. And you want a dedicated switch for that. So a lot of people are like you know do I need a dedicated switch? Can I VLAN my switch? You know I've got the Cisco switch, can I just now VLAN you know the switch I have? Well, yeah you can do it.

But the problem is you got to guarantee that you're going to get to that data in time. And the only way to really guarantee that is to have a dedicated switch for your metadata. So spend the money, you know you're spending $100,000 on a SAN, make sure you're doing it right. And don't, not, don't try to shave you know $3000 for an Ethernet switch.

Right? So you know lets talk about how does the, the SAN actually write, you know how do you write data to your SAN? Well again, imagine a MacPro, a Final Cut machine and the task I want to do is I want to ingest 16 MB of video. And in this situation we're assuming we have a single storage pool with four LUNs.

So you have four LUNs in a single storage pool and you've got this other RAID up there that's just for your, for your metadata, okay? And so again from a theoretical perspective, what we usually say is you're getting about 80 MB per second from each of the LUNs. So considering you have four LUNs, you're going to get a total throughput of 320 MB per second .

Make sense? And so when you're writing the data and considering you've got you know a stride breadth and a block size that equals a MB, right. So usually you do you know stride breadth of 4k and 256 stride breadth or 8k, you know 128 or 16k block and you know 32.

It always equates to a MB due to you know the, the settings on the fiber channel card and kind of the optimized settings for the Xserve RAID. What's happening is it's going to, it's going to write you know the first MB to LUN1, the second MB to LUN2, and then the third MB to three and four.

And then it's going to write, you know the fifth MB back to LUN1, two, three, four, and then it'll go all the way to 16. But its doing that in parallel. So that's how you're getting that optimized throughput and its how you're getting, you know the 300 MB per second is that its writing one, two, three, four at the same time on all four LUNs, and then four, five, six, seven, you know, five, six, seven, eight to again those LUNs. And so it's doing that very, very quickly.

So that's kind of a high level review and now we're going to dig a little bit more deeper into kind of some best practices guide lines. And what I wanted to do is bring up Stle Bjordal whose one of my, one of our engineers at Apple to talk about some of the best practices around, around Xsan. Stle?

Thank you. Good morning.

Let's see if we can go the right way here. JD promised top shelf alcohol by the way at the beer bash if we stick. If we go past 58 minutes, so IX'm thinking we can start off with a joke, maybe a Canadian and a German. Walked into a bar.

But anyways, what we've seen for a lot of our engagement is that power distribution and cooling is something that a lot of people either skip out on or don't prepare carefully for.

So one of the, one of the highlights of, on Xsan engagement is to make sure that you have enough power and you have enough cooling. Because you don't want to be the guy that brings in all these great Apple equipment and then plug everything in and downs the data center.

That, that makes us kind of look bad. And also you can get fired, so let's try not to do that. Make, make sure you know that you have adequate temperature in there and that your power sensor is not right underneath the air vent for the AC. Because you know that can give you a false negative.

So we've just listed a few of the, a few of the, the power consumption and the BTU for some of the most normal things that we deploy there. But keep in mind that a ton AC unit equals about 1300 BTU and that, we see that that's pretty normal.

You want to make sure that in addition to your, your power that you think about how do I make sure that I maintain power in case of the building for some reason should go black. And what we've found is that the APC grade makes some great components that can help us with this.

We have a serial concentrator, a simple signaling cable, and we also have the network monitoring card for APCs equipment. And APCs stuff is qualified with the Xserve RAID and we don't endorse them but it's important for us to understand that if you buy their gear then you know they, they have some stuff that works with what we deploy.

You always have the PowerChute software that they make which is a free download from their website. And what that allows the software to do is the Xserve will get a notification when you're running low on power and will automatically start shutting down when you've reached whatever threshold you're set on the APCs themselves.

Also you want to make sure that you have the right plugs in your datacenter. For example you have the, the APC 3000s they come with the 30 amp twist lock which will not fit into your standard you know 120, 3 pin outlet in the wall. So make sure that you plan for all of these different things when you, when you decide to deploy Xsan or drop a bunch of RAIDs in there.

The basic set up for UPS integration is something like this, where you see it up there. You configure your, your cards in the APCs by, by turning on the network monitor and you deploy the power shoot software. And what that will do is it will, like I said, shut off the servers when, when power is getting low. The RAIDs act a little bit different in that the simple signaling cable will not actually shut down the RAIDs.

What the simple signaling solution does is that it will force the RAIDs to write all the data directly to disk as opposed to storing in the drive cache first. And when power finally goes out then the RAIDs are just going to die because of loss of power. But at least all the data that's been written to the RAIDs are stored on the disk as opposed to being stored in the cache. If it were stored in the cache, then you would lose it if you, if you lost power.

The best way to manage all of this is probably through the web interface. The network management card from APC has a nifty, nifty web tool. And it allows you to configure all the parameters there. The best of setting this up initially is to make sure you have the MAC address of the card. And you, you send an arp command and assign the MAC address and IP address, but you telnet into the card and this is just going to be to just enable the web interface.

One thing that's worth mentioning here is that simple signaling shut down is not enabled by default. So you have to go into the web interface or telnet in and actually enable that option in there. So after you plugged in all the simple signaling cables, you have all the equipment and your testing, your not going to see anything being written straight to disk unless you enable it in the web interface. And a lot of customers are not aware of that. And then they come screaming and because of data loss. So keep that in mind.

We also have to mention that the Xserves, they have a pretty, pretty feature where you can allow it to delay the start up after a power failure based on an interval that you set using the set wait for start up after power failure. And you would set this is increments of 30 seconds.

And there's as you can see in the graph here, there is a preset for this command in ARD. So you can easily highlight all your Xserves and send this command out to all your Xserves at the same time. And the reason why you want to do this is fairly simple.

You want to make sure that all your, all your storage and all your network gear is already up and running before the, the Xserves start back up. And in this case especially because the Xserves in this case would be metadata controllers. You really want to make sure that, that all the other gear that you have is up and running prior to the Xserve starting up.

Let's talk a little bit about how the shut down procedure works for Xsan because if you don't follow the, the steps then there could be a potential loss of data. So you want to make sure first step all the clients and this includes servers as well as, as Xsan clients that you unmount the volumes. And then you use your Xsan admin tool and you stop the volume.

And you turn off the SAN clients. In this case a SAN client is not just a client that runs Mac OS X client. It could be an Xsan server that is an Xsan client or it could be a server computer that's running ADIC's StoreNext file system. Next you want to shut down the secondary MDC and by this we don't mean necessarily the one that you have designated as a secondary MDC. It's simply the one of the MDCs that are not hosting any volumes. So you want to make sure that you verify with Xsan admin which of your MDCs that actually hosting your volume before you shut them down.

Next you shut down the primary MDC and finally you shut down your storage. And you shut down the other servers that may be on your network such as Open Directory, Mail, DNS, etcetera. And finally you shut down all of the other hardware. Such as your fiber channel fabric switches, your Ethernet switches, be it metadata or public.

The start up procedure is basically flipped the other way around. You want to make sure you start up your fiber channel and your metadata network switches, followed by the servers that are not related to Xsan but provide services. Especially DNS and Open Directory so that the users can have access to all those.

Next you want to get your storage up and running and you want to make sure that the, on the Xserve RAID that you got four you know green lights for each controller that's a part of the Xsan environment before you, before you go on. Next you want to start up the primary MDC and then you want to go with the secondary. And then you fire up all your Xsan clients. And then you start the volume and you mount the volumes onto your clients.

Let's talk a little bit about directory integration and ACLs. Because with Xsan 1.4, we introduced ACLs as a part of the, of the Xsan package. And this is a great feature because it allows us an enormous amount of flexibility when it comes to assigning groups and permissions and you can get really granular at decided who's going to have access to what. It makes it really easy for you to allow the users that are in your environment to collaborate on projects, you know be it video editors that need access to the same files or, or you know however you want it to configure it.

For those of you that deployed Xsan prior to version 1.4, you can enable this retroactively. Once you upgrade to 1.4 make sure your volumes are shut down, stopped and no clients have the volumes mounted. Then you can use Xsan admin and enable this on the volume. Excuse me, the ACLs will also help you with the, with the umask problems, when not problem but work around rather that we saw when, when we, you wanted to allow users to share some of the same information. The umask would allow people to get access to files to folders by the SAN by modifying the umask for the user. And this right here negates the need to set that up. I know it's quite messy.

Workgroup manager, you use that to assign to ACLs on to Xsan just as you would assign ACLs on any other file system for OS X server. So you fire up your workgroup manager, you find the Xsan storage and you, you use your OD users and you assign permissions like you would as if Xsan was not involved.

You want to make sure that if you deploy ACLs that you don't use ACLs on your Final Cut pro scratch base. Because of performance, there's a potential of dropped frames if you doing some ingesting and there are ACLs on the scratch space. ACLs will, there is a little bit of additional read write activity there to resolve membership using ACLs. And better safe than sorry.

As far as DNS goes, we have seen that it's better to have as much DNS as you can here. We want to make sure that your forward and reverse records are in place. And you want to make sure that you have that for both the private and the metadata network.

And if you're not sure if DNS is working or not working, Xsan admin will tell you right away because you'll have a general sluggishness when refreshing basically anything in Xsan admin. Refreshing volumes, refreshing to look at the LUNs. So it's very, very important that your DNS infrastructure is set up and it's up and running. Once you have that piece figured out Xsan performance is great. It's smooth sailing.

Also you want to make sure that you know you spanning tree, if that's running that you use Port Fast to make sure that those ports are really up and running as soon as the client network interface has been initialized if you had to reboot a SAN client. If you're not sure what the FSM, the file system is doing with regards to DNS, you can Use LSOF and you can, you can grep for the, for the FSM process.

And the switch is there, -ni4 will basically do a translation of all IP addresses to DNS servers using you know regular IP version 4 and then it will tell you exactly what kind of name resolution that's happening. Make sure your host name is set right on all the clients and all the servers. The best way to do this is using scutil with a switch set HostName then followed by fully qualified domain name.

And verify your records please. Verify them a lot. Make sure that you know nsLookup can resolve or dig or use hostname. But we really want to make sure that your DNS infrastructure is good because it's going to help you in the long run. You can definitely have, if you decide to host your own DNS records, then you can set up DNS server on OD master. Given you know one of the metadata controllers You can host a public zone and a private zone on the same DNS server, that's not a problem. As long as you can just resolve on both of the two, both of the two networks.

Label your network interfaces because if you have the fire wire interface there, you have, if you have Kona cards installed and they throw some network interfaces in there and so its just good practice if you label your interfaces. For example, you know private LAN or metadata LAN and public. Because when you're in there troubleshooting, you want to make sure that you know which network interface you're looking at.

And to make it even less messier, disable the ones that you know you're not going to use. And with that, I'm going to turn it over to JD and he is going to go talk more about fiber channel. Thank you.

Thank you man, good job.

So let's talk a little bit about optimizing your fiber channel switch. And, and first you know I wanted to quickly chat about okay a lot of people have datacenters in one area of the building and they have their editing suites or their SAN or their storage in some other, or clients on some other side of the building. How do you interconnect both? And, and I just wanted to make sure that you guys, I understand that, you know with fiber optics, it makes it obviously a lot easier.

The standard transceivers that you'll buy from, from Apple or the Finisar transceivers go up to 500 meters. So you do have quite a nice length to allow you to interconnect work stations to, to the back end fiber channel switch that might be, that will be probably in your datacenter.

Another thing you can do as well is if you have a large amount of storage and your workstations are definitely on the other side of the building, you could potentially segregate and have a front end set up of fiber channel switches and a backend set up of fiber channel switches and kind of ISL those two sets of switches between the two, the two locations.

And, and again you can use 2GB using the standard transceivers but you'd probably need a lot of those because you want to have a really nice fat pipe between the two sets of switches. Or you can also use some of the newer 10GB transceivers that are available. Those are a little bit more expensive. They're in the $2000 dollar price range.

But they'll give you, you know 10GB for each of those transceivers. And we usually recommend you get four of them right? Always redundancy. So make sure you put two on the, on the front end set of switches and two on the back end and, and pull two pairs of cable between, between the two.

Another we run into is people sometimes have a tendency to again try to cut corners. And they'll try to pull fiber themselves. And if you're not certified and you don't know what you're doing, I mean fiber is not Ethernet. So the days where you know we can pull cat-5 cable is kind of nice, but cat-6 and fiber optics require people who have good skills and good certifications. So make sure you spend you know the $10,000 dollars or whatever to get a guy who can pull fiber and actually test every strand of fiber within that he just deployed.

And at the end, he should give you a dB loss on each of those, those strands that have been tested. Very similar to when you buy a 10 meter or a 25 meter optical cable from you know a vendor, there's a little paper that comes in that bag that tells you the dB loss for each of those cables. Make sure you get that done, because if you start having dropped frames on your SAN it could very well be that there's a bad fiber, there's a bad cable on your network. So, so just best practice is make sure you do that.

But let's talk a little bit about kind of how do I lay out my data on, on my SAN? And in this example this is actually a real world example and this customer has a SanBox 2-64. Okay, so they're not, they haven't deployed the newer, the new switches, but kind of an older switch.

But it's a switch that's been widely deployed everywhere. And so how do you, how do you kind of group things around? Well, the first thing you want to do is you kind of want to make sure that your metadata server and your metadata LUN are on the same blade.

Okay, because you know again the metadata LUN will only talk to the metadata servers. The clients will never write to the metadata LUN, okay? Only the metadata servers. So make sure that even if you deploy you know in a smaller environment and you have a couple of you know 16-port fiber channel switch, make sure that the metadata server and the metadata LUN are on the same switch, okay, because you want the lowest latency possible.

You always want to think about latency in making the distance as short as possible. So you want those to be on the same blade or on the same switch. The next thing you want to do is you're obviously going to build storage pools, right? And we'll talk about best practices around storage pools. But in this situation what we did is we built four storage pools of four LUNs each.

Okay? so what we did is we put for the storage pool number one, we put storage pool number one on a blade, storage pool number two on a second blade, and then storage pool number three on a third blade, which is slot seven and slot eight. So we made sure we kind of grouped those LUNs together because again when you were writing data to specific storage pool, you're writing in a stripe way like we discussed earlier, in writing that data in parallel to all four of those LUNs that are within a storage pool. So make sure you put those again on the same switch. And that again will really optimize your, your throughput. And then what we did is we put all the clients machine around that.

Right? Again for lower latency, when those clients are going to write to those storage pools, you want to make sure they have again the lowest latency possible. So put those, always put your, your storage kind of like the core of the switch and put your clients around you know around it.

And then what we did is we had a tape library right, because back up is really important, right. Obviously all of you back up in this room, I'm sure of it. And in that situation, you know the customer is kind of running out of ports on their, their high end switch, so they bought a more entry level switch.

And really that's perfect for the tape back up. You pull a couple ISL cables, you interconnect those two switches and you put your tape library and kind of like your graphics machines, kind of the lower throughput machines on a kind of of a more entry level switch in your, your cool with that.

On the QLogic side, some of the optimization settings that you want to be aware of. If you have multiple 16-port fiber channel switches or 20-port fiber channel switches if you count the 10GB interconnects, you want to make sure that you lock the domain ID. Anyone remember the SCII days, the old SCII days where you had a little SCII wheel and you put the number, you punch in the number from zero to seven? Okay, well in the, in the QLogic web interface, its something kind of similar.

Where by default every switch that you're going to buy is going to have an ID of one. And it's all fine and dandy. You're probably going, you're going to do your setup, you're going to power the first switch, you're going to power the second switch, you're going to power the third switch and they're going to take, you know they're automatically going to go ID 1, 2, and 3, which is all great. But what happens if there is a power failure? If there's a power failure, they're all going to come up at the same time and they're all going to take an ID of one.

So you want to make sure that the first thing that you do is you go in and lock the ID and lock it down to 1, 2, and 3, and 4 and 5 and you know again if you have five switches, make sure you lock them down. Just best practices because if that doesn't happen, you will, you will get into some interesting scenarios.

The other thing you want to do is you want to make sure that on the, on the client side, you want to enable IO Stream Guard and you want to disable Device Scan. Okay? So again, once all your clients are on the switch, and they're working and they're all lit up and everything is fine, make sure you disable the Device Scan because it scanned the device, so it knows where the device is. So there's no need to scan the device again, right? And one would hope that you're not moving fiber channel ports around and moving machines around all the time. So make sure you disable that. And enable IO Stream Guard.

On the Xserve RAID, the Xserve RAID uses a QLogic controller on the RAID. So you can leave it on auto. It's smart enough to know that the Xserve RAID is actually storage which is considered a target device. And it will automatically disable IO Stream Guard. If you want to be super safe, just go ahead and disable it on the, on the initiators, on the targets I'm sorry which is your storage and your tape libraries.

Let's talk about Xsan you know volume configurations, kind of like best practices around that. What we found is that you know a lot of customers in, in this room I'm sure want to build very large volumes. And or in the early days, what we used to do is build a single storage pool and put all the LUNs in one single storage pool.

And what we uncovered is really what happens is a lot of disk contention that's happening when you do that. And some of you might have run into situations where when you're trying to ingest, you kind of hit this, this limit where you know you're ingesting a lot of data. You can't really ingest more than like you know 800 MB per second or something in that area. If you, if you build one single storage pool with, with a lot of LUNs.

And what you also, what also happens is then you run into situations where you've got your, your SAN up and going, but a year down the road you're like you know what? I want to expand my SAN. And when that happens, you know you really need to keep everything symmetrical.

So if you have a storage pool with 10 LUNs, it probably means that you have to buy another five Xserve RAIDs and build another storage pool with ten LUNs. Again to make sure everything is, is very balanced and very symmetrical. So what we found is kind of the best, the best way to, to kind of work this and make sure that you've got the lowest amount of, of disk contention when you, when you build that, is to build smaller storage pools.

And kind of the, the mix that we found is four LUNs per storage pool, four to six LUNs per storage pool is kind of like best practices. And that'll give you per storage pool about 320 MB per second. But assume you have four storage pools, your getting up to 1200 MB per second of throughput.

And Xsan is very smart about you know writing the data across different storage pool and segmenting your data across the storage pools. So from an editing perspective, the people that are actually then editing the videos, those videos are going to get laid down very nicely across the multiple storage pools.

So you really you're not going to get stuck where you know one single machine is going to use the full 320 MB off that single storage pool. So just keep that in mind. That's kind of like the, the best practices around, around you know creating volumes. But again, you know test it, make sure all that works for, for your situation.

And then, and then really you know in terms of the metadata size, just, just so you know, if you have about 10 million files you need about 10GB of space. So its not a matter of you know the metadata LUN. You're probably never going to exceed the size of your metadata LUN. So our recommendation is just get 500GB drives which are really nice, you know fast drives and use those for your metadata LUN and you're never going to see that 500GB of space on your, on your LUN potentially. Unless you use this for archiving.

And if you're using that for archiving, you could, you know we did have customer who had like 600 million files. And in that situation we did a zero plus one on the metadata side. But again, it was not a video, it was not a video customer. And each volume will require about 512 MB of RAM. So make sure you've got plenty of RAM on your metadata servers.

So again just to illustrate this, kind of best practices for us now a days is four LUNs per storage pool, 16K block size, round robin obviously because you want it to go across the different storage pools that you're, you're putting together, obviously enable ACLs, that just standard now.

You also want to set up the stride breadth to 64 which again, dyou know usually you want it to be a multiple of 1MB and when you do 16K block size times 64, it equals 1MB which is exactly how you want to optimize it for your, for your Xserve RAID. And obviously rotate for the, the multipath method.

And then, and then this is a typical you know SAN deployment. In this situation where you again you've got four storage pools and you've got four LUNs per, per storage pool. And you want to if possible, you want to keep in an even number of LUNs. So you get a little bit of a performance degradation if you do odd numbers.

So try to keep it in an odd, in an even number of LUNs per storage pool. So let's talk a little bit about affinities. Because affinities are very powerful but we just want to make sure that we kind of get a high level overview of how you would want to implement affinities.

So affinities really allow you to steer the data based upon a policy that, that you would set up. So a good example to set up affinity would be, hey I want all my Final Cut projects, because again, those Final Cut projects are kind of small, tiny files. You know in, you know 400K to a MB potentially on average.

Lets say I want to dedicate a LUN to you know to for the Final Cut projects. So you can do that. You can set up an affinity and call it you know projects and have all your users store all their project in a, on, on a dedicated, on a dedicate storage pool that has an affinity attached to it where people store their projects.

Maybe a bad example of using affinities is, is you know trying to do an ingest affinity and an edit affinity and a play out affinity. And, and why is that? Well, simply because let's assume you ingest video on your ingest affinity. And then you have your editor that takes that file and copies it from the ingest affinity over to the edit affinity.

Its not really copying, its not really copying the file. Okay, it's only copying and moving the header of that file. So when that editor starts to edit the, the file, it's actually playing off the disks that you've dedicated for your ingest side of the house. The, the way to move the file is to option copy it and this time its actually really moving all the blocks of the file and moving it over to the, to the edit side of the house. And what we've seen is, is a lot of corporations use you know freelancers to do a lot of work.

And in that situation, try to go train a freelancer to option click copy a file over. Not the easiest thing in the world. So make sure you, you, if you do implement affinities, make sure you, you understand how affinities work and again, very powerful but use them and use them properly.

Let's talk about backing up your config files. Because that's something that as soon as you have your SAN up and running, you should be doing. And you know it's very simple. You just run cvgather -f with your volume name. And that will basically you know archive not only the config file but the .cfg file, but also you know my logs and other information.

And you want to do that on both metadata servers. And once you have that config file, you want to take it, burn it to a CD, store it in a, in a safe place. Because if you ever have a situation where for example, we had this happen so I'll share that with you. Customer plugged in some Windows servers.

Right? And they hadn't, they hadn't, they just plugged it into the fiber channel switch. They had not logged StoreNext FX on those machines. And they opened up the disk manager on Windows. Well the disk management has this little set up assistant saying hey, I see a whole bunch of LUNs. You want me to work with them? And what happens is it goes in and unlabels all your LUNs and your SAN goes offline. Okay? Just a bad situation to be in.

Good news is we had a backup of the, of the config file, so we were able to use CV label and reload all the, the labels on the LUNs and get the SAN you know back up and going in a few hours. But just a situation you don't want to get into. And, and just make sure you back it up.

Its, its simple. It's a small file. Just, just back it up. And, and every time you make a change to your SAN, make sure you, you do that again. So again what we recommend is just maybe set up a cron job right? Or use launch D or you know just to back up that file on a regular basis. The other thing is, let's assume you run into a situation where you do have an issue.

You want to, you know you want to go back to the log that actually has the real error. Because your log is going to get filled with the same error over and over again. So make, so that allows you to go back and actually see, aha! That's really what happened to my SAN. This is how I can fix it.

And then the next step is really backing up your data. Right? You're setting up a RAID-5, you've got hot spares, that's all great. But what happens if something happens to your building, right? You've got to have some way of replicating the data, either disk to disk or at least disk to tape and take that data offsite. So there's a couple of really you know good vendors. Atempo, ARCHIWARE is a European product that we've had a lot of you know good success with.

And the product that's called PresStore and then you obviously have the BackBone which also does you know really well. So those are kind of like the three solutions that have both client and server solutions available on the, on the Mac. So you don't have to rely on your IT person whose running his Tivoli server and or his you know Veritas server and back up you know 10 or 20 terabytes over a gigabit Ethernet using the Tivoli or the Veritas client available on the Mac.

And then lets quickly talk about configuring your tape back up unit, because that's something else you need to be aware of is when you configure your tape back up unit, you need to make sure that you create a zone, a soft zone on your QLogic and assign all the ports of that dedicated you know backup server.

Make sure you have one of the fiber channel ports and all the ports of the tape library in a soft zone. Again our fiber channel driver is very smart and does a lot of really nice things in terms of balancing the data for video. But in the backup situation, you want to make sure you've got a dedicated port for your back up. So make sure you do that.

And then migrating data. So this is kind of interesting. We, we we're getting into now more and more situations where a customer have, they've had a SAN deployed and they now want to upgrade the SAN and in some situations they have not configured it best practices, so you need to back up all your data.

And what we found is a good way to back it up is actually to build a new SAN, kind of like a back up SAN and, and that allows us to quickly mount the two volumes and copy the data from the, the original SAN to the backup SAN, reconfigure the original SAN and copy the data backup. And this example we copied about 10 terabytes of data in about 11 hours off a single client. So very, very fast way to actually back up your data.

Make sure that you have a UPS as well on those arrays that you might be using temporarily to do your backup. You know again, one more thing just to be cautious. You never know. Murphy's Law right? How do you repair your SAN? Again, first thing you should do is call AppleCare, right? All of you have I mean at least in the US and people who are in countries where AppleCare is supported, you should have your Xsan support agreement and you should have Xsan support on and make sure you call AppleCare. That's the first thing that I would recommend. And then you can also follow up with the article number that's up there on how to actually you know run it in read only. For instance run a cvfsck and then after the problem is detected, use the -w option.

Measuring performance on your SAN. So you know we kind of use, we use the Xsan you know tuning tool that we have, which is, which is nice, but its not really real world. So you're really, you know this is great to make sure you don't have like a bad cable or you're, you're getting the throughput you're expecting. But really, you know you want to use the application you're going to be you know using on your SAN.

So make sure that once you, you run the Xsan tuner and everything looks okay, you want to then switch to Final Cut and make sure you're using the application you're, you're deploying on, on the SAN. And also make sure that you're actually, that, that you know that third party application you might want to use against Xsan has been tested against the product. I had a customer wanting to backup their SAN and they started using ChronoSync.

Well, I mean do you think ChronoSync has ever heard or has ever tested with Xsan? Probably not. And you know when they started launching ChronoSync, it said that their backup would be done in about two months.

So again, not a good situation. And we just said well you know just use CVCP, it just, its just fine, CVCP or CP and those are the tools that allow you to actually back up your SAN properly. And they did that and then the number came down to two hours. So just make sure you do that. And then quickly, you know again, best practices. We talk about the DNS, Stle insisted on that. It's really important. So make sure you do that.

Make sure your date and time is synchronized on all the machines. Okay, that's also really important. Make sure you got a good NTP server working and you're not blocking it from your firewall perspective. Backup your config files, backup your logs. Make sure you plug in both Ethernet ports on the Xserve RAID.

There are situations which are probably fixed on the 1.51 firmware, but we do have situations where the RAID still works but when you're in RAID admin, you get that spinning beach ball and you can't manage your RAID anymore. By plugging both Ethernet ports, you're, you'll be, you'll be good to go. Here's a new one.

Make sure your Startup Disk is set to point to the local hard drive. And, and the reason for that is because we, we now have the new Intel, the new fiber channel cards that we had the 4GB fiber channel cards will allow you to boot over fiber channel.

And what's happening is if you don't have it by default set to the local hard drive, it will spend you know a few minutes where the machine is just going to be at a blue screen, not a Windows blue screen. And will just look to try and boot off a fiber channel. So make sure you, you point it to your local hard drive and it's all, it's all going to be good.

And then if you have a spares kit, which every one of you should have a spare, an Xserve RAID spare kit when you buy a RAID. Usually one, you know one spares kit for about five Xserve RAID. Make sure the firmware is up to date. Right? Because you're spares kit it probably collecting dust on a shelf. And it's probably still running 1.3 or 1.5 firmware. So make sure you update the firmware on those.

Don't use the defrag tool unless you have to use it. Okay, defragging is not quite optimized for Quick Time. So make sure you don't use it unless you really you know your volume is full and if your volume is full you probably will have other things to worry about like getting more storage. If you, you know another option is if you're having some drop frames on a specific project, you might want to just duplicate the file. Or duplicate the project and see if the problem goes away.

Make sure you test your metadata fail over. And don't you know don't connect the two MDC servers in the same you know power distribution circuit. Seems basic, but we see that happen all the time. And especially now that we have Xserve RAIDs that have dual power supplies. Right? You guys are all excited about that right? Take advantage of it! Right? Connect each RAID, same thing, in a different PDU. Make sure you balance that across, okay.

And, and again make sure that you understand the bringing up and down process. Biggest thing that I've seen is if the SAN goes down, it'll go down while you're sleeping, right? Make sure the guy that's actually on that night shift knows what the bring up and bring down process is, so the best recommendation is just print that out and stick in on the side of the rack. So he can actually follow it and, and do it properly.

And make sure you have a UPS. You know? You know so many times I heard customer say like, oh yeah, we've got central UPS, everything is good, you know it's all, it's all fine and dandy. But you know what? What happens if the breaker you know breaks? And you don't have a UPS? You can get into a problem. Or we had this customer that said, hey you know we have 1200 amps of power.

You know we're never going to have a problem. And then they call us and they say, oh well, the electrical company is having to go in and switch the transformer and we're going to have to you know the whole facility is going to have to run on UPS for 48 hours and, thank god we had a Symetra set up there, because we would have been out of luck. So make sure you have UPSs. Those, it's also good to have two UPSs than one.

And make sure you configure email notifications on your Xserve and on your RAID and on your SAN so you get notified if there is a problem. Okay? If the RAID starts beeping, don't just shut it, you know turn it off. Look at the logs. Go see what's going on. Okay? So that's kind of like best practices around Xsan.

Now what I wanted to talk about is and we, we talked about this last year. But we hadn't deployed really that solution. And now we've had quite a few under our belt under the past 10 months. And, and I wanted to talk about kind of Xsan and the more of a high availability situation.

You know and again if you look at the requirements from some of our customers, which are broadcasters, they want obviously very close to 100 percent up time and there is no such thing as 100 percent up time. But they want something very close. They don't want any drop frames. They want some good workflows, you know from ingest to error. So they might go with like a building for media solution for, for that kind of, that kind of set up. And they want ability to quickly grow their storage.

And, and again by following the guidelines that I mentioned, where you have four LUNs per storage pool, its really easy to then create a new storage pool, add four more new LUNs and present an increased volume size very quickly with no down time. Okay that's again back to the whole point of smaller storage pools as more storage pools is better.

And, and also they want ability to scale very easily from SD to HD. And we all know that you know with Apple's workflow and with Final Cut, its just, it just works out of the box. So you can go very easily from SD to HD with, with our products and our solutions.

So we worked very closely with, with Viacom Technology. And they have this 1U appliance which basically allows you to mirror, in real time your, your Xserve RAIDs. And it's just been, you know they have this great UI that looks a little bit like you know RAID admin, very simple to use, very simple to configure. And very scalable.

And you know here's a simple example of a video broadcast example where you would have two RAIDs, mirror to two other RAIDs. And you obviously have in this situation you want very high speed. So you would have two engines. And each engine mirrors you know one RAID. So you kind of have a two for one ratio. Two RAIDs for one, one Vicom dual engine. And, and we've just had a great amount of you know results and the product has been very reliable over the past, over the past ten months in multiple locations.

And what that allows you to do is to really kind of have two fiber channel switches you know that are, that are independent. You have one metadata server being hosted off one switch and well you would obviously use the two ports on each of the metadata servers and interconnect those to both of the switches. So you'd kind of have everything is pretty much you know redundant in that situation. And you know this is a diagram I showed last year, but this is NFL network and you might have heard about it in the IT overview session on Monday.

This is the deployment, this was the first deployment of Vicom you know in the world. And they have two sites and it's been, I mean it's saved their life at least you know three times since we deployed this. And again, it was simply because the, they had some power issues. And, and that actually you know really, really worked out for them.

So and, and here's the funny stories. How did we, we you know again, how do you get a customer to sign off on, on a product like that? Well you just start pulling controllers out. So we were literally ingesting video and we pulled a few controllers out, just to show the customer that there was absolutely nothing happening. The SAN just kept on working, kept on behaving properly.

Pretty ballsy, but we got the, we got the sign off. So you've got in this situation, you've got you know seven Vicom units because you've got 14 RAIDs. You've got 7 RAIDs mirrored to seven RAIDs. They're actually expanding that, so we're going next week and expanding that with five more, five more engines.

You've got your backend switch, which is your, your storage. And again in this situation we've got two pairs of two, you know two logic switches which manage your backend storage. And then we have the front end storage which manages all your clients and your metadata servers. So again we've segmented the backend storage from the front end storage, which again allows us to very quickly scale but also from a redundancy perspective, you don't want you know to mix and match your backend with your, with your front end.

So rule of thumb, you know they have a single engine version which we've never deployed. I mean why would you have a single engine? It, it adds one more point of single point of failure. So you want to get the dual engine and the dual engine will give you about 350 MB per second of throughput.

So if you don't need as much throughput, instead of going from a two to one ratio, which is two Xserve RAIDs, you know with one dual Vicom engine, you can go to you know four Xserve RAIDs, you know two mirrored to two with a single engine. And that'll give you approximately you know 140 MB per second. But for play out it might be absolutely fine.

So that's it on the, the Vicom side. The, the third topic I wanted to touch on today is you know using Xserve RAID in a datacenter. And you know some people try to use Xsan in that situation which I don't think it's always a good fit, especially in the datacenter world And, and so we've been working with, with this company called Cloverleaf which allows you to, to take that inexpensive storage that we, that we offer and give you some of the functionality that we hear you know from a lot of the IT folks that, that talk to us. You know so what we hear from those, those customers, you know some of the requirements are they already have this existing storage, right.

But they, you know they're interested in adding more storage obviously. Because we all know how storage is the growth it exponential going you know how people need more and more storage every, every day. So they want, they want if they buy new storage from another vendor, it needs to fit in.

You know they want to be able to slice the storage and assign those, that storage to a server very, very quickly. And they want to to be able to grow that storage also very easily on the fly, right? Which is one of the features that Xsan brings to you. They want to be able to snapshot the data and they want to be able to replicate the data you know to another site. And again they want to quickly do a provision, you know they have a new server that comes up.

It's a new database. And they want to quickly be able to assign you know some 100 gig of storage to that server and maybe that product that does that for you. And you know today might be 100 gig, tomorrow might be 10 terabytes. You can dynamically grow that volume on the fly as you need it.

And they're getting quotes from other vendors. I mean, you know we know all those vendors, EMC NetApp. And they're all in the $400,000 - $500,000 price range probably. Not you know with support and maintenance and you name it. And they, they're looking for something cheaper and that is definitely I mean, that solution is probably a third of the cost of some of the other vendors. So what is it? Well its you know Cloverleaf is basically this agent list appliance. Let's just call it an appliance. That you put in the middle of a store, of your storage.

And that allows you to like I said, do snapshoting, do mirroring, you know mirroring your data, doing site to site replication, dynamically expanding your volume, on the fly. So you can pretty much you know fake it and say hey I've got 64 terabytes assigned to that server. When you really don't have 64 terabytes. You might have a few 100 GB. But its, its set and its ready and when it sees its running out of space, it automatically allocates more storage to it on the fly. So it's very powerful in that situation.

And it works, and they work with pretty much an switch vendors, so you just plug in, plug that in the middle of your, of your system, of your storage, you know farm and it just automatically then detects the storage and its kind of like a pass through and then you can assign it to all your front end storage.

So here's an example of a, you know a diagram of how you would deploy you know a clover leaf solution. You've got this appliance in the middle, which right now is, is the two, the two engines are running on Solaris. They're porting to Leopard and they're porting to the Intel Xserve. So that'll be switching over to our hardware in the future.

But we, we already use Xserves for the management console. Which is all, which is all Java. And you can see that on the back end you've got all your storage, you've got your iSCII storage, you've got your direct attach storage. You can even, lets assume you're out of storage. I mean you're in a situation where, you know this, you're just, you're out of storage. You can take an NFS reshare and proxy that and make that available as additional storage.

So you could take a machine that has a 500 gig hard drive or a terabyte hard drive and reshare that and make that available as storage to some other machine if you, if you completely run out of storage at some point. You can obviously do snapshoting so you can snapshot your exchange database, your Oracle database which is, which is quite powerful and you can, you can obviously put a lot of nice Xserves and you know re, you know share, create AFP SMB servers on the fly.

So again, some of the really cool features is the dynamic LUN expansion where like I said, you know you're building a new infrastructure. You have no idea six months down the road how much storage this department is going to need. So why spend the money ahead of time and why you know buy a 10 terabyte storage solution or 20 terabytes when you're only going to use you know a few hundred gigs for the first six month, year or two years. So Cloverleaf allows you to do that and allows you to allocate only what you need for that specific group but quickly, dynamically expand it.

Another one of my other nice features is, is writable snapshots. So imagine if you're running a database and you know you've got your, your development group that's actually working on the new version of the database. Right? And they have a whole bunch of patches they need to apply.

But they might have you know this database has been running and it's a live database. And the database they're testing their patches on is some old version that's probably about a year old. Right? How many times does that happen? Every day. So what you do with Cloverleaf is you can literally take a snapshot of that working database, mount it read write on your test server, apply all the patches, apply the new updates and actually test it on a snapshot that you just took a few minutes ago versus some old you know version of the database that you've been, that's been deprecated that doesn't have the real content that you're actually using today. So very, very powerful. And imagine in the email virus situation, that's also quite useful to go back to a snapshot five minutes ago before you got hit with some or before one of your containers got corrupted.

Again, it also allows you to do, you know site to site replication, so you know nice, nice for customers who have multiple locations across the country and need to replicate their data. It'll actually snapshot, it'll only replicate the snapshot so it's also very efficient on the, on the network.

So in summary, again just, just go look at them, talk to us if you need any help. And you know looking at a solution like that. If you want more information. No host licensing, no capacity fees. You know don't you love that? The only thing they, they had like three different versions and really more the how many snapshots do you need? So they've got like 1,000 snapshots, 10,000 snapshots, or you know they've got different levels of snapshots, but that's pretty much all you, you get to, you pay for and the solution is sales were between $70,00 and about $100,000.

So again, yeah its not cheap, but again you take a lot, you know all the storage you have on the backend, you divide that by multiple terabytes and you're getting a cost per terabyte of the order of $3 or $4 and, and you get all that, that functionality. So it's very nice.

So as a wrap up and you know I got 24 seconds left, so I'm really happy here. Thank you for using Xsan. You know just don't cut corners, plan your set up, let us help you if you want help to deploy your SAN. How to contact us, [email protected].

That's the best way to get in touch with us. We have project managers that are, that are located around the country, east coast, west coast. Those are there. Email addresses, but again, just send us an email to [email protected] and we'll be glad to help you. Or apple.com/services is our, is our website.

So, and then, obviously I wanted to mention that the you know we talked about Xsan 2.0 and that's going to, going to be available later this fall. And Jason will be talking about that on Friday. So don't miss the, the session around Xsan 2.0 at 10:30 in, in this room.