Well, good afternoon. My name is JD Mankovsky. I run Apple's Enterprise Professional Services group for the U.S. And we're here to talk about advanced backup architectures for Mac OS X. Only two more sessions to go, and then you're free to go home. Isn't that exciting? What I wanted to do first is quickly talk about some backup terminologies to make sure everyone is up to speed on some of those. Talk about some basic building blocks around different backup architectures.

And then what we'll do is we'll talk about some advanced backup architectures and bring on stage three success stories of customers like Dark Horse Comics, Kansas City, Life Insurance, and the Simran Group who have been using some of the third-party solutions that are available today that run on Mac OS X. And actually server solutions that you can use to actually back up your data.

So, you know, some quick backup concepts. Fiber channel, you know, SAN type deployments. When I, you know, when I'm going to talk about SAN deployments in this solution, it's really true clustered file system backup solutions. SAN means a lot of things to a lot of people, including like, LUN masking.

And so I just want to make sure that when we talk about SAN, it's really, you know, a clustered file system like an XSAN type deployment. NAS, Mark Attached Storage, disk-to-disks-to-tape. We'll spend quite a bit of time on that because that's definitely where most of you are headed in terms of your backup and to meet your backup windows. And disaster recovery, replication mirroring, we saw a great example. How many of you were in the large XServe RAID deployment session? A few of you. So actually there's some great third-party products that are available now to do this and snapshots as well.

So first let's talk about some basic backup architectures and review some of those. And then we'll finish by some more advanced deployments around Xsan Backup. The first one I wanted to talk about is just a simple basic backup server. And that's something that most of you probably use today.

I mean, how many in this room-- just curious-- how many of this room actually backup more than 5 terabytes of data? Okay, how about 10? How about 20? Okay, how about 500 gigs of data? Okay. How many people backed up their laptop before they came to the conference? Wow, that's impressive.

That's impressive. So this is a really basic concept. You've got a bunch of Xserve RAIDs. You have your tape backup system. And you need to basically backup the storage onto the tape library. That's just regular server-based backup with the software loaded on the Xserve in this case. And you've got everything attached through the fiber channel switch.

You know, one of the great things about this is it's really simple usually to administrate. You know, it's a one-to-one relationship between the tape device and the server and the storage. And, you know, it's fine. Some of the issues around that is that from a scalability perspective, from an IT management, it's much more complicated to manage. Especially when you have, when you then grow to more than one server and one or two RAIDs, it becomes quickly unmanageable. You know, you have to swap the tapes. You have to do a whole bunch of things. And it doesn't scale very well.

What a lot of you are probably putting together or have today is network-based backups. And that's through using backup client agents. How many people are in this case? Yeah. And usually in this case, we have a backup server in the middle, and that's running your software. And you load an agent on the backup client. In this case, we have an Xserve on the left side, and we have a Windows PC on the right side. And those have different agents running, you know, they could be running database backups.

They could be running file data backup. It doesn't really matter, even mail. The great thing about some of those solutions is, again, they fit very well in a heterogeneous environment, which a lot of our customers are in. You know, they definitely have a mixed environment, and that's great. It does use your LAN most of the time, unless you kind of separate that off.

So it does require some good networking on the back end. I've definitely seen, I've seen customers who back up their data, and they put that on 100BaseT, and when a problem happens and they need to restore, it takes them 24 hours. So just be real careful to make sure you've got a good pipe, good fat pipe on the back end when you need to restore that data. A lot of people do backup. They never try to actually do a restore. That gets really scary.

When time comes. So, you know, again, the restores are a lot slower when you need to restore it. Again, it's much more CPU intensive on the servers. And, again, it uses more of your LAN bandwidth. So people are trying to move off of that if they can. Again, because their backup windows are getting just, you know, shorter and shorter. So what most people are headed into is a, is a disk-to-disk-to-tape type architecture.

And in this case, you see, we've got a similar diagram where we have, you know, the backup server. We've got our backup clients. And what we're doing is we're migrating the user data, which is that XSERV rate at the bottom here. And the workflow is basically that the first stage is you're backing up your data over to a second set of disks. And the reason why most people are actually putting this, together is just the sheer cost of disks.

If you look at XSERV rate, you're talking $2 per gigabyte. So it's becoming really affordable for people to actually deploy those disk-to-disk backups. And it also improves reliability. You know, how many people have had tapes that have failed? Yes. We're all in the same boat. That definitely happens. You know, people forget to rotate tapes or they don't buy enough tapes because they're expensive. And they try to cut corners. And then when you need to restore, you get into some problems. So in this, the first case is you back up the disks.

It gives you, it allows you to back up very quickly because you're talking some, you know, you have some really nice throughputs. In the case of XSERV rate, you're talking, you know, easily, you know, 100 megabytes per controller. You've got two controllers. So you're getting some really nice throughput on that.

And then finally, once you've backed up the disks, you can take all the time in the world, you know, to then do your incrementals or do your full backup over to your tape library. And that allows people to really stay within their five to six, five to six hour window that they usually, you know, give us when we need to, you know, implement a backup strategy.

So again, it's great, you know, it's easy. The central administration is much easier in this case. And again, it's a little bit more complex to set up, but nowadays it's almost a necessity. And we'll talk about some of the different compliance, Sarbanes-Oxley and other issues that you probably deal with on a regular basis nowadays. And again, the reliability and the restore is essential.

I mean, reliability, now you have two copies, you have a copy on disk, your restore is much faster. You can take those tapes, you can rotate them, you take those tapes, you load them into, you send them to like Iron Mountain or some area to store them in a safe location. And that usually, you know, is a lot easier. will help in terms of a catastrophic failure if your building burns down or something. You have a safe location where those tapes are stored into.

Then, you know, when you go into, if you don't want to use any of the networking, you want to go into pure kind of fiber channel tape backup solution. Now you're talking deploying a Xsan architecture. And in this case, this is really a direct Xsan to tape backup solution. That's the first one we'll talk about.

And again, in this case, we're back to your backup window and making sure that we're not stepping on the Xsan throughput that you might be needing if the workstations are like, you know, Final Cut video editing workstations. So again, you need to be careful that you've got the adequate amount of throughput available from your tape on the backend to be able to meet the amount of storage that you need to backup. In this case, we've got eight terabytes. We've got two XServe RAIDs. There are, you know, about four terabytes per RAID. So you have about eight terabytes to backup, and that's a pretty large window.

But the advantage, again, here, is because it's a Xsan environment, each of the servers, the Xsan client one, the two, and the three, have the same data mounted on all three. So again, from a backup and restore capability, it makes it easy to load on one of the clients, the backup server software. You won't have to load any agents on the other machines, and you can pretty much dedicate one machine to do your backup.

So again, this is kind of how the flow is, how the data goes from the disks over to the through-fiber channel, pure-fiber channel, and over to tape. So again, what's great, you've got a dedicated server. You're not using any of the LAN, any of your water network or your LAN network. And again, it gets a little bit more complex. And again, you're really dependent on the throughput of your tape backup mechanism.

And you need to look at LTO2, LTO3, AIT, and really calculate how much throughput you can get. And you can scale that. I mean, if you've been to our data center, how many people here went to the data center downstairs? Okay, pretty much all of you. I mean, you saw Exabyte, they've got different types of hardware. And if you look at a Magnum, it can have up to eight LTO2 tape mechanisms in there.

So you could pretty much sustain almost a terabyte an hour with one of those devices. And, you know, if you haven't looked at those, you know, people are pretty blown away. I mean, you're talking $25,000 for something that has like 140 cartridges and that can back up up to 30 terabytes. So $25,000 for 30 terabytes is not a bad deal for tape. And so you should definitely, you know, check it out.

So most people will deploy Xsan and disk-to-disk, and that's really the combination that I think is the most appealing in any type of deployment. And you've got your user data on this volume. Again, we've got 8 terabytes here. You've got your disk-to-disk backup, and that allows you to backup your incremental. So you could have many days, depending on how much data rotates on a daily basis. In this case, we've got 8 terabytes of disk-to-disk.

So if you have only 10% of your data changing on a regular basis, you could pretty much have almost 8 or 10 days of disk-to-disk information stored on that RAID. And then once you've done your backup, you then backup the tape. And you can take however much time you need, do your incrementals every day, and then do your fulls over the weekend.

So that's kind of the flow we're looking at in this case. So again, it's really easy, much easier to manage, and again, it's much more reliable. But again, it gets more complex. It adds another level of complexity. But it's just great for off-site, long-term compliance and just safekeeping of your data.

So I wanted to quickly talk about some of the advanced backup architectures. And the first thing, when we come in as the consulting team, and really that's probably one of the first questions that we ask a customer if they deploy a Xsan, is like, what is your backup strategy? Because you can have RAID 5 drives, you could have hot spares.

You know what, if you don't have a tape backup or some solution like that, or a disk-to-disk, at least a disk-to-disk type solution, There is a huge risk that something is going to happen and your data is going to go away. And we're very adamant to make sure that the customer has a backup-type architecture. If they don't, on our statement of work, we'll put, the customer has denied any type of backup solution. And if they call us a few months later and the data is gone, we can't guarantee that. They have to be in charge of that.

So, you know, when we talk about disaster recovery plans with a customer, those are definitely questions that we ask. You know, what is your business continuity plan? Or what's your, you know, business process continuity plan? You know, BCP or BPCP. And then, of course, you know, we all hear about Sarbanes-Oxley. How many people here in this room have to deal with Sarbanes-Oxley? Not as many as I would expect, and maybe some of you don't know that you will have to deal with Sarbanes-Oxley very shortly.

There's about 40 laws around the Sarbanes-Oxley compliancy today. There's another 200 that are in the works. So you should definitely start looking at those because it's probably going to affect most of you in the room if you have to deal with enterprise-type data that is required to be, where you're running an enterprise and financial-type information, you will have to comply with those.

HIPAA is another one. The European Data Protection Act is also something that the European guys have to deal with. And really it's all about policies and process management and really understanding the process of your company's process and the data, the flow of the data throughout. throughout the corporation.

Interesting enough, if you look at Gartner report, the latest number for storage management solutions which include that, the number was 5.6 billion in 2004, and it's going to be in the 6.3 billion in 2006. It's a huge number in terms of just sheer storage management software to be able to be in compliance with some of those regulatory pressures.

So, you know, quickly, you know, bare-metal restore. If you haven't heard what a bare-metal restore is, it's really you reload the OS, and then from there on, you have a way to restore the data that was on that server. And just wanted to quickly touch on that and make sure that people were up to par on what a bare-metal restore is. Pre-OS recovery, basically. Replication and mirroring. So, you know, we definitely talk a lot to customers about replication. You know, we hear customers want off-site disaster recovery-type solutions.

And really, you know, it really depends on, you know, what kind of transfer do you want? What kind of pipe do you have between the two locations? Do you have a huge, you know, fiber channel, dark fiber-type deployment between the two sites, which would allow you to pretty much do, you know, real-time mirroring? Or would you have, are you going to use more of a synchronous-type method where you'll back up your data every other, you know, every few hours or every day or every 12 hours? So it's going to really depend on your bandwidth that's available between the different sites that require backup and your disaster recovery sites.

And then, of course, you know, there's file-based replications. There's volume-based replications. You know, the snapshotting portion of that, of course. And if you were at the other session, you saw that we, you know, there's some new technologies that are coming around mirroring and around snapshotting that are actually in the switch.

In the fiber channel switch. And that's really exciting because it moves you away from having software-based replication, and it brings it down to the layer three, right into the fiber channel switch. And that was discussed in the other session around a Miranti. Miranti is a new company that has this type of replication and snapshotting across WANs over fiber channel and IP and allows you to accomplish those kinds of solutions.

So let's talk about example of deployments. There's a few here. This is an example of a deployment that we would do around either network home directories or QTSS, QuickTime streaming, web serving, and file serving. So in this case, what you have is this is a typical Xsan deployment. And you've got your DNS, DHCP. You've got your LDAP, or your Active Directory, for authentication. And those are the two top servers. We always have two, again, for redundancy.

Then you have your metadata servers. And those are running Xsan. And this is what manages the file locking on the metadata servers. And you've got a failover server. And you notice on the failover metadata server, I've got two red disks. And what we did here is, because the XServe has some pretty nice 400 gig drives, you've got about 800 gigabytes available for disk-to-disk.

And so what we're doing is we're doing disk-to-disk on that second XServe. And we're going to do the math in terms of the amount of space we have on those RAIDs at the bottom here. You're probably at six or seven terabytes. So if you think that-- in this case, this customer, they had only about 10% of their data that changes on a daily basis. So that gives us the opportunity to do disk-to-disk during the day, and then at night, do the incremental over to the tape backup system. So typical type Xsan deployment with disk-to-disk. So that's the first thing.

This is another example. And in this case, this is a CNN. This is actually CNN in Washington, D.C. It took us about two weeks to do this deployment. But you've got about 18 Xserves. You've got a whole bunch of Power Macs. And you've got a full Xsan deployment with like two Qlogic.

We didn't put both of those switches on the slide here. In this case, the customer really wanted, they didn't want any tape. So what we did is we built two 12-terabyte RAID deployments, Xserve RAID deployments, and we're replicating the data on a daily basis between the two Xsan volumes over fiber channel.

So just two examples of different ways and different architectures to backup your data. So what I wanted to do now is kind of switch over to some of the customer success stories that we've had around some of the third-party solutions that you might have seen in the data center around, you know, a tempo, you know, backbone, and retrospect. And the first person that I wanted to invite is Chris Irvine from Dark Horse Comics. Chris? We've missed the opportunity to do some high def with QuickTime 7. We've been having a lot of fun with that.

That shows you a little bit of the titles that we're using with Dark Horse Comics. Mike Richardson actually formed Dark Horse in 1986. He had been running comic book shops and decided he could compete with the big players, DC and Marvel, and started his own publication company. We've been able to attract a lot of top talent for publications and then we've also done a lot of licensed comic book properties. We're the biggest licensed publisher of comic materials. We do like Star Wars, Buffy, and recently The Incredibles, so we get to do a lot of those fun projects. We also have a company we started in '92 that our department supports at Star Horse Entertainment.

And so we are, that company's producing movies Let's see, what do we have? We started out with the mask with Jim Carrey, which was a big hit. And then also recently we worked on Hellboy, which has worked out really well for us. And then the comic book shops that Mike started with are still doing quite well. They've expanded into pop culture stuff. So we have actually retail stores in Oregon and California. And we actually have one right in town across the street, so check that out.

As far as how we're doing IT at Dark Horse, it's fairly typical structure here. I'm just going to skip over some of this so we can save some time. We have about 80% of our servers are running OS X, and then we have a mix of Windows, Solaris, OpenBSD, so we have some heterogeneous server requirements.

Lucky for us, most of our desktops are running OS X, so about 95% of our users have OS X workstations. And then one of the other things we're doing is we're using remote home directories. As far as backup goes, that's great for us because we can centralize all the data on the server and gives us a central point of backup for all of our, at least for our desktop machines.

And then one of the other things we love is Netboot and things like the Disk Image Framework. So getting those machines back up sort of the bare metal situation at the desktops is pretty easy because it just takes us a few minutes to get Netbooted and our image restored and users can be back online. As long as we have redundant storage behind those home directories and those critical servers, uptime is actually quite good on that.

As far as the data that we're backing up on a regular basis, we've got about one terabyte of data that we backup on a regular basis. That would be our key production and media files. We've got about four terabytes of archives that we've run over the last, you know, digital publications we've done over the last 13 years. And then we've got about another terabyte for all the other supporting data for just operating a business, so your marketing operations and accounting, all that sort of thing.

I already mentioned that our user data is centralized on the server, so we don't have any user data that we have to worry about backing up on the end-user workstations. And we do have a little bit to deal with on laptops, but we're looking forward to moving those to Tiger with portable home directories.

As far as the data that moves around inside of our company, our core business is publishing. Data comes into the pipeline, that's often scanned or FTP. During the production process, there's hundreds of phases probably, but those might be like color corrections and proofing. And then the final product comes out, usually as a PDF, often that's FTPed straight to the printer where it's ready to be printed. And all that data for us, we've made a decision that we want to retain nightly backups of that production pipeline for at least 12 weeks. Also important to us and anybody else who does publications is archiving.

You know, a good chunk of our revenue comes from reprinting material or licensing it to be printed overseas. So whenever a publication is shipped, the data is all collected and archived. Over the years, you know, way back, people started just throwing that stuff onto a DAT or a DLT or whatever.

It's grown into things like DVD carousels and things like that more recently. And then even though we've got a library of information that production users are backing up archiving, the IT department also takes on the responsibility of running extra media and platform diverse copies of those publications that we can vault and, you know, safeguard

So towards the end of last year, we sort of recognized a lot of deficiencies in the way we were doing backups.

One of the big things for us was just the risk of downtime. We were looking at our platform and how long it would take us to restore our critical data in some sort of worst-case scenario. Heaven forbid an Xserve RAID ever ends up with corrupted data on it, but you have to look at situations of how long would it take me to get that back online, and that's actually probably our strongest argument to pitch the project of making changes. It was just sort of, can we live with being down for X number of hours sort of thing.

Also, we were faced with media that had really hit the end of the line. We were using Mammoth 2, and there is no Mammoth 3, so we were looking at changing directions there. Just like our restores were taking a long time, our backups also were taking a long time. We had to split those across multiple days, multiple weekends in order to fit into our backup window.

We were really facing a major administrative overhead. In order to accomplish the requirements that we had, we had set up like 20 different tape pools, and we were pretty much manually tracking hundreds of pieces of media. So we wanted to do that in a more efficient manner. And then those archives that we looked at weren't very uniform and had a lot of problems, so we wanted to fix those problems too. We started out by comparing our media options that were out there.

We were using Mammoth 2. It was costing us about a dollar and a half per gigabyte for tape media. And there was obviously a speed problem there. So we started out by just comparing the platforms that were available. These numbers were from late last year. The big players, Quantum, LTO, and Sony, all have very competitive media in the market right now.

Those are pennies per gigabyte, which is great. One thing that stood out to us there was the throughput of Ultrium 2 was a notch better. And so that was attractive to us. And I should mention that all these vendors have started shipping their next generation equipment that's going to hold twice as much, run twice as fast. As soon as the volume of those shipping units starts to go up, we'll see the cost per gigabyte of those becoming just as competitive as this.

And then we also wanted to keep disk in mind. Apple XRV RAID has become very competitive in the storage market. I don't work for Apple, so my number is a lot more competitive. But we looked at the actual usable redundant space on a RAID and saw that around $3 a gigabyte, which is actually quite good. And we're easily seeing benchmarks 63 or higher depending on your attached host, just even from one side of an XRV RAID. Ultimately, we went with the Ultrium 2. And we did that with Overland Storage Power Loader.

When we looked at our archive system, we really wanted to solve this problem with the DATs and DLTs and DVDs and everything coming from different directions. And we looked at HSM products and nearline products and really for the size of our organization, cost performance wasn't adding up. So we went back and looked at disk again and that's what we ended up going with. We've decided to just do all of our archives online.

We're going to restore all of our old tapes and quit messing with that. And so when we run our archives, we'll just move data from the production systems onto these archive storage volumes, lock them, and then run off archival copies that we can protect in case of a disaster.

Looking at the software, we wanted to make sure we were using the best software as we moved forward. The big priorities for us, one of those was scalability. We obviously wanted software that could handle multiple jobs, multiple administrators, multiple tape drives. We sort of saw those as things that we really had to have. Price is a factor. I don't look at any project really without making sure it's competitive in the market.

A big thing for us was Macintosh Enterprise support because we're so heavily running OS X. We wanted a company that was going to be 100% both behind Macintosh clients and Macintosh servers for the backup end. And we wanted a company that would be responsive to us when we say, "Well, we've got this problem or whatever." And, yeah, we're running OS X and they should respond that that's no problem too. It's a great Unix platform.

And performance should be good. We don't want to consume any more resources than we need to on our, especially our backup clients when we're running a backup or restore. When we compared all these and a few other issues as well, we ended up going with the NetVault software from Backbone.

And it met all these quite well. As far as the way we've deployed, we've The NetVault software, we're using a single backup server and sort of that network backups server configuration like JD talked about, where we have the server sitting between the tape mechanism and the LAN. At the moment, that machine's actually just a dual 800, which is enough to... Whoops.

Thank you for joining us. We're going to talk about our back-up architecture, which is the core of our system. We're going to start with a full backup. The full backup is a backup to the Mac OS Xserver. We're going to run it every six months for auditing and then our full and incremental backups are the core. The fulls are running every two weeks on the weekends and then incremental backups are running every three weeks on the weekends. The full backups are running every two weeks on the weekends and then incremental backups are running every three weeks on the weekends.

Performance we've been really pleased with. This is an example of one of our clients. This is a 1.4 and we're backing up about a terabyte during the night. It's pretty easily saturating the Ultram drive at around 36 megabytes a second. And we were pretty happy with the load average sitting at about .6 through the entire operation. And at that sort of throughput, we felt like that was a pretty good fit.

So just to wrap up really quick, the problems we looked at, the downtime, the media that we hit end of life, all those issues. The changing the media platform, moving to Ultrium 2 really helped us to break through the barriers with the throughput problems and helped us to have the ability to cut down our backup and restore times.

Moving to Xserve RAIDs gave us the option to sort of re-approach how we do archives by just storing data online. And then using NetVault, we can sort of manage all this stuff and manage how our media is handled and the jobs as they run. And we're really happy with how this is going. So far, our estimation is we're probably going to cut the administration time that we were spending in managing our backups about in half.

So just to close, we're happy to be here. And I've been able to actually offer you guys a 20% discount if you're interested. Right across the street on the second floor of the Metreon, if you just show your badge at our Things from Another World store, we'll hook you up. Thanks.

So I'd like to invite John Welsh from Kansas City Life Insurance. John? Okay, so my name is John Welch. I work for Kansas City Life Insurance. I'm probably best described as, well, my semi-official title is Unix slash Open Systems Administrator. That really translates out to I don't work on mainframes and I do stuff besides Windows.

We're a publicly traded financial services company based, obviously, Kansas City, Missouri, not Kansas. We've been in business for 110 years. We've been run by the same family for 110 years. 4.6 billion in assets, you can read that. We've got 600 employees, about 800 computers. 20 of those are Macs, the rest are Windows.

This is probably pretty typical of a financial services group. Insurance is a Windows world still. I'm trying to get that right. I'm trying, but luckily one of those Macs sits in a corner office run by one of the brothers who runs the company, so that's kind of handy. In our case, data preservation is not only a business decision, but it's mandated by law.

HIPAA, GLB, SOX. I think probably if there's a government regulation that affects data preservation or data handling, we get it three ways. We live with auditors. I think we should just give them offices. It's not quite as bad as the guys from Japan, but honestly, I would not want to be the guy. I would want to be the guy who would put a company that's been in business across three centuries out of business because I messed up a backup.

We traditionally, like a lot of people, only backed up our servers. This was pretty much how it worked. The Windows servers were backed up automatically. Everybody else, whether they ran Macs, even my Xserve, or if they ran Windows workstations, you just dragged that stuff up to the network. The Windows people had their U drive or whatever drive it was. The Macs had, we have a share called literally Mac Share. More recently, they have their home directories also mounted, but you got to do it on your own. Yeah.

The problem is backup really isn't just for servers anymore. I mean, you know, clients at this point, we call them anything not in a server room. This includes everything from the security desk up to the president of the company. Now, we use a Windows-based product to backup our Windows servers to AIT2 jukeboxes. And again, as I said, the clients are all backed up manually. Fairly common.

There's problems with this. I mean, there's one thing you want to say when a crash happens and someone says, "Well, I hadn't dragged a bunch of stuff over to the network." You want to say, "Well, that'll learn you." However, when the person who crashed is in a corner office, up on a high floor with a real pretty view, and they have certain last names, you're not going to do this. That's not an option. And we had a couple of high-profile crashes that showed there's a weakness in this philosophy.

And when it came to the Macs, although our backup system is excellent, it's really great for Windows. The Mac client has some issues. For one thing, security. It required an admin-level password and an unencrypted text file and slash Etsy. No. As soon as I realized that, the answer was no. I did some speed tests anyways, but it was still going to be no.

The client pulls a server. In a desktop situation, great. We have laptops. You know, I don't want my client sitting out there in an airport on an unsecured wireless network going, can you back me up? Can you back me up? Because eventually someone's going to go, sure. I can do that. Thanks. And honestly, speed, we couldn't even saturate 802.11b connection, much less 100 meg Ethernet, 802.11g, anything with any kind of speed. It just was not going to happen with this.

Backup window is an issue. Even just our Windows servers, our backup window, we're shaving that real close. Restores, we dread them. The only thing that saves us is 99% of our people are in the office from exactly 8 in the morning until exactly 4:15 at night. We can predict that really well. We have bells. Again, 110-year-old company.

If we start adding in these really slow Macs because the Windows client for this product didn't speed along nicely, it just blows it up. I mean, we tried it. There was just no way it was going to happen. The Windows backup guy said, "No way," once he looked at the speed.

And we tried everything. We just could not get that speed up. We also realized that we wanted to start backing up PC laptops because we didn't want to just do a redirecting of my documents to the network because then if they're off the network for a while, their virus definitions don't get updated on time. These are Windows boxes. You know, we didn't want to backup viruses to our backup server. That could be, I believe the technical term would be bad.

So we had to go with a separate system with its own server and its own tape drives. And there was also a philosophical issue. I kind of get off the bus with the people who say client data doesn't count. It does because all our data is created by clients. And it's not always backed up, you know, reliably or sometimes they're offline for a while. And I don't really like punishing people because they're not running on a server, they're running on a laptop. And that's kind of what I view that whole server-only backup thing as.

So this is going to be the new toy in town. It's got to have solid Mac and Windows support, obviously. It's got to have good support for laptop and mobile backups because along with the Macs, its primary reason for being here is to backup our laptops. It's got to be reasonably secure. Now, reasonably secure means not plain text files. It doesn't have to be kerberized with six layers of encryption. It just has to be hard for the script kiddies to get into.

It has to be able to run well on an XServe without needing a SCSI card that we didn't have the slot for anyways. It's got to be pretty much fire and forget. I do not like writing servers. You know, the perfect server is the one I forget I have.

It's got to be pretty much zero effort for restore and very little effort for, or almost zero effort for backup and very little effort for restore. By very little effort, I mean they email me and call me and say I need this file restored. That's really all I want the user to have to do. they should not even know it's happening.

The solution, as it turned out, was Retrospect 6 from EMC Dance. Understand we have modest needs. At some day, again, we only have 600 people, some day we might have a whole hundred clients with just their home directories backed up to a small tape library, and not everything in that. We chose the Exabyte VXA2 1U FireWire 800 library. It's brilliant. Suitable speed and capacities for our needs. It was easily racked. It's actually smaller than the XServe by a good bit.

Fastest client connection we have on our network is 100 meg Ethernet. So FireWire 800 speed, tape speeds, that's never going to be the bottleneck. It's going to be the cable coming out of the back of the client. It's a 10-tape library. We do an incremental every day. We don't do fulls on the weekend. Again, most of the laptops are gone on the weekend. We really, when I say we never have people working on the weekend, I'm very serious about that. You know, if you're not underground with the mainframe, you're not working on the weekend.

So that's what it looks like now. We have a dual processor Xserve G4. We've got the FireWire 800 stuff to the tape drive. And it just talks to--it backs up a few things on the Xserve that I care about. It backs up home directories on Windows laptops. It backs up home directories on the Macs. It backs up some stuff in slash library. But I leave systems alone. I can always reimage those. I don't need to worry about the system files or anything like that.

The tape library itself is a 10-tape single-drive library for under $3,000. It's small. It's dead simple to set up. I mean, like, plug it in, turn it on, you're 90% done. It's got a couple of command line utilities for management, which lets me update firmware on the library and on the drive separately.

The front console, I think I read the instructions once just to see if it was different than what I'd found. It really wasn't, so it wasn't really needed. It's got a barcode reader. Barcodes are good always. And it's really been a fantastic bit of hardware for us. I mean, if any of the Exabyte guys are here, I love you all. It's been brilliant. We're actually looking at some of their other stuff now because this little piece of hardware has been so good.

Those speeds at 280 megaminute versus 20 megaminute, that was during the middle of the day, and I happened to be using remote desktop to make sure that the designer who was getting backed up, she was running Photoshop on a dual 800-ish G4 without a whole lot of RAM. So we're not running really hot hardware. She's on the same 100 megabit connection. I was getting 280 megaminute versus 20 megaminute. I can live with that speed increase.

The selectors allow for really fine-grained selection, so I'm not backing up video files. I'm not backing up audio files, and that means on either platform, whether it's WMA, MPEG-4, QuickTime, MP3, AAC, whatever, I'm not backing it up. Once I got the president's brother to buy in on that, that his music library wouldn't be backed up, that kind of gave me really good political capital for everybody else, because I could say, he's fine with it, what's your excuse? The only dances mailing list has been all the support I've needed for almost 10 years of using the product.

I definitely tend to plan a little bit more, but I've never really had a problem that the mailing list couldn't solve. The other nice thing with the fine-grained selection, although, is that although people in corner offices are willing to play by the rules, sometimes they want their rules to be a little different.

So I was able to create different selectors for the corner offices than for everybody else. The only issues was the firmware on the tape unit and the Windows client version issues, and both of those were solved in about a day. Literally, I think it was about three hours. You know, once I updated the firmware on the tape drive, updated the firmware on the library, made sure my Windows client numbers were right, it's just been working.

It's excellent laptop support. The server pulls the clients. If the client's not on the network, it just doesn't get backed up. And that's really been great when I'm explaining how this works to people because that's what they ask. What happens if I'm not on the network? You don't get backed up. When do I get backed up? As soon as you're back on the network, you get backed up. So I don't have to do anything? No, you don't. Oh, cool.

The backups are transparent. They don't know. I did tell them that when it's first starting the incrementals, you're going to hear a lot of hard drive activity. That's fine. They can live with that. Once it gets done, the incremental data changes, again, insurance company, not a lot. Even the designers don't change stuff that much. We reuse stuff over and over and over again.

The Mac servers client support fits our needs perfectly. We don't have a lot of, we don't have any Linux clients at the moment. We might, but it's all Macs and PCs, and Retrospect backs those up really well. And we're a small company. This is a small, part of a small company.

We don't need the ultimate backup tool. We don't need something that will back up, you know, 16 petabytes overnight. We just need small, and Retrospect is handling this, you know, really perfectly. It does the job. It's been running six months. I don't mess with it much. I check on it occasionally, a little remote desktop. Oh, okay, I don't need tapes yet. Cool. That's pretty much what it is. Oh, still don't need tapes. Fine. Oh, need tapes.

We've got about 25 clients at the moment. We're adding about four a month. I am adding them small and sanely because that way if something goes wrong, I know right away, not after adding 50 clients at once and then wondering why things suddenly went straight to heck. It's also been a PR bonanza for us.

There is nothing like having your party planner, the vice chairman of the board, the VP of operations, the VP of IS, you know, all those kind of people with those titles suddenly really happy with your department. That's great political capital because then you can go in and say, you know, we could use a budget increase. We're really happy with you. We may not be that happy with you, but we're really happy with you. It's also the most boring server I have.

Boring in the sense of I don't have to mess with it. It lets me go home on time. This makes two people very happy, my son, my girlfriend. Well, it makes my bosses happy too, but I care less about them. So that's, we're not as cool as Dark Horse. Thank you.

Thank you. And now I'd like to invite Mark Miranda on stage with the Cimarron Group. Mark. Thank you. Good afternoon. My name is Mark Miranda, and I am the IT Director for a company called the Cimarron Group in Hollywood, California. Cimarron Group was established in 1979. We are one of the largest advertising agencies for the motion picture industry.

We run the gambit. We do movie trailers, movie TV spots, one-sheets, movie posters that you see in the theaters, DVD box design, and even now DVD menu animations when you stick a DVD in your DVD player. We also do celebrity and feature film websites. As you can see, we have quite a few rather important clients up there, and we like to keep them happy.

So you probably haven't heard of the Cimarron Group, but I know you've seen some of our work. On the top row, we have all our theatrical audiovisual campaigns that we've worked on. Troy, Collateral, War of the Worlds, which is coming this summer, and the critically acclaimed Million Dollar Baby.

On the bottom row, we do all of the Star Wars DVD packaging, which as you probably have seen over the years, they tend to come out once or twice, three, four, five, six times. And now that we have Episode 3 coming out, we know there's more coming there. We were also just recently awarded, we are the official advertising agency for Lotus Cars, so that's exciting for us. And we also do political cartoons, or not cartoons, commercials.

Arnold Schwarzenegger was one of our celebrity clients and he's also become one of our political advertising clients, so that's also fun to work with. And we do all the print and video advertising for Universal Studios Hollywood, so hopefully you've seen them. You like them. Our architecture. One thing I just want to say, we do all kinds of things in terms of print and video. For the focuses of this, I want to focus on our theatrical audio-visual side because that presents the biggest challenge in terms of backup and restore.

So we are a 24/7 facility, which means we don't have a lot of maintenance window, don't have a lot of downtime. It makes things difficult for us. We exclusively use Apple Final Cut Pro for our edit bays, which is really cool. Back right around the first of the year, actually the day after Christmas, I remember that very well because we were all stuck at work. We converted everything over from Final Cut 3 OS 9 to Final Cut 4 5 OS 10 with Xsan running as our backup. We had to do a lot of work to get our file system, so that presented a few challenges for us.

Anyway, we have 25 compressed DV edit bays for offline rough cut editing. And then we have two uncompressed finishing bays that take the rough cuts and make them ready for broadcast. And all of our stations share media. And this is really key, which is why we needed Xsan. It's not uncommon at all for an editor during the day to be working on multiple campaigns or even within a single campaign that you're working on.

There may be four separate editors during the day creating different creative TV spots because you never know what the studio is going to like. And then on a more basic level, we share our media because we need access to a centralized music and sound effects library. So that way we just maintain one master library. It makes work easier for us.

So here's a simple diagram of basically how everything's hooked up. We use a SAN strategy called SAN to the Desktop. So all 27 of our G5 edit bays have a direct fiber optic connection to a fiber channel switch, as do our metadata controllers that run the SAN, and then of course the backup server.

One thing I want to point out on the backup server, you'll see there's actually two connections to the fiber channel switch there, and we use multi-pathing for that. Multi-pathing basically allows you to achieve, in theory, up to twice the throughput with Xsan, as well as the benefit of if you lose one of those fiber optic connections for some reason, your backup will keep running. And then of course there's our LTO3 tape library attached to that, which hopefully will soon be Fibre Channel itself.

So there it is. That's our Xsan system, seven Xserve RAIDs, switch at the top, and two controllers, modest little system. We have three volumes set up on those drives, 16 terabyte only because that's the maximum for Panther, which we can now change. But that's our offline video volume. We have an online video volume that's four terabytes, and another music and sound effects library that's four terabytes as well.

Now you might be thinking, that's an awful lot of RAIDs, and that's a lot of storage, but it's not that much storage. Well, all of these RAIDs are configured with only six hot drives on each side, so there is a hot spare on each drive just to minimize our downtime if an array becomes degraded.

So our challenge. Our challenge was we needed to back up 24 terabytes of data. The RAID 5 protection and having the spare drive, still not enough. You never know what's going to happen. And we wanted to implement it in a very specific way. We needed to be able to do full backups on the weekends when not all the editors were there, but some of them inevitably are there. And then incrementals on the weeknights, again, when not all of the editors are there, but some of the editors are there.

And then it's got to have minimal impact on their workflow. If they start dropping frames while editing video, that's not going to reflect good on us. And then we have a separate need to archive our data. 24 terabytes may seem like a lot, but we ingest feature films all the time, getting new footage from the studios, so we're constantly having to take things on and off the SAN. And when we do that, we like to generate two copies, one that we keep on site for immediate retrieval, and one that we send off site for disaster recovery.

So the pains of evaluation. As I said, we made this jump back at the first of the year, and so we needed to immediately start evaluating some enterprise-level backup programs that would work for us. And we ran into quite a few challenges. First and foremost, that's not even on this list, is, yeah, you can test backups and backup a few folders, but when you're backing up an actual production system with 24 terabytes and something goes wrong, your window is shot.

You have to wait until the following weekend to actually run another test and see if you've fixed whatever is wrong. But in addition to that, we ran into maximum file size and volume size limitations with some of the programs we tested. Little or no Xsan support, no LTO3 tape library support, which we had recently acquired, again, for performance and capacity. LTO3 seemed the right way to go. No simultaneous stream support.

And by that, I mean... Backing up from multiple sources onto multiple drives at the same time. Not everybody does that. No parallel streaming. Parallel streaming allows us to take a single data source and make two copies at the same time on two different devices, which we use for our archiving. And then I'm sure none of you have run into the experience of calling tech support and not getting answers that you like.

When you're evaluating software, it's even harder. But a lot of them we found had bad support. And then poor backup and restore performance. The backup and restore performance is critical for us because we have a lot of data to backup in that window. And if your performance is subpar, we know we can't make our window.

So what was the solution? Well, a temple came to us a few months back and showed us this product that they had called Time Navigator. And I have to say we were skeptical at first. We had already evaluated quite a few products and been really disappointed. But a few of the features that we liked right out of the box, Xsan support, it's a plus, LTO3 tape library support, another plus.

They had a professional service team that was able to come down to our facility, help us set it up, go over our general backup strategy, which is key. You always need to have a plan before starting a backup. And really able to tune and tweak both our setup and their software to make sure that we got the fastest performance possible. And the most impressive thing out of it all is we had it all up and running in one day.

We were doing test backups, doing performance benchmarks, everything on the very first day. Extremely, extremely fast. In fact, so fast that it is the only product that we tested that when we gave it the full 24 terabytes to backup over a weekend, we were able to meet the very first weekend we ran it.

Even more, couldn't fit it all in one slide. There were some bonus features that we really liked. The object-oriented approach of the GUI is really fantastic. It takes a little getting used to, but you can get an overall view of what's going on really easily. Everything's drag and drop, moving slots around, all drag and drop, very, very nice.

The Time Slice Approach So, time slicing, the way they work it is you can create a virtual file system on your screen at any given point in time. You say, "Well, I want to see what the disk looked like two weeks ago at 3 o'clock." And sure enough, it'll pull up exactly what it thinks was on the file system at 3 o'clock two weeks ago.

And then you click on a file that you want to look at, click Instances, it shows you every version of that file that was ever backed up. If that's not the file you want or you don't see what you're looking for, you can just scrub forwards and backwards in time until you find what you are looking for. Extremely, extremely handy when you're not quite sure what it is you're looking for off the bat. Synthetic backups, we love synthetic backups.

Again, I've said before, you know, things will go wrong. It's inevitable you're going to have SCSI problems or a tape jam at some point and your backup window is shot if that happens. So, a Tempo has kind of a unique solution to that problem. They can go back to your previous full backup, they can go back to your previous incremental backup, your previous incremental backups and generate a new full set if for some reason you can't complete your full backup. So, that's a fantastic feature. And then the parallel streams I mentioned before, we can then take our archives, archive on to two separate tapes, keep one on-site, send one off-site, all works really beautifully.

So, we like the product so much that now, going into phase two, we are going to implement it across the board. We are going to run it on our Mac OS X server with a centralized unified interface. We are going to use it to backup all of our Mac OS X file servers, our Windows servers, and even our application servers like our Exchange servers and our Microsoft SQL servers.

And all utilizing our existing equipment. We have a Spectralogic AIT4 tape library as well as an older Mammoth 2 that we use for archiving. So, that's about it. I hope you take the time to evaluate the product. We think it's really great and we hope you will too. Thank you very much.

Great, so I just wanted to quickly summarize the backup solutions that we have available for X. For those of you who have been around quite a while, if you remember the early days of Mac OS X, there wasn't a lot of them at the time, and we definitely have a lot of new vendors on board, thanks to the Unix underpinnings. So we've got all those vendors on the platform, and we have a grid as well that we've put together in terms of Tiger support, Xsan support, and of course server availability on our platform.