Hello everybody and welcome to WWDC 2007 -- the second-to-last presentation.

Today we will be talking about building system images for large-scale deployment. Where this information is coming from is from the group I am part of; Custom Software Solutions. We are responsible for imaging within Apple. First of all, let's go over who I am.

First of all, I am a build engineer for Custom Software Solutions, and I'm one of the people who actually builds the images. I act as a consultant between internal Apple groups, external Apple groups, and our customers to work together and actually be able to fix their problems. Third, I'm an advocate for both internal and external issues resolution.

This means if we have a problem with third party software or one of our customers has a problem with third party software, we will go to that vendor and work with them so that we can deploy their software for our customers. Oh. Sorry. And I'm also a big geek. And a second generation V I user. Because I already run one operating system.

So Custom Software Solutions. We're lucky. We're in the middle of everything. We get to see our big projects. Our main customers education, Enterprise -- we put government and Enterprise -- and then we also do iPod imaging. This is usually for corporate gifting, but we have many customers who are using it for new and novel ways of advertising to welcoming new sales people.

Also, when we're talking about large scale, what is large scale. Our biggest to date is probably about 37-and-a-half thousand machines. Within the next year we hope to bump this to 60. The thing about this is the only limitation on how many custom machines Apple can produce is how many machines are coming out of the factory.

Every one of our machines can be customized. Our factories are ready for it but -- come on! So we're going to go through first a very general overview. What is imaging. Get everybody in the same vocabulary. Then I want to talk about where in this whole process you can actually do customizations. Where the easy places to effect change.

Then let's talk about some of these implementations. Talk about how they actually work, and we'll actually go through some command line to accomplish it. Then I'm very lucky. We have two case studies for you. The first case study is going to be from Josh Wiserbaker (assumed spelling) from Enterprise. The second one is going to be for education, and that's by Daryl Haws (assumed spelling).

Then we're going go through a Q and A. First, the wonderful world of imaging. What is imaging? When we go down to the very base components, it's a series of customizations. We want the system -- sorry -- we want the system preconfigured before the user gets it. Which is why we build the images. One, because it costs less. If we're doing things exactly the same, we can streamline the delivery.

Also, if you have a computer that fails, you need to add a new one, we have rapid return to service if everything is on the same image. Consistent computing environments are what a lot of our custom machines go into. For example, university computer environments, or kiosks. The one thing that your customers are going to expect, and of course the quotes, all problems are going to be fixed before they see this image.

How do you deploy these images? Well, for new and existing images, we can use all of Apple Suite. ARD. System image utility, ASR. And we just went over for new Macs, we'll deploy them for you. We'll worry about putting all of the bits on the disc. In all of this, we'll just cover all of our different deployment types.

Our deployment times are segmented in two major categories. The first one is common computing environments. This machine will be used by multiple people. It's not a one to one. It is for a lab, a kiosk. The second one is when that computer is owned by one specific person. We call these one to ones. We do those for higher Ed and K through 12.

And to give you our vocabulary when we talk about a monolithic image. This is what I bet you guys do now. This is bits on top of hardware. We are going to build an image. Get ASR, pull it off the disc, and then deploy it as many times as we can. Incremental is where you build your image slowly, one piece on top of the other. Third, hybrid. This is what we do within Apple. We use a monolithic base, and then we use incremental improvements on top of that. We'll go into that in just a little bit.

The advanced characteristics that I want all of you to remember as I'm going through this is we want to be hardware independent. I'm not talking about our competitor hardware independent, but any Mac, you should be able to put your image on. So for example, I am deploy a MacBook and a MacBook Pro from our factory with an identical image. There's no need to have very specific hardware independence. Second, modularity. We are in the business of pushing very large, very complex images through our factory. So we want to break these images into as small pieces as possible. And use these pieces between projects.

Having these pieces separate from each other let's us test these, and we get reproducible imaging component by component. And again, layering. I get that a lot. Traceability. Traceability is very big because you need to know your customer's environments, and then you need to engineer a solution that accomplishes these requirements. Traceability let's you know what you put on there, how you put it on there, and how you need to do it next time. Reduce complexity.

Focus on the customization opportunities. A lot of the Apple ecosystem is there to help you. You don't need to worry about finding two Novel of solutions. We'll show you here how to make your life quite a bit easy -- easier. And then, also the economies of scale. I need to be able to put an image on 100 computers this week -- maybe 7,000 next week. How do you accomplish that.

How do you know that when you build your smaller image that it can then be ported to the larger environment? So customization opportunities. This will be our -- most of our content section of the presentation. So what is a customization opportunity? We define that as the little places within the praising system where it's really easy to change how an application works. For example, changing how an application runs is registering it. It's preconfiguring user settings.

So what are favorable junctures? Your first set of favorable junctures are file system ones. Because we're going to say that your machine has not yet been booted. If it's not booted you can't use any APIs on that system. You must be sure that all of your file system is written before hand. The next, is when you are actually on the system booted. You have many avenues here.

Boot. Launch Demon, launch agents. If you're running on older versions of OS X, start up items. You know? When a user logs in, many operations can be done. And they can even be system-wide operations. And the first time a processor application is started is also a very good place to do customizations.

Let's look at the big picture. So we're talking about deployment. But unfortunately deployment is the center of this whole equation. You're going it lose contact with your machine as soon as it is deployed. So what happens after its deployed. Hopefully they turn it on. When they turn it on you, of course, have your boot, your services, set up assistant will run, an then the user logs in.

Well, if the system is running, you also need to worry about current users, future users, users that are Admin, users that are standard, and by the way, of course, all of this is on OS X. And in order to make everything work after the deployment, we always use the Apple installer. But we did drink the cool aid.

So your base OS. What cat are you using? This presentation is mainly going to be for Tiger. I apologize for that. But all of the concepts you are going to see here you can definitely pass on to Leopard. But Leopard will make your life quite a bit easier. Because you have far updated tools and session before this and session after this -- very good technical detail.

So why am I so worried about OS X, why am I so worried about the operating system? That is the decoupled base of our image. We always build on top of whatever the latest shipping operation is from our factory. A note on Tiger, remember, there is a Power PC version, there is an Intel version.

With Leopard, that, of course, goes away. On your base OS, we always include software updates and drivers. So that way after you've completed your base OS, you can then test it, log in, see if everything works correctly, and then worry about laying your image on top of it.

Now we're going through and we're talking about processes, and we're talking about time. I want to step back for a minute, and I want to talk about the file system. Because we're definitely straddling do you write a file, or do you call a process. So this is my view of the world. The kernel, of course. Do you require kernel extensions, drivers. System domain is generally what Apple gets to control. You will turn services on an off within there, but you're not really going to do a whole lot in there. And of course these are template.

The local domain or the UNIX domain is where most of your customizations will happen. Besides user customizations. User customizations are always layered on top of everything else. So the one thing that this picture does not show is that your standard users and your administrative users will have different permissions in different domains.

Your first time that you can customize during the execution of a Macintosh is of course boot. But why is this important? Well, we do everything when the machine is not booted. Therefore we can not get I P addresses -- Mac addresses, and we cannot get the serial number of the Macintosh. So we use first boot to go through and configure all of these hardware customizations we could not do while the system was not booted.

The next thing that we use -- I apologize -- what we also do on boot is paranoia and security. So we will reset the firm ware password. Either the first time or every time, if you would like. And a lot of our customers use tattle-tale scripts. Is this student adding software. Is this student adding MP3s. Is this student changing permissions on the file system.

After boot, set up assistant is your user's usually first introduction to OS X. You can invoke set up assistant even if I user is defined. So for example, you have a head administrator for your IT help desk. So that way a student calls in, you can get into their machine and help.

Well, you can set up this user, invoke set up assistant, and then that user will get to create their own administrative user with their user name and their password. This will let you have control over the system, and then let your user customize on top of that. Please remember; it does create an administrative account. We're going to talk about localization clean up here in a few minutes. Localization clean up is a service that runs after set up assistant that will go through and trim language content, or local-specific applications for you.

Final note; set up assistant does step on preferences. So if you want to set a user name and you set it on first boot, then forget about it. Whatever your user defines is going to be the name of that host. Log in hooks are very frequently used by our customers. First main purpose is updating the software, or updating virus definitions.

How do you know when that update is complete? Or do you care if that update completed? So what we would like you to do is create completion-based log in hooks. For example, let's say you need a trim software. Well, if the user logs in and they say, oh, I want all the software on there. And kill your completion based log in hook, they should be prompted the next time they log in. This also enables user independent configuration, but be careful. These log in items -- log in hooks -- do run as root. So you can definitely cut your foot off.

Application execution is the final place to do customizations in the timeline I'm presenting. When an application is executed it should always check for its user preferences. Anything in the local or system domain that it needs, including frameworks. Any services, demons, or agents that it relies on, and, of course, kernel support. We'll talk about this in detail during our implementation discuss. Let's revise the big picture.

So now we have a mirror image of our boot process or each one of our installs. I want to show you that for each software we decide to include -- or our customers decide to include in an image -- we need to worry about all the different phases of customization opportunity for that software to be successful.

Give you guys some testing considerations to think about during this whole presentation. You must know your destination environment. Not only, you know, is it going into a lab, or is it going into, you know, teacher's office. What type of lab is it going into. What type of teacher is that.

Who is this destination user and how will they be using it. What support software are they going to need to accomplish what they want on that. If you install two pieces of software, will one of them step on the other's preferences, or will one of them cause another piece of software not to run.

And, unfortunately, we do have to worry a little bit about hardware. For example, let's say you are deploying on a very old iBook. These have modems. Now you want to put your image on a MacBook. They do not have modems. If you use the same monolithic image for the same configurations, you're going to see an phantom modem on your MacBook. And that will definitely confuse your users.

So how do we control this image as we're going through and figuring out how to implement it. First thing, except 30 percent, or at least a third of your time is just defining the image. Just figuring out what the customer actually wants on there, as opposed to, well, could you turn that preference on. And that's the next one. Do task-based definition. Not turn off ARD. Say, we don't -- we would not like our students to be able to ARD other machines. Different perspective on it.

Treat image creation as traditional software. This is -- this is creating a product. So please use software development cycles, and of course, your favor, Quarter Vision. Be creative. Remember, a lot of the people who are writing the software aren't necessarily thinking that it's going to be deployed to 100,000 kids.

So please be sure -- please be sure that you be creative, but also keep it simple. The next two, of course, be methodical, be organized. These definitely relate back to your software development. But the more methodical you are, the more organized you are, the more reproducible your images will be.

Also how do we reduce our image complexity. Again, you saw for development techniques. They have been studied for a very long time. They work. Specify desire. Not implementation. So the customization opportunities that we go through, that's where we have found that customizations work the best. You might find other ones, but I would suggest starting at these.

Everything within Apple is an ecosystem. Use the Apple ecosystems. Also Apple mailing lists. A lot of engineers do troll those, and if you have questions, a lot of times they will be answered. But I cannot say that they will always be answered. So let's try -- so how do we implement these customizations of the how do we take this knowledge of the customization opportunity and what we want the image to do and combine them? Start at the beginning again. Preconfiguration.

Your development machine should be big. For example, if you're trying to do an image on an iBook compared to a Mac Pro, you're going to have a very big difference in the imaging speed. Also, I heard in the last conversation -- in the last discussion -- please use target disc mode. That is the easiest way to get your image monolithically on a machine.

But remember ownership much and where do you get the base OS. You get it from your DVDs that come in the box. Or you can go out and get a shrink-wrapped copy of OS X. But remember, if you get a shrink-wrapped copy of OS X, that doesn't include iLife in the iWorks trial and other applications like that.

When you're installs software, the drag and drop ones are of course easy. But what else is there? You know, of course, we're Apple. We're going to say we want you to use the Apple installer. There are other installers, and they work very well. But just be sure they do work in your environment. If you're sure they work in a non root volume.

During your application installation, remember, you have a current user, you have other users on the system, and you have future users of that system. If you need to do configuration, do it for all three categories of these users. And local preferences and resources, for example, frameworks and preference files should be read and verified.

Local domain customization. For example, Mac address. You need your Mac address to be able -- or -- start from the top. Booted versus non booted. So -- if your machine is booted, you have access to all of the APIs, you have access to all of the command line tools. If it's not booted, you don't. How do you go through and customize those changes. In our environment we will usually pick a P-list or modify a P-list. Also, hardware-specific settings used by applications need to be set.

And other hardware. So are your students using a microscope? Do you have a big board for those students to use? Let's talk about how to implement setting the computer name. In this situation we are dealing with Tiger. So we do the conventional start up item. Within this start up item we use the command line utilities, S C Util, to go through and set the computer name. But remember, this is on start up. So we have two different styles of start up items that our users use. Self-destructing ones based on completion. Okay. I set the computer name. I'll just go ahead and die. And then the continuous ones.

The only difference between setting the user name programatically and setting it on this image is just when it happens. When is the information available on that note I would really like to talk about ARD real quick. Because when you're going through and implementing your local domain customizations, ARD can help you quite a bit. Network set up and systems set up are two executables you can use to ply your system customizations. The reasons these are good and interesting is that they are an API. You are not hacking P-lists. You can probably assume that they're not going to change significantly.

Now we get back to user customizations. Bi-host (assumed spelling) preferences are the biggest question that our group gets asked. Well, let's go over what bi-host setting is. Let's say you have a common computing environment. That means you might have very low end machines on one side for just e-mail and Web.

And you might have very high end machines for video editing on the other side. If a user sits down at the lower powered machines, they're probably not going to want a very graphically intensive screen saver. Whereas if they are on the final cut machine, then they will have the beefiness for a good screen saver. Well, that's where bi-host settings come in.

They let you configure your settings for the specific machine you were on. User customization for software is our next biggest problem. A lot of times software developers will say, yeah, registration belongs in the local domain. Our problem with that is a lot of times they'll put it in the local domain and not give standard users right to see that license. That means that your standard user is going to log in, and they're going to be prompted for license information every time. Software configuration, also -- and user preferences.

So let's go over bi-host configuration. What we are trying to do here is we are trying to suppress the Bluetooth menu item. So the Bluetooth symbol in the upper right-hand side of the screen. This is actually done through the system UI server. If we look at the format of a bi-host P-list, it always goes domain, hardware address, P-list. So we'll see com Apple system UI server is the domain. Decaf bad is the IP it the Mac address -- and P-list.

So what we would do is we would go through and in each user's bi-host directory we will put a template file. We will put the file with all the content and either put decaf bad, or any other token in there to show us that we need -- that that file needs bi-host provisions.

On the first boot we are going to look in all of the bi-host directories, find the files, and replace decaf bad with the actual condition Mac address. By the way, this also can be done with parental controls or N C X settings. Localization clean up. Can I get a show of hands of everybody in here that has used localization clean up or loc-setup (assumed spelling).

That's three. So one thing that I wanted to tell you guys. Always run localization clean up if you suppress setup assistant. What localization clean up, again, does, is it will go in and for the main lo-cal of that machine, condition it for that environment. What is localization clean up? It is a script at user S Ben loc den setup.

This script will not be on your machines right now. It is removed after it is run. So please go and get a fresh install and then this script will be available to you. What this script (Inaudible) is your language and your country. These come from dot global preferences P-list. But because setup assistant has not been run, you are going to have to put these values in there so that localization clean up runs successfully.

Another very common customization is enabling services. In this situation, you have a very large lab, and you want to use Apple events for some reason. So how do you enable remote Apple events. Well, remote Apple events are a launch demon. You can go in a system Library Launch demons system PC dot P-list. And use defaults to remove the disabled key.

Sometimes you don't need a restart. Usually you can go and just go directly to the pref pane and test it. But in this situation, let's say we restarted. Now we want to go in and actually verify that our preference, that our configuration took. Well, unfortunately, it didn't. There's a problem. We restarted. We've gone into System Preferences, and we click on the check box and nothing happens. We click on stop. And nothing happens. And after a while it will just grind to a halt and then really nothing will happen.

what's the issue? Who -- oops -- who broke the UI. Where did this come from? Well, default broke the UI. Defaults creates a new file, and then it copies it on top of the old preference time. The default emask will leave it as 600. The permissions need to be 644 for the preference pane to interact.

Other verification steps that I encourage you to run are not only checking permissions, but know where your resource works are, and know which ones need to be there. Also P-list formats. This is if you want to get the most picky you possibly can. Keep P-list formats identical. So if you have an XML P-list in the retail bundle on your base OS X image, please keep it as ASCII -- I mean, as XML in your image.

So we've talked about applications, we've talked about the local domain, well, how do applications actually interact with that local domain? I would like to use ARD as an example. ARD will settle for an example if it cannot find all of its resources. So you've installed ARD on your main lap top.

You get a new lap top, and you're just saying, oh, I'll drag over the dot app. Drag over the dot app to your new machine and double-click it and what happens? It will go through, first, verify your local domain, and then verify your kernel extension. And then it will go through and setup your user. In essence, reinstalling itself.

And now let's talk about common pit falls. I've done every one of these a hundred times. So please don't feel bad. First, your installer scripts are probably going to be the center of a lot of issues with the installers. A lot of installers do not take the script argument as they need to be. For example, the -- install location versus the install volume. The install volume is given to you because even though you want to put your application into applications, other services need to be littered throughout the system.

Second, bad permissions cause all sorts of havoc. We have seen countless random failures, UI issues, that all come back to permissions. If you have to modify a P-list, test it more. Because in essence, you're going around what the programmer has defined. With the API calls, you can actually go in and within reason, make the same API call on different versions of ROS. Whereas if you go in and actually modify the P-list by hand, it's probably going to break the next time -- or it will break the next time the developer changes that P-list.

Over complication. I know that I just presented you a very long, drawn out, somewhat complicated picture of the world. But watch out. You can simplifying a lot of things. You can leverage layering. You can leverage modularity, you can leverage software development techniques. So please don't, please use those to simplify your image. Also, very common pitfall is running repair permissions at the wrong time. So what is repair permissions? Repair permissions will go into your receipts directory and verify the permissions on all of the OS and all of the files.

Therefore, if in your image you've installed a piece of software, noticed it had incorrect permissions, and updated those permissions, next time the user runs repair permissions -- it's a big problem. They've gone through and they've reversed your permissions change. How do you get around this is repackage it. If you change permissions, show the permissions in the bill of material. That is the convention for the Apple installer.

And since we're talking about implementation, let's talk about actually getting your image out there. If you want an image a ton of Macs at a time, probably not going to have -- probably not going to boot every system, go through and configure everything by hand and then keep going.

You're probably going to do a monolithic or a hybrid image. So if it's a monolithic image, remember there's multicast -- multicast ASR. You can use a DVD, but this will be your slowest way to install. And FireWire target disc mode. And then ARD is used to layer the rest on top of your image, once it has been deployed and you're in a managed environment.

An example of a different -- the difference in deployment time of a minimal image monolithically as compared to incrementally, and again, this is a very simple situation. I know that there's a lot of detail that many of you will be liking -- or many of you would prefer. But 1 gig is our retail bundle size. Base OS plus iLife, plus iWorks. It's about 14 minutes over FireWire 800 to deploy. Whereas if you only have about a 10 Meg change, a very simple system customization package, 30 seconds.

This will -- we will loop back and talk about this when Josh Wisenbaker comes back on stage and talks more about the larger deployment cycle. And for those of you coming to next year's WWDC, there are a bunch of Leopard features we would like you to be aware of and learn. Code signing, D trays. Probably a lot of you actually saw these during this WWDC. And then enhanced parental controls. This is the session that's coming right after us. Now I'm very pleased to introduce Josh Wisenbaker. He is one of our enterprise consulting engineers.

So one of the things that you do, even though our imaging tools are very easy to use, and we can really blast out lots of machines, there's just a heck of a lot of work involved in this still. So even though we've got multicast ASR, and even though we can do, you know, as many switch ports of machines at once as you have, if I had a thousand machines and a thousand gig port, I still have to unbox a thousand machines.

I still have to inventory them, carry them around, put them to tables, plug them in, start them up with the N key down -- and then have to rebox all of them. And I have to deliver them to where they go. Unbox them again, put them on some guy's desk.

If it's a professor -- whoever, make sure it okay and make they know how to use the mouse and what not. And we can go on from there. So it becomes difficult. There's just a lot of effort in this. So what the CSS group can help us do in this case particularly is we went and worked with them and developed these prebuilt machine.

And in this case, the physical effort on the customer's end was just reduced to unpack and turn it on. Mouse lessons still required for the people that need that. So a real world example of this. A customer who must remain nameless for me right now, decided that they wanted to put out about a thousand iMacs.

Now, the initial plan called for local imaging. The customer decided, you know, we can do this our selves. We can image all this stuff. But some networking issues, some staffing issues, and some physical space issues. Where do you find the time to set up a thousand machines? Where do you put a thousand machines? You know? Those boxes take up a lot of room. And in this case, they're iMacs. The box is fairly large.

So, it took them about a week or so to get the first 200 units out the door. And then once they did that, they realized, wait a minute. There was more we should have done to those images. This isn't working right. So then they had to figure out how to reimage all this stuff. This is a new to the Mac customer.

This is not a real good first impression for us to make, as far as imaging went. So because it was a fairly large Enterprise customer, the Enterprise C E team, meaning me, went out on site -- the other guys didn't get to go to this one -- so there are more of us than just me. So I went out and I helped them work on their image. And we created an optimized round two image at this point. So one of the most important things that you're going to do when you build images or do anything, is you're going to document, document, document.

And in this case I used an out lying program. I just clicked it open and as I went through the image I documented every step and everything I did. This way the customer could rebuild this image later from scratch if they needed to. I didn't have to get on an airplane and shoot across the country again just because they wanted to change one thing.

We also gave up the local policy decisions they've been trying to do with group policy -- with the parental controls because they had OD servers in place. And if you've got an OD server in place, why the heck are you doing this stuff with the parental controls. So we helped them move on from using local policies for this large deployment to using open directory OSs. Now when we do this, it makes things very, very easy.

Like if I want to change one line in the log in hook we can do an M C X log in hook, I change one line in the hook, stick it in Work Group Manager, and I'm done. All the machines pick it up the next time they refresh the preferences. The application load and the OS load was trimmed down significantly.

And this helped us take their deployment from about 18 gigs to about 10 gigs. So actually, reduced it by 10 gigs in size. We went from 18 gigs to less than 8 gigs of size. So this really cut down on the time it took to deploy each image, if they did it locally. And then we also trained the customer staff in this.

And a lot of this was part of the documentation that went on. I'm telling you guys, if you don't document it, then it's not going to work. The problem is, is it still takes all that effort. Box, unbox, rebox -- and at this point, we were running out of time. In this case, we had to get these things on desks. So we gave the CSS group a call.

And they said yeah, we can work with you guys and get done fast, and we can help you out with that. And that image documentation that was created is a perfect checklist. How to go through and configure these initial images. So we sent that up to them. That got a great head start for us. If we did not have that documentation for the image build steps they would have taken much, much longer for this process to go on.

So in less than one week we were able to up load that, build it out, send it out to the customer, have them test it, have them verify it, go back, turn around and release it to manufacturing. So we were able then at that point that people were able to order from a custom skew, machine comes in a box, they take it out and put it on a desk.

There wasn't anything else that had to go into that. That cuts the deployment time way down when you don't have to do any of that. We did also set up some local images options though. I spent all this time and spent, you know, the better part of a day creating this image for them. So we still had the need to image on site though. And when you've got a large deployment, you've got a couple different ways to image.

The main two, we've got a unicast and multicast. And unicast is great for these onsie-twosies. Point to point. You can go select the machine at ARD. Have a fully automated image cast imager store say go, and that machine will boot off that and go. But what happens if there's a refresh.

That they decide, hey, you know this week we're going to take all the Macs from building 3, level 2, and we're just going to reimage them all. So we also setup a multicast solution for them there as well. That way if it ever came down to if they wanted to do 30 at once, they could do 30 at once. And boot them up off the network, wipe them all down with a multicaster image, and be done very quickly.

So we do the combination of the CSS and the custom keys for doing the initial setups with local imaging used for doing our up dates and redeployments of machines. So -- at this point I am going to hand things over to Daryl, who can talk to us about an education case study. Thank you very much.

- Hello. My name is Darryl halls and for about the next 20 or 30 minutes I'm 0going to be talking to you about imaging. It worked, Tim. Most people are running out the door. We're getting rid of the light weights. Fooled ya'. Anyway, so --

I work in education, I am a system's engineer up in the Maine area of the world. That's not primary, that's Maine with an E. And yeah, whatever.

( Laughter ) >> For about the last five years or so I have been pretty heavily involved with a little project that we have up there which is known as the Maine learning technology initiative. LTI. I wanted to point out that it's focused in the right place. A lot of people see LTI and they think lap top. It's not a technology initiative. It's the learning technology initiative. It's focused on education, where it should be. It's not focused on the computers and the technology.

So what we've done is we've deployed a state-wide program which is state funded. It's a one to one, meaning that every student has a machine, every teacher has a machine. All 7 and 8 grade student throughout the entire state. There's about 242 schools in the state. That is about the size of the rest of New England. It's quite a large geo to be able to travel around.

In order to meet our budgetary constraints to do this entire project, provide support for the project, provide a yearly image, provide a mechanism for, et cetera, et cetera, we did not have enough money to work in a server model. We would have had to deploy a server per school, we would have had to support a server per school.

And when you're talking about a state that has a lot of rural areas where the local technology teacher happened to also become the tech support person, and that one person is support for the entire school -- you run into an issue with running -- jumping in a car and going to help them with their server.

So we tried to avoid that. Each school has their own tech support. Maybe one or two people who support the entire school. It's not uncommon for you to have one or two people supporting 400, 1,000 machines, et cetera. Our support teams work to support the project. A gentleman named David McKee is really the key focus here. We have a special 800 number in the Apple care that the Maine folks call.

So they call Apple care at this 800 number. And if Apple care cannot assist -- this is the education tier of Apple care -- they actually escalate back from Apple care to David McKee in Maine. And David in money is the Apple care liaison who works with the individual issues that pop up, trying to find out where there are commonalties to support it.

And then we go back to Apple care and solve the problems. So the first project was a four-year lease. A four-year timeline. I am happy to say that Maine came back to Apple and we went forward with a second project. Which is another four years, now starting the second year of the second project. Around 40,000 iBooks. Well, 30,000 iBooks. In the first phase, it was phased outs over two periods, over the first two years. And in the second phase all 37,000 dropped at the same time. The first image I created in a hotel room in Boston.

Back on Mac OS 10.1. And there were a lot of concerns at that time over making sure the project went well. The first image was about 3 gigabytes in size and took me a very long time to do. I've been pretty heavily involved on the imaging from that point forward. But again, David McKee is the gentleman who took over.

We worked side by side in several different images. The current that is deployed is a 30 gig image. So you can see over a four year span of time that things have changed quite a bit. The very first project which is just ended went from 500 gigahertz iBook (Inaudible) 3s all the way up to 1.34 gigahertz iBooks.

Around 40,000 machines -- one single system image. So earlier, Tim was talking about categories of customization. We have well over 150 individual customization line items in Maine. So you could safely say that we touch every one of those potential areas of customization. So at application level, installing third party apps, customizing application behavior -- who has access to what system preference, what System Preferences behaves like, the system level. We add kron jobs, we add log in hooks, we had log out hooks when Virex (assumed spelling) was going zombie on us. All kinds of fun.

Other categories of customization around security. We make sure there's an open framework password on the system. We disable the console user. So tricky folks don't get the Terminal when they shouldn't. I mentioned we don't have servers. So unlike Josh's model, he talked about saving disc space on his image. And he talked about saving time by using a server. We don't have that luxury.

Instead, what we had to do was create local, generic users. So there is an Admin user, a student user, a parent user, a teacher user on every single system that ships out of the factory. And every time we do a new image, those users are -- the passwords are customized when the machines are deployed.

I wanted to take a step back and talk about why we image. Basically, when imaging, this is a task that we do in order to shape our operating system, our final deployment, to meet some local requirement. That local requirement may be based upon our available technology. Or the technology that's not quite yet available to us. How many of you are going, oh, I want that Leopard feature. But I can't deploy Leopard quite yet. Well, welcome to my world.

It's definitely without a doubt shaped by local policy. So when creating an image, I just want -- I want to put the plug in there that you, the engineer, are really the liaison between the policy makers and the final deployment. And a lot of us just blindly apply what's asked of us without thinking it through. So you really need to define what's being asked of you. I had a couple of anecdotes that I brought with me to talk about the disconnect between engineers and policy makers. And I have my little cheat card.

I work in education. It's interesting to me that in education, as soon as you introduce technology, problems become an entirely new issue. For instance, I had a student who's kind of a bully. Out on the playground that student bullys another student. They go to the principal's office and there's a whole entire hierarchy of discipline actions. I can suspend the student, I can put them in detention, et cetera.

That same student bullys another student over iChat or over e-mail -- oh my gosh! It happened through electrons!

Why is it that we must always solve social issues with technology? This has been something that we've gone through with our imaging quite a bit, all again and again and again. I want to give you an anecdote from early on, the very first image that we did five years ago.

When we were in the RFP process we were going back and forth with the state and there was a lot of concern, you know, there's a lot of fear for the first time you've handing out iBooks to middle school student. They're going to take these things home. They're going to have access to them all day. There's a lot of fear about what they'll do with those machines. And so we were asked basically can you ensure that they don't see bad pictures.

- Those bad images shouldn't appear on the screen. And we had engineers running around. There's a third party that makes some software that analyzes the skin tone on the screen and darkens that area. And we're literally talking about this. And Jim Tobin came in, he's a resulting engineer in education. And he said, you can't teach a back pack not to carry a Playboy.

- Why are we trying to solve that problem?

So we lucked out on that one.

And I -- one last anecdote -- the governor of our state -- I wasn't sure how far back I wanted to go with this one. But the oftener of our state was given a loaner machine. And he's -- the governor at the time, Governor King apparently was give a loaner lap top which he brought home. His son loved the lap top, grabbed the iBook and went to town with it. And filled the hard drive with MP3s.

If you fill a hard drive, you can't boot it. So he effectively broke the machine. And you know, it came back. And that's the anecdote that led to -- we were told by the state when doing our first image, ensure that students can't listen to music files. You know, it's five years ago. We didn't have iTunes, we didn't have the iTunes music store. People were worried about piracy. They were worried about swapping MP3s. Kind of makes sense.

But again, the policy in the engineering side of things, right? What am I actually being told here? What do you want me to do? We ultimately ended up deciding after a lot of discussion that what you were asking me is to disable access to the iTunes application so that students cannot easily create MP3 files, swap MP3 files, and playback MP3 files.

But along the way we pointed out to sales management, which is yet another policy maker along the track, that students actually could put a CD into the computer, and drag the A F -- A I F F file off on to the desktop, double-click it, and live to music in QuickTime.

And I hope I get as big a laugh on this one, because it killed me. We were told by Apple sales management -- can't you just disable QuickTime?

So be careful that you really understand the question. As to why your customizing your image. So a quick peek at some of the actual customizations that we do for our image, and this is a really small sub set. We suppress the setup assistant. We have those four users that are precreated. We define ARD settings. So the predefined Admin user is already configured to be able to ARD control that station. We have a one-time run start up item that sets the computer name.

It actually sets a little string, my M L T I, and then the last three octaves of the Mac address. The Mac address is pretty much guaranteed unique. You have that problem when machines auto name themselves, and you bring a hundred of them up on the network at the same time -- and they all choose the same automatic name? And now, you know, N computers minus 1 chooses a second same automatic name? And N computers minus 2 -- so the other thing that we do is this is not a self-destructing start up item. We use plenty of those. But this is actually a start up item that creates a little touch time. Just like Apple setup done and other things. The touch file is wonderful because what we can do is use ARD to destroy the touch file.

And now the next time the system boots it automatically goes through and does that exact same operation as if it had just come out of the box. We set our time zone. Obviously all retail machines coming out in the U.S. are set to Coopertino. Because that the center of our universe. But in Maine, we wanted it to be in the east coast. If you just set it to east coast, it grabs the first city alphabetically, which happens to be Atlanta.

So we added Augusta. And I think the first time we did that, we removed Atlanta. I like them, but we had to remove it. Subsequently, we were able to select Augusta. We -- software updates. This is a big one for me. What in the world are we thinking, folks? Why do software updates configure themselves to automatically alert a user that there is a software update, when they cannot install it?

What is up with that? So we go through and we disable all applications.

Apple applications, third party applications, and the Apple OS to not prompt user that there's a software update. There are two techs in your school that would actually install the update, probably remotely. But you know, they're being prompted. You've got 400 users knocking on your door all day long. It says security up date.

I think I should have this one now. Ahem. So, we disabled the Dashboard in Tiger. There were several reasons for this, but we made sure the users were not launching Dashboard. A lot of people complained. Power management concerns. This is a mobile project. Right? These students are using the machines without being plugged in for about a school day. Right? They're running them several hours a day.

So we wanted to be very careful about our energy saver settings. We tweaked them from the default. And anything else that we could do including turning off Dashboard to ensure that the system wasn't wasting resources. We install X 11, and for things like grass, G A I, et cetera. We added some additional network locations where even the automatic network location is tweaked.

See, by default, everyone's wireless. So Airport is our default. It's not, in the OS. And we really don't need that modem at the top for some reason. Safari customizations. Each user gets their own homepage. Each user has a favorites. And here's a fun one. The QuickTime Player. When you launch the QuickTime Player, do you know about the hot picks window that opens up? Well, first off, it's a little slow.

Because it's making a network connection. Now, imagine that times 30 kids in the classroom. When you go to launch QuickTime. But then number 2, I don't know about you, but middle school students seeing the QuickTime window, every time you launch QuickTime, there's something about necessarily in his underwear or whatever that is, you know, it's a little bit beyond P G 13. So we turned that off by default. And I wanted to support some of the stuff that Timothy was saying.

We really used to do a lot of monolithic imaging. That disc image that just goes right to the raw metal. And we had these considerations when we would get to the point of actually capturing the disc image. All right, now on the master machine we're about to capture. Let's go through and make sure everyone's trash is empty. Let's make sure that, you know, browser. All of these line items that you do before you actually capture. Then you capture. You deploy it. And then you find out you missed two of the steps.

So you go back and you do it all over again. From scratch. Et cetera, et cetera -- all of these things ad infinitum. Really prone to failure. A realtime consumer. So if you look at the modular approach to it, where you're just creating packages of changes -- you're just creating those packages that actually leave behind receipts with permissions on them.

You save so much time. Because the base OS is -- was already supplied to you. And you're just modifying the base OS. You will save yourself a ton of time if you can get into a more incremental approach than strictly monolithic. But worst case scenario, you take the base image you currently have, you apply the packages you've been working on, and you capture a new base image.

You really didn't do anything in the middle to cause all of those tweaks that you have to do to go back and fix. And you've got yourself a new base image. So at this point, I think they're going to bring everybody up and talk about some common questions. So we can answer your questions before you ask them.

- There's one common question I forget for all you developers in the room. Have I modified your installer? Yeah.

Yeah.

- So if we can bring up the house lights -- we're actually going to go through some questions, but if you guys can get to the microphones, we're going to have an opportunity to get some questions from you. But these are the ones you're going to ask first.

(Inaudible) --

So please feel free to chime in whenever.

Monolithic images are not bad. They're just really bad when you've got to do 40,000 machines.@ Two, must I use the Apple installer? Uhh -- yeah. We do. It's one of those -- you do not need to. You can do this by laying files down. You can use your own development techniques. But we have been using it. And it's been great. We don't have stability issues with it.

Should I modify temperamental installers? I do. Yeah. You're not going to get your (Inaudible) materials correct unless you go through and change permissions of the or in my case, e-mail Timothy. And he'll fix it for me. Yeah.

Where is help? First, the Internet, of course. Second, Apple mailing lists.

If you are part of an Apple sales cycle, your presale engineer will actually be there to help you out. They will -- they can come over and show you how to do your image --

So this is known as the tier approach. If you have a problem, you come to me. The S C in sales and education.

Or to Josh, the, well, S E in business. And when we can't answer it, we e-mail Timothy and ask him to fix it for us.

- And then I ask Bert.