So while I'm bringing our panel up, if you'd like to begin lining up at the microphone so we can call on you if you have questions about what we're going to do. First, I'd like to bring up Greg Neagle, Senior Systems Engineer for Disney. In reverse order, Nigel Kersten, Systems Administrator, Mac Operations for Google. And Joel Rennic, Consulting Engineer, Manager, Enterprise for Apple. Thank you.

My name is Schoun Regan. I'm the provider development manager for Apple, and we're going to have a good time. So the first question that I have for our esteemed panel here is the favorite line from their favorite movie. And while they're thinking about that, We're going to give a chance to answer in just a minute.

If when we have topics that come up, if somebody has a different way of doing something or you would like to add to the conversation, Chris Bledsoe, he's right over there, has a handheld mic so he'll be playing Phil Donahue, running around, sticking it in your face. Just raise your hand and Chris will sprint over there as fast as he can. We ask that you don't stick your leg out and trip Chris.

So the first question was, who's Phil Donahue? Thanks for playing the feud. You can get a lovely home version of the game. and I'll see you next year. All right. All right. Favorite line, Greg? I've got a bad feeling about this. Nigel? I don't think you want to do that, Dave. Is this being recorded? Well, that won't do the voice very well. Say hello to my little friend.

Nobody's up at the mics yet? So we'll start-- Session's over. Thanks. Yep. We'll start it off this way. Greg? So I have a question for Nigel. Actually, Nigel and I have been talking on and off during the session, and I think Nigel has a really... important idea about best practices about the future of systems administration and something about his position at Google makes him kind of uniquely qualified or at least in the right kind of environment to express this really well.

So I'm just going to -- it's not really a question to Nigel so much as an opportunity for Nigel to elaborate on his grand unification plan for how all Mac systems administration must be done in the future. Since Google is going to own everything shortly anyway, it will be done his way. So Nigel. Thank you for that buck, Greg.

So I guess what I wanted to get across was that I spent the best part of a decade in higher education before I finally escaped and came to Google about a year or so ago. And one thing I really noticed, this was my first experience working with a software development company, not working in education, not doing consulting for advertising firms and the other sort of bits and pieces I'd done, but working for a company whose core business was software development.

And so we have a lot of infrastructure at Google and some of you might have seen at Google I/O Guido van Rossum came out and demonstrated bits of how Google does code reviews internally. We have a lot of big tools already in place that are for software development, for code reviews, release processes. And coming there as a systems administrator, these things are amazing.

And this is the sort of stuff we don't do as systems administrators. We still sort of manage things in quite a complex way. And I think that's what I'm really interested in. We don't do these things in quite a clunky, manual manner. We don't tend to do peer review and release processes as well as software developers do. And I think there's an awful lot we can learn from software development as systems administrators that we're not necessarily doing now.

One kind of side correlate to this is that just the same way that programmers rarely write assembly or machine code anymore and use highly abstracted APIs because it's faster and it's more efficient. I think that's the way systems administration is heading. Tools like Puppet and CFEngine that are based around managing resources on your machines and not just managing everything at the file level, like a big tripwire system.

You're a member of the reformed Rad Mind Administrators Club. Which Greg has not yet been made a member of. Along those lines is something that we've been trying to do when we reach out to a lot of customers that we've talked about, is building that whole methodology and kind of heritage of administration on the OS X side that I think has been missing. Partially because a lot of you were thrown into this environment not having necessarily come from a traditional or a real system administrator background, but being thrown into OS X to do it just because you were the one that didn't raise your hand fast enough to say no.

And so we miss a lot of things, and a big part of the conversations that we have when we go out, especially around client management, are the creation of things like service level agreements. And kind of defining the relationship between you as the administrator, the users, and management, and understanding where the boundaries of all that are. All too often in the past, the Mac administrator was the person that made everything work. And so if it was the phone that broke, or if it was the iPod, or if it was the internet, you were called in just to make it work.

And so you had no real concept of where your boundaries were, and you were getting pulled into a lot of different directions, and perhaps spinning your wheels into doing things that just shouldn't have been there. Constantly I get a lot of questions about how can I prevent my user from doing this or doing that or something else? And he said, well, why don't you just enforce a policy? If you can't do it through the system itself, say, hey, don't do that. And that's where, for the stupid things, you can actually draw a service level agreement.

And define what's actually handled by HR as a fireable offense, if they do that, and what's actually handled by policy on the system, and what's handled by you as an admin, and so that you're not having to run around and try to do everything. These two gentlemen up here have much different system administration methodologies for their users, whereas Greg, and he'll correct me if I go astray here, is a much more locked-down environment. You control a lot of the pieces using RadMind, making sure that everything's the same way all the time. Nigel is much more of a, you know, you're not going to be able to do everything the same way all the time.

And you're going to have to have a lot of, I think Gartner calls it the consumer of the cafeteria approach, where you kind of hand the machine out, and you wave, and you really hope that they do some of the right things. And you can put some pieces and some administration in there to kind of whack files and put them back into shape when things change. But it's a very different methodology for kind of managing this. You know, we're kind of coining some terms in here, fences, not walls. And that's where Nigel is, with more fences around appropriate behavior and asking you not to do other things.

We see this a lot in the high-tech industry. If you're a high-tech developer, you want to have root access on your box. You get paid a lot of money, they want to make you happy, you're the lifeblood of the system. So they're happy to give that to you.

But then how do you manage them? How do you guide them into the appropriate level of behavior? On the other hand, if you're working with some of the creatives that don't care about root access, they just want everything to work, and they're not really aware of how the things work under the hood, it's almost best if you try to remove as much of the complexity as possible away from them. So that all of the stuff is being done on the spot.

All of the stuff is being done under the covers, and they can't get dangerous, and they can't break things on their own. And I think all too often people spend too much time trying to get somewhere in the middle when they really need to sit down and kind of define which kind of methodology are we going into.

Because there's definitely pluses and minuses on both. Yeah, I'd agree with that. And just to add to that, the system administration still tends to be reactive instead of proactive. You don't go to the desktop until the end of the day. You don't go to the desktop until the user has a problem.

How do you know the user has a problem? Well, they call the help desk. Well, there are several tools that Mac OS X already has built in to alert you, launch D scripts, watching items if something goes wrong before that user picks up the phone. So you can be proactive.

And I think going along with what Greg and what Nigel do, that also is an excellent way to better manage your time. So with that said, thank you. With that said, we'll go ahead and start over here. Yeah, I'm curious with the panel if you guys have looked at idle framework and seen if that works in your enterprise or not.

It's a framework. It's ITIL. So when I escaped from Higher Education Australia, we were constantly being threatened with the ITIL stick, but I managed to leave just before it landed on my back. We do get some questions about this, especially if they've come from that more very methodical system administration side of things.

And typically, how many people know what ITIL is? Actually, that's a lot more than I expected. Most Mac administrators don't. But you're right, that is kind of setting a service-level agreement and kind of setting some of those basic standards in there. What I don't like about ITIL is it's so massive and so encompassing that it really swallows a lot of people whole.

So of those that knew what ITIL was, who's actually implemented it? That's a much smaller set of hands. But we are looking at that. So at Apple, when we talk about, and specifically we've built this around client management, we're kind of looking at scales and differing ways of codifying and identifying how well you're doing.

Because a lot of the, probably a big question a lot of you have here is, well, what is everybody else doing? What's the complete scope of what I could be doing? You know, some people might be doing a little bit of managed policy, some people might be doing an awful lot of managed policy.

What's a good level? What's a good scope? So we've actually gone through and kind of, in the spirit of ITIL, but not really ITIL, kind of condensed it down to about eight or nine different categories and are ranking people on them. And they go from one to five, but that doesn't necessarily mean that five is better.

For example, if you're following Greg's model, you might have a very, very high level of policy locking everybody down. And so we might say on the policy level, you're a five. But on Nigel's model, where it's a little more free-flowing, we might knock you down to a one or a two.

That doesn't necessarily mean that you're bad, but it does mean that you have a different methodology of approaching this. And so what we've been trying to do at Apple is to go through and actually provide a dollar value behind each of those. So if you want to go and do a five on deployment, which is getting them imaged at the factory or whatever else and pushed out, then your costs per support are going to go down, because you're doing a lot less work when the box arrives in.

If you want to do a high level of policy management, your costs might go down as well, because your users aren't getting into things they shouldn't be getting into. On the other hand, if you do a low level of user management, your productivity might go up, or your user satisfaction might go up. So there's definitely trade-offs between all of these.

But we're trying to develop a body of evidence that shows if we do this kind of methodology for deployment, we do this kind of methodology for directory services, this kind of methodology here, that I can kind of ballpark how many -- and we might equate it back to dollars. We might equate it back to number of administrators per machine. I mean, because that's a lot of the numbers that people like to throw around.

Gartner says 1 to 250 for the PC. We'd like to say 1 to 500 for the Macs. And we have some customers that are well into the 1 to 1500, 1 to 2000 range. And so I'm looking at that ITIL stuff. People that are hardcore ITIL get mad at me when I say, well, it's complicated and big and all the rest.

But I agree, more of that needs to be in here. And Chris keeps waving his hand like there's a comment. Yeah, comment on the ITIL, because I took a look at it and went, ah, and wanted to run out of the room. There are a number of resources out there for doing that.

There's a number of resources out there for doing what could be better called ITIL Lite. And basically they say, do this first, do this second, do this third, because you'll get more of the value out of ITIL than just starting from one end of the, what is it, eight volumes or something now and working your way to the other end. So if you're heading in that direction and someone says you need to start doing ITIL, start looking for those kinds of resources out there. Baby steps. If I can make a quick comment. I found that I think assistant administrators It is useful to use ITIL.

I think the intimidating part is figuring out you might only need three of volume one and forget the rest of the volumes. And then the other thing is I think when you're talking to the CIOs and CEOs of implementing ITIL, it's strategic never to mention ITIL. Because, you know, they Google it and then they look at the volumes and I think they get shit. Overwhelmed, yeah. And they're shit scared. Sorry. I didn't have -- Okay. Yes.

Just wanted to start a conversation here. How many people are allowing their users to be admins and -- Stop there. Okay. Admins. Okay, wait. Forced to let them run as admins. Okay. Forced by the people who sign your checks. Why? Why do they need to be admins? Adobe! What is Adobe? Why do you need an admin to be, to use Adobe products? Okay, truth finally wins. Faculty with laptops.

All right, so, so, no, no, so there are legitimate reasons to, to give admin to your users, but make sure that it's not you're just taking the lazy way out. Sorry, lazy person in the back there. Yeah. Once you give a user admin, you can never take it away. What's the other piece here, question? And how many people have root enabled? So, so we're, we're not done with this.

All right. So, root. So, following up part, part C, how many people could be an admin but run as a non-admin because they know that's a better way? Ah, well, this is starting to get better. So when a lot of people ask me about this, should I run as an admin or a non-admin, I run as a non-admin. I know my admin user password, but I run as a non-admin on my box.

And that way, when a user comes up and says, you know, I really need admin privileges because of this, I can say, well, I run as a non-admin, so let's talk about why you need this. Or let's create an account that you don't log in as. It's not your normal, everyday, day-to-day account, but you can elevate those permissions if you want to. And that way, it gives you even more of a firewall between a non-admin and an admin account. I see.

I set that up on a couple users and then they just ended up running as that local admin account. And you know, some users were good with that and some users just, well, hell, I'm going to do it this way. Well, so okay, and so there's also the concept of neutered admins, right? So you can play with the rights and the rules.

You can do it in some cases. Yeah, but it gets so, so crafty. You've got so many holes. It's Swiss cheesy. Yeah, he's sitting over there. All right. I mean, it's much better to take a non-admin and elevate him than try to go the other way around. I'm sorry you have trouble figuring that out, Joel.

Well, I'll just say who's up front and who's in the audience, all right? The problem gets to be, particularly when you've got high-end creative types like lots of scientists, that to really do it right and take away those privileges, skip the lazy comment. You have to figure out exactly what they need and make sure that you've got it all there. You've got the patching in place.

It is a lot of work to make sure that you can take away those rights and they can still do their job without complaining to management. I do agree with you, but I think Joel's comment, and I do this too, you should run as a non-admin day-to-day so you know what it's like to be a non-admin. I'm there. I'm doing it.

Yeah. Good. So that way, if there are things that prevent you from doing your job and you think that those might apply to other people in your organization, you can then work on those issues. For my users at least, and everybody's environment is different, except for the developers, there's almost no reason any user in my building needs admin.

Even Leopard, you needed to be an admin in order to add printers. Well, if you reconfigure CUPS, you can fix that. There's stuff you can do to work around some of those issues so that there's really no real reason to give the end user. In my environment, there's really no reason except for the developers to give them admin. But you're right.

Your environment's different. You need to know what they need to do and how they need to do it. But don't jump to, I'm just going to give everybody admin because that's easier, because it's really going to cause you problems further down the road, in my opinion. And I'm just curious, how many people edit the Etsy suitor's file? Okay, so in essence, you're taking away an administrator's ability to do sudo, right? No. No. Or adding more. Yes. It's usually adding.

Yeah, Apple could help on that. Just a quick follow-up to Joel's point. I mean, I'm running a deployment where it's exactly what you said. You know, I consider it kind of a pilot project. It's an entire lab. They know the admin password, but they're running it as non-admin.

And my concern is that it's a human factor, so I don't, you know, we can just pass it off. But at the point where they're prompted for the password and they just automatically enter it without thinking about it, what did we really gain? So I consider it a baby step. And it came up in an earlier session that can we get a little more dialogue out of the login window? And it was, I forget which session it was, but it was shot down. It was like, no, that's not going to happen because blah, blah, blah.

But I kind of feel like, you know, sometimes we get the info as to what the authentication is for and sometimes we don't. So I just didn't know if anybody had any comments beyond that. I do feel it's a step in that direction. But I was in the same session. I was sitting in the back and they did.

They just kind of, we're not going to do that, right? Alright, John? John Welch: Okay, yeah, I was going to say if Apple... John Welch: John Welch from the Zimmerman Agency. John Welch: Zimmerman, yes. John Welch: Yeah, I know I tripped you up there, Greg. I'm sorry. I change companies.

I do it every so often just for that. Yeah, Apple could kind of help out with not teaching people how to be admin password monkeys by maybe going back to adding printers without a password and that kind of stuff. Sometimes there's a lot of that, and I understand about reconfiguring cups. John Welch: Everyone that likes that, there's a person that hates that. John Welch: Well, I... John Welch: And that came specifically because there was a large amount of complaining from the educational environment that non-admins could add printers.

Yeah, you can manage that. Now, the complete X, I'm as angry as you are about it because it wasn't completely finished. And that's because there are too many different groups involved. So you can easily allow a non-admin user to add printers and leprechauns. Little chunk of text into etsy.cups.com.

It shouldn't be as hard as that. It should be a better, and that's something that there are a lot of feedback was given on that. And I think that should be changed. But there's no black and white on this. No, no, there's not. I agree. No, the thing, the comment I was going to say is, and this is something, it's not as bad these days as it was a few years ago.

But I still see a lot of comments on the mailing list that show that a lot of Mac admins are doing what I call, you know, admitting by running around with a Firewire drive. And the thing that saved my sanity over the years in the many and bizarre environments I've worked in is being relentlessly lazy about stuff. And your comment about, you know, not waiting for the user.

And it's like, I didn't write a massive SNMP article because I was bored that day. It's because that saved my bacon over the years. When they say their server's done, I'm like, yeah, I knew about 20 minutes ago. Leave me alone. And I see people who, you know, think that somehow if they do everything manually that, you know, they're working harder, people appreciate them more. And it's like, you're not doing stuff you should be doing. If you're running around with a Firewire drive and tweaking everybody's desktop settings or doing that kind of stuff all the time instead of embracing things like SSH.

And ARD and other methodologies, you're really killing yourself on the ability to look at the bigger pictures of infrastructure improvements and that kind of thing that do take real time that you can't just do 15 minutes here and 15 minutes there. And I actually don't think there's much of an excuse for that anymore.

You could say maybe five years ago Apple didn't have documentation or good tools in place to let you do this sort of stuff well. And the people who did sort of built home-built systems. But, you know, you can pretty much follow the standard Apple way as far as imaging and deployment goes.

And it's a good way of doing things. I also would request from Apple that you guys at some point put a more detailed guide on how to do installers. Because while I don't mind beating people with that stick, it's kind of fun sometimes. There isn't a lot of things that talk... That's excellent feedback. We'll take that into consideration. You don't even work for Apple, Nigel. You don't even work for Apple.

Shouldn't you be under the waterfall of money with your hat out? So I actually think installers and I mean I'm sure this is feedback a lot of people have given at various times, developers and sysadmins alike, but I think package management is still one of the main weak points of the Mac platform. Yes, without a doubt.

Well, in Puppet, in what you and Jeff talked about, is a nice alternative. Well, we're still using the package format. So I sort of more mean that there's limitations on the package format itself. It seems, if any of you hang out on the installer dev mailing list, it's, you know, there's a lot of bugs in there, and it doesn't seem to be as flexible as people want. We can't do rollback.

There's, just compared to most other major operating systems package format, it's pretty weak. Yeah. But some guidelines, and they don't have to be like, you know, formal anything, but because I know sometimes getting software developers to realize you do need to talk to IT people, yes, it's scary, and we're not always nice. Do you have any company in mind, perhaps? Adobe. Adobe. Over the years, Adobe, there's been quite a few. I don't want to Adobe necessarily name anyone Adobe, but, you know. Thank you, Mr. Nealon. There you go. All right.

Yes. Neil Clennon, University of Michigan. My question is about performing major upgrades to the OS on the servers. In the past, the recommendation we've been given from Apple is to do clean installs and then import, and the problem in the past has been with getting passwords over. I attended one of the sessions where there was some talk about how to do that now, where there are ways to get the password. There's been some talk about how to do that.

So, I'm wondering if the suggestion from Apple is tending more towards doing upgrades or migrations or things that are moving away from the clean install option. I was also at a lab at one point where I was told, well, there's a service you want to implement, and maybe you're going to want to wait until version 10.6 is out because it's changing everything. And once again, I was hearing it's going to be a major pain in the butt to migrate all of that over, and there's not necessarily going to be a tool to make that work well.

So, I'm wondering which way Apple is headed on that. So, let's qualify your question before I turn it over to these guys. When we talk about a migration, we know the user data is either stored on a RAID or someplace else. It's very likely not stored on the same drive. In some cases, maybe.

Yes. Not the same drive as the OS, right? The server OS. So, what is absolutely critical on that server OS that you need? It's the LDAP database, the KDC user principles and service principles, and the password server database, right? Right, and data. What data? Data that the users create? Yes. That shouldn't be on that same drive.

Right? Xserves come with three drives. Oh, oh, I see what you're saying. You know what I mean? Yeah. Right? So what else is critical if you're migrating from box A to box B? What else is there? Justin? - Basically the data you can't easily restore or reinstall. You have to have backups of that.

When you're managing client machines, AFP548.com came up with a great methodology called Instant DMG, and the idea being that you modularize what you're building. You start with Apple's install disks, and then you layer packages on top of that to get to your final result for your client machines. You can use other tools to get to the same thing on OS X client.

I use a combination of tools, which include RadMine, sorry to say. But basically, I've modularized things so I can take a base OS and then I can layer my stuff on top of it, and now I have a machine that behaves the way I want to do it.

If you're managing a large number of OS X servers, I think you need to be thinking that same direction. How can I document, encapsulate, The bits that make this configuration different from what Apple has dumped on the disk when I run the install disks, and that doesn't just help you from a migration. That helps you from a disaster recovery. It helps you with replication. It helps you with load balancing, any of those sort of things, bringing up a new remote site.

I'm not saying it's an easy answer, but if we're talking best practices, that's really where you need to be going so that you are keeping separate the data, the OS, and then your configuration changes that make the server useful to you. And then, you know, the easy thing is to not pay attention to any of that and just do an upgrade install and hope like heck that that works. But obviously that's not the recommended approach from Apple and that's never the way I've done it because for whatever reason, I think we all find that when we do a clean, fresh install, stuff that was broke before just suddenly starts working again.

And that's because there's configuration changes that were made during the time that, you know, during the time that the server was alive that maybe you forgot about or didn't know about that are suboptimal. And you don't really want to bring those over to the new server. A lot of times Apple has done some tuning. They've done some tuning that requires, you know, different base settings. So from -- So just one thing along before we get too far off track.

My kind of approach is that if you've got a server and you can't -- you're backing it up because everyone backs up their servers. And if you can't get from your backups to your services being up and running, why are you backing up? And so the same approach really holds true for going from one OS version to another. You should be able to get from, you know, your LDAP backups.

To an LDAP server running. From your KDC backups to a KDC running. Those three components, VOS10 server, password server KDC, and LDAP, you need to be able to get from your backups to those services up and running. And it's the same approach holds true for a new version or the same version. Well, now in the past, the ability to bring the passwords over with the entire description that was given in the documentation about how to bring you over to the next server, it says passwords aren't coming with it.

And that's where my initial downfall was. Yeah, I mean, the documentation leads you to do a DS export and then a DS import, which -- correct, doesn't -- doesn't get the passwords. We had an article on the site a while back about how to hand take the password server database out of 10.3 box and move it over to 10.4. So there is methodology for doing that. But I agree, it's not as simple and as easy as it should be.

And it's certainly not currently available in the GUI, which is what leads you to do that upgrade in place, at which point you get everything still in there. Yeah, the upgrade in place had gone wrong from 10.3 to 10.4 for us when it turned out it doesn't work from one set of disks but does work from the other set. You may be familiar with that problem. Yeah, I remember that. Yeah.

Okay, thank you. So that being said, you've got the tools from the command line to dump your password server database and bring it back in. So yeah, the archive function, the log is very, very verbose in telling you exactly what that is. And actually, there was another article where you just added to it where you, that was up talking about stopping the mail database, stopping the SQL server, sort of adding that information as well.

So I don't think it's unreasonable to ask for Apple for a server migration assistant that would help you move that data over rather than all of these techniques that, you know, us grizzled systems administrators have had to discover for ourselves. You know, whether that be an actual nice GUI tool or just a very well-documented set of procedures you can do from the command line. I personally don't care either way, but it would be nice for Apple to document that and explain that to people. So would it be -- I agree.

So would it be fair then to say that the recommendation from Apple at this point is to do the clean install but that there are more tools that have already been provided and perhaps more on the way to make sure that all of the data, including passwords and everything, come over cleanly? Yeah, yeah.

And that's why I'm such a stickler about the word data, because we need to segment it out, right? The configuration files for the services, the databases that we need, including LDAP, KDC, Password Server, MySQL, Mail Store Database, things like that. If we can segment that out, that's doing what Greg said, that documentation process.

Wait a minute, I tweaked this one launch the item, right? Because I didn't want something else to run on demand or I wanted it to wait more than 300 seconds. So documenting all those little tiny bits that you've changed is going to make that my goal. I mean, I think that's the key to the integration, just it goes so much easier. Thank you. Thank you. Justin? Go ahead. Actually, I think David might have been before me. We're going to let Dave stay in for a little bit. Oh, OK.

Sorry. Fair choice, I guess. I want to go back to package management very quickly. We've been using Radmine, and we're very happy that we've been there for many years. So I know that it's been bashed on a couple of times, but I don't think that's really fair. Not to pick on Greg. Greg and I are friends. But I understand the pains of using Radmine is that it's got a large learning curve, but it's very powerful because of system rollbacks and things such as that.

Now, it's not always easy to do a system rollback with Radmine, but most of the time, it's very good at doing that. The power there is amazing. OK. We've actually been thinking about possibly moving to other third-party products. But every time we start thinking about that, I personally think that that is such a low-level system operation that it really needs to come from Apple and have a more refined package management system. I mean, Apple knows all of the ancillary files that it's going to take us longer and poking at our SEs and going to WWDC and get secret answers that we then post to the web of how it's solved.

And I'm not saying system rollbacks are easy, but the best person or people to do that would be Apple. They make the OS. So I'm frustrated that-- yes, I know PKG has been out there for a long time. Iceberg is out there for making packages, package installers. If anybody knows about MSIs, I admit I know very little about them, but I talk to my Windows guys because I have to eat lunch with them occasionally.

But they talk about MSIs, and the power of MSIs on Windows is very extensible. Self-repair. I mean, there's a lot of functionality there. Now, Microsoft has had a lead and a head start on this. I understand that. But I think it's now time for Apple to finally get serious about package management for system administration.

So before the Apple people come in, I guess I was feeling really pessimistic about packages until 10.5, and if you actually poke around in the internals for 10.5, you can see I know nothing about what Apple's plans are with packages, but things are changing. We have a database now, there are tools that let you track what files have been installed by what package, you can search. There seems to be groundwork being put in that looks like it's making it better.

And I think all we can do is keep applying pressure as to what we want. Now on to the sales guys. So what I don't understand, and I think this is where I'd say you'd fail in the first place, is why did that package get out if it doesn't work? Could be a package from Apple. I understand that. So you blindly download and install every single update that comes out.

Okay? Yeah, that's one application. And that's specifically, that's fine. That's entirely scripted.

So, specifically with hardware, but if all you care about is hardware, your problem is much, much less than what it started off being. We've got Joel Sputter. Understand, but if all you have left is hardware, your problem is much, much less than what it was before. Joel, package management is a lot more than just what you're talking about.

I mean, if you look at the whole RPM under Red Hat, which, by the way, RPM is under the Mac. Let's separate out two things here. Let's separate out the server side, where package management and what you're talking about is very, very significant. No, I'm talking about for clients.

How often are you updating packages on the client? Daily. Daily. Why are you updating things daily? Because Red Hat comes out with a shitload, pardon me, a boatload of updates. It's Friday. Hey, you were right the first time. The naughty mic. No one go to the naughty mic.

Apple comes out with a patch, you know, a lot. - The small percentage of your boxes, that's the cost of doing business, those vulnerabilities. It's gotta be patched. - Yeah, there's security and defense that gets before that. - Over here? So an SLA is nice for keeping your job. That's great. Okay? I mean, it really is a great thing for that.

But in a former job, I was in the position of supporting scientists with a lot of, you know, each biologist has his own application that hooks up to his own half-million-dollar machine. You know, it's great to be testing. Like I said, it's not the panacea for everything. But if you don't even have that, how many people in here have an SLA? So we've got two-thirds of the room that doesn't have it.

How many people are using it in the right way? How many people have turned that SLA into a testing matrix? There you go. I know, Mark, do you want to speak to that real quick? 30 seconds. 30 seconds. Only 30 seconds? Well, I mean, I agree the SLA is good to keep a job, but we're arguing we need better package management on Macs. Well, actually, I'd like to touch on both those animals, if I could, real quick.

We actually do use the testing matrix, and we actually incorporate it as part of our internal SQA process. I mean, again, Genentech is kind of bizarre in this way. We actually have an internal software quality assurance group, and they test the stuff that goes out into the general enterprise before it actually gets out there. And they come back with an SQA report that shows where things worked, where it did not.

And we basically have to sign off on it before we flip the switch on it, before it gets unleashed, so that pretty much everybody is on the same page. It also gives us an insight as to what sort of things may potentially crop up as it travels through its path.

We do also run the SLAs, and the SLAs are incorporated with our service desk, so that pretty much everyone along the way, when a new application, something that someone wants to install, gets brought in, we have some insight as to whether this thing is going to work out of the box or not. We've also worked very hard on trying to get the users to... Only put on stuff once we've been able to vet it.

Now, does that stop the CS3 people from out there? No, and actually CS3, for example, is one of our banes of existence. But a lot of that is kind of how it's done from Adobe in the first place. It's very, very hard to manage application because, for whatever reason, they decided they want to just kind of do things on their own.

Fair enough. All right. So-- And I'll back off the mic. Yeah, Chris, we have some people up over there. Dave? This almost ties into package management. One of the banes of my existence in using OS X servers facing the internet is that Apple seems totally unable to grasp the concept that open source software is updated more often than every major release of OS X.

and David So you end up with half of your application or half of your binaries being built under user/local and then Apple comes out with an update and you're like, "Oh my god, Clam AV just made it up to within two revs of what's current. Do I need to go back and test this?" My own environment I end up just building the open source software, packaging it and basically doing all the work that Apple should be doing for me. But I'm curious how other environments are handling this. So the work that Apple is doing for you is why it's two revs behind? Then Apple needs to learn to work faster.

Give me one engineer and two interns and I'll keep it up to date. But you as the admin have the ability and you understand both sides. If you wanted somebody to QA and all that, you'd pay a lot more and you'd get Solaris. Or Red Hat or Debian. Sure. Or freaking Fedora for free. Get done in that. So there is definitely a trade-off between that.

Okay. Go ahead. I'd like to make one little quick comment on that. I'm actually a free BSD administrator as well as other things, and I work with various packages that are based on other open source packages. And I have to say almost every environment that I've seen where there's a third-party application that's built on top of open source, there's always delay.

So with -- I administer Sophos PeerMessage, which is a mail-filtering, spam-filtering environment. And there are always behind on what releases of free BSDs that they support, et cetera. So I've run into that problem a lot with other applications as well. So I do wish that Apple was able to come up with a way to incorporate some of those changes quicker. But I also acknowledge that, yeah, there's delay because of the vetting that you're doing.

Okay. Alfred. Yeah, I was just interested in knowing how you document your configurations. And you use like Word, Pages, a Wiki, what? Mine's a little more primitive, so maybe we should start with mine and then I'm sure his will be puppet. That's the beauty of a modular methodology like DMG or like using RadMind is the configurations are self-documenting, meaning they're encapsulated in a package or they're encapsulated in a RadMind load set.

And so you don't have to write something down somewhere to go, oh yeah, to make sure that the user can't do this, go in and edit this file and then make sure you save it and then make sure you set the mode to 755. You don't have to do any of that stuff. You just apply the package or you just apply the transcript. And if you're familiar with RadMind, you basically get an audit trail of what's on each machine because each machine has a command file which tells me which.

Packages it gets. And other package management systems do something similar. So you don't have to spend a lot of time documenting. You make your change, you capture it as a package and then you just use that package going forward. Do you use the same for server configurations like IP addresses and everything? I don't, but I would think that would be the right way to do it if you were managing it correctly, which I'm not. Right. I mean, even Apple has a very simple server worksheet that you can print out, you know, and you fill out. All the information just to get people started on documenting that sort of thing.

So for us, this is one of the reasons I actually really love Puppet so much because it's a really simple declarative syntax. And what you're doing is you're describing the state you want the machine to be in. So you're saying, you know, I want this kind of resource to have these attributes. And so it's pretty much self-documenting. Like, we're still kind of working out the best way to document which image revisions at which times had these packages, whether they're delivered by Puppet, whether they're delivered some other way.

But honestly, I don't think the way you document matters anywhere near as much as getting something down. And making, lowering the barrier to keeping documentation up to date. There's something I seem to keep running into in jobs over and over again that it's all well and good to come up with a really robust framework for creating documentation. But if you can't quickly and easily update it when you find a mistake, it may as well not have it at all. Because within a year or two, it'll be so far removed from reality that you may as well not read it.

So, yeah, I think it's a really good way to do that. Can I add something to that right here? I've noticed with our deployment that if the process doesn't document it, then it doesn't get done. And that's why, you know, what Greg was saying about RadMine. So when we make overloads, we try and -- when we create them, we try and make sure that there's nothing in there that's going to throw us off when we look at it in six months.

Real quick, on documentation, I've learned over the years there's kind of two levels. There's the long-term status documentation. One of the reasons I've become such a cheerleader for Nagios over the years is because I can look at every switch router, everything above the clients on my network, and I can tell you firmware revisions, all of it banged that fast. Completely blew away a Cisco guy when we were doing upgrades at Kansas City Life because we were the only site he'd seen in two years that could show him our revs without him running their little bizarro, you know, snarf the network tools.

The other one that I found even more important is that the aha moment documentation. You know, you've been beating your head on something, you finally get it done and in the midst of running around it, you know, or pushing it out to a dozen machines, a hundred machines, a thousand machines, you want to document it but you're busy and anything that's more, that takes more time than about 15 to 20 seconds to get it up there.

You're going to forget to do it. And so in my case, I've become a complete and utter Wiki fanboy on Leopard. It's one of the reasons I jumped on Leopard so quick and I put up with having to manually restore the Wiki pages a lot, getting it stable and all that. But that Wiki having, you know, just that URL right there in my toolbar going, oh, did something to click and it doesn't, it's ugly sometimes, it's not formatted, sometimes it's barely English, but it's there.

00:19:00:00 - 00:19:11:00 Unknown I can just go back later and make it pretty if I have to add in screenshots or whatever. But having something like that where all you have to do is open a link in a browser and you can brain dump into it real quick has saved my bacon over the years so many times setting up stuff like that. At worst, a pad and a piece of paper is better than some complicated process. 00:19:11:00 - 00:19:29:00 Unknown I agree.

So we actually have Google has a really strong culture of peer code review and which is one of the big things Mondrian does that Guido talked about. So with our puppet configs, 00:19:29:00 - 00:19:35:00 Unknown we're actually having to justify why we're making a change to our peers who have to approve it before we check it in.

And I think this is one of the big things that sysadmins can learn from software developers that peer review and code view, however you're creating your packages, however you're doing your configuration, if you can somehow get that out to your peers to review before it gets committed, that's just a much better process. And Greg had some interest.

00:19:35:00 - 00:19:45:00 Unknown Yeah. So Nigel's got a great environment where he's got a bunch of people. It's a big deal. It's a big environment. So he's got a bunch of colleagues that can help him do peer review. I'm sure there are a lot of people here that don't have a lot of colleagues that work on the same stuff that they do, so they don't have those peers.

That's actually my situation. There's really no nobody else doing Mac engineering in my group, so it's hard to to get that review. But I have a solution, and that's I post stuff up on the Internet. You know, I've got to put it on a blog or put it in the Mac enterprise mailing list. And I let my virtual colleagues around the world comment peer review my work. And that's actually one of my justifications for for publishing stuff that I do at Disney out on the Internet is is that that way I can get peer review from from other administrators.

So that's that's an option that's available to everyone. If you don't have local peers, use the peers out on the great cloud. 00:19:45:00 - 00:20:00:00 Unknown And I will say we know that there are a plethora of third-party tools that will help us with that package management. And I just saw one the other day, a web interface that ties in, does policy management, package healing, really, really nice, nice tool. So, in the back.

What tool? What tool? Well, who said it over here? Casper? Casper. There's now a little add-on to it. Yeah. I was shown it. Is it called Wendy? Is that the self-service one? No, the developer showed it to me earlier this week. I think it'll probably be up on Casper's website.

I'll tell you after. I don't want to throw it out because I think they're going to make an announcement about the software if they haven't already, but I don't want to jump in. Is he in the room? He's not here? Oh, he left yesterday. Oh, he left yesterday? Okay. In the back. One of the challenges that we're going to be seeing probably before the end of the year is going to be that we're going to be mandated to be doing full disk encryption on our Mac laptops.

So -- Why would you applaud that? Yeah. I mean, I can understand that you have to do it, but why would you be excited about it? Because of one word, "vile fault." Still, encryption is like a necessary evil. It's not something you're excited about. The main reason that I'm, the big thing that I'm taking a look at is it's going to knock my existing imaging methodologies kind of into a cock-tap because it's going to be an instance where in order to re-image you will have to spend eight hours unencrypting, re-image, and then spend another, well, you can hand it back to the user at that point, but still, you've pretty much taken their laptop away for re-imaging for a full day.

So, I'm wondering what methodology... So, why do you have to decrypt? Like, are you not, you're not running backup clients on the, backup software on the clients? Well, I am for the most part, but... So, their data's somewhere else? Sometimes I don't necessarily, my laptops aren't necessarily always backed up in a timely manner.

So, that kind of seems, so we're kind of facing the same problem, and I think a lot of people who are thinking about full disk encryption have the same issues, and I think really it just sort of exposes... Yeah. ...other areas that we have, other weaknesses we have in our methodologies, and really you should be able to just get to the point where I need to re-image this machine, I wipe it, I re-image it.

Right, and that's where I'd like to get to. Tempo has a nice Windows option for that with their live backup product, which they're going to be bringing for Mac this year, but right now it's not going to be a full, you know, basically bare metal install. On live backup on Windows, basically you can just wipe.

And, you know, put it back... Yeah. ...and put them on there, or they've used some of these self-servicing portals from the various client management vendors. And that way, the only piece that you care about is their specific data. Now, if they're running as a non-admin, it's pretty much entirely within their own home folder. So, now we have a much, much smaller subset of that 160 gig drive that you have to back up and care about.

So, that's where something I know, a couple people here, I think, playing around with CrashPlan, which is also a nice little backup agent that runs through there. They got a shout-out over there somewhere. The ATempo stuff is interesting. We've had a few people... ...people that do this with portable home directories. Who's using portable home directories for this kind of functionality.

All right? And for the most part... So that concept of just using that portable home directory functionality, which is already in there to sync back, who's playing around with Time Machine to do this?

Yeah, it's a tiny, it's an agent. There's an agent, so it doesn't, and Casper's free? No, no, no.

ARD's not free. No, ARD is a couple thousand dollars, and, you know, that's it. System Profiler, SSH, and Shell Scripts are free. Yeah, I know, I don't want to roll my own. Will, as well. So you don't want to roll your own, you don't want to pay, and you want one that works really good. Very good, all right. Good luck with that.

Let us all know when you find it. He did say it works at Los Alamos, so. That's right, this is your money we're spending. It's not like Google's money.

And that's the problem where that gets into, is that if you do do a peer review between different people or different organizations, that what you're reviewing is IP.

In theory, I mean, that's your business. Nigel, how do you run your data centers? And if you did know, you wouldn't tell me. Maybe. But to a point, like, if I'm having issues with AD, I know there's a bunch of other people with the same things. It's hard to find different companies that have that kind of relationship. We do, a few of the Fortune 1000 that I work with have kind of developed their own networks, typically with other customers or other companies that aren't in their same business area. Yeah, just to kind of make it an easier networking for us here.

That's why you can get a random handle on a number of different websites and start going into those forums and kind of start doing some of that there where nobody really knows where people are coming from. And a little more neutral ground. This has been brought up before, but I don't think that it's exceptionally feasible to be able to do in kind of an organized manner for Mac OS X. Real quick, yeah, come on up on that. Go ahead. Yeah, we do do peer review. We belong to an organization called Advertising IT Group in Minneapolis.

And all of the CTOs or system admins get together once every three months. We talk about how we do stuff. And if I have some solution that helps other people, I fully disclose it. All right. So it exists and face-to-face, but it's local. I think Apple can help in some ways.

WWDC is a great place. I see nothing about birds of a feather. It all happens at Dave's, you know, buying each other a beer. So I, and we didn't even end up talking about technology. But that's it. That's another point. So I think that, yes, Apple can help.

I'm happy to give you a forum on the site if you want to get a little meet a friend section. Right. I think that would be a great place to go. I hope I don't. Actually, I need friends with benefits. That's a different site. Look out. That's a different forum. And I'll second that for the other major big conference that happens in January, right? Macworld. I can do that. So, yeah.

On the advertising front, just because I work in advertising, if you're an indie, the 4As has an IT committee that does that kind of stuff. So get on that and talk to people. If you're not an indie, you're part of a major network. Major network partners, if you're in Publicis or Omnicom or whoever, you're different companies, but you're owned by the same entity. So that's a really excellent venue for peer review as well. Okay, good. Thanks. Up here.

Richard Glitzer, University of Utah. I had a question on your feedback on if Apple should provide Delta software updates for new hardware. Do you think that's something Apple could provide? Do you think it's unreasonable? And how do you guys handle it now? This is why we built Insta DMG, so that you could do that kind of on your own. And I'm not speaking with my Apple hat. I don't disagree that that's something that's necessary. But do remember, I don't make the stuff.

So while I'm the only Apple badge person sitting in a chair up here, I have a lot of the same pain and feelings that you do about these things that need to be done. So I'm not arguing that package management and things like that aren't important. What I am saying, though, is that we are working in the community on tools to help do that, to make things simpler. And while it'd be great if Apple creates de-installers, kind of the Deltas between the different updates, these aren't necessarily exceptionally hard concepts.

And since a lot of your environments are very, very custom, and the OS, while it's a big piece, is not the entire piece. Things like Final Cut, if you're in a video production environment. Things like CS3 and stuff like that. I add a lot of complexity to it, where you need a process and a methodology that looks at the whole piece, and not just at the piece that came from an individual vendor.

One thing that would really, really, really help, and it wouldn't involve uninstallers, Delta releases, or anything else, is when, say, 10.5.3 came out, within, say, a couple of weeks, maybe a month, reference disk images so that you could download a disk image with the absolute most current version of the OS as far as 10.5.3, 10.5.1. Any hardware that came out after that, fine, you've got to deal with that separately.

But when, you know, you've got to, you can't download Mac OS X 10.5.3. You have to download some earlier version of 10.5 if you want to get the disk image, or you've got to try to, you know, hopefully figure out, you know, read the tea leaves and the chicken livers when a disk with 10.5.3 or whatever the most current version is is going to show up in a retail store or get one from your rep. You know, Apple regularly releasing, you know, it's IT, we'll pay. We got money. Okay.

You know, that we can. So what would that get you? So. That would mean that every machine, unlike, like, right now, we don't do net boot or net install on that because which image goes to which version of which MacBook? Which image goes to which version of which MacBook Pro? Oh, we just have some brand new Mac Pros. Oh, we can't use the images.

So you're not actually building images then? It's not worth the time. Well, that's your first problem. No, I've, it's actually less time to just hope that whatever. How many machines do you have? What do we got, like, two, three hundred? I really think you're looking at how to build images incorrectly then.

I just don't feel like dealing with the hassle. I'm sorry. Because I can get that from other vendors. I can get reference OS images, reference OS downloads from other vendors. I can hit Microsoft's TechNet site, and within about 60 minutes... I'm still confused as to, so if we gave you a 10.5.2 image, what would you do with that? Then I could make that the new master image for every piece of hardware we bought up to the point of that release.

Well, why not just take the installed DVD that came with your latest hardware? Which piece of hardware? Which piece of hardware is senior enough that I can use it for every other piece of hardware I have? So, it's pretty rare. Like, the end of 10.4 was kind of a special case because of delayed models and the whole forking. But I find it's pretty rare for the latest, newest hardware installer to produce an image that doesn't run on your earlier hardware.

In a couple cases, that has been true. No, you don't install from it, but that's what you use to create your image using Insta DMG or whatever. But it only has to fail once or twice before you develop a very deep-seated inherent lack of trust in that idea. And it's failed on me enough that I don't, you know, it's going to take a while for me to believe it'll work again. So we have about ten minutes left. We're going to open it up for questions now.

In the back, go ahead. Jeremy Reichman from RIT. We've seen a tremendous amount of growth of Mac OS X on our campus. And I spend a lot of my time dealing with the licensing issues surrounding the platform. And I liked when we were doing a show of hands earlier. And I was wondering if we'd get a show of hands of people who were satisfied with Apple's volume licensing and maintenance arrangements versus what they get from other vendors.

All right. People that are satisfied. So do you mean just OS X on its own? Just OS X on its own. Okay. People satisfied with that? Okay. Down. People dissatisfied with it? What about people dissatisfied with the combination of OS X and iLife on new machines and iLife maintenance? Yeah, I mean, really, per-seat licensing is a big issue for us.

Right. And accommodating growth over time, because we're at the point where we're going to have to pre-buy so much in advance to accommodate the growth that we're seeing, because we can't get the same per-unit pricing over time. I think there's a problem in quite a few environments where iLife is so core to how Apple markets the platform, and yet very few businesses find that they can produce a decent business case to buy iLife for all their Macs.

In the Richard. Hang on, just a minute. I had a comment on this. Oh, yeah. Oh, go ahead. Go ahead. Bill, it's your name to mention. We ran into an issue with downgrade rights that Apple would not let us buy Leopard and install Tiger. And that was a big deal, and they wanted to know how many seats that it affects. And we had our account, but we'd like to know if anyone else has that issue. Anybody? Yeah? I think you have some shared pain. Yeah.

Okay, I have a question relating a little bit to the discussion we were talking before with Time Machine. So ZFS is coming out in 10.6. This is actually, I guess, kind of a feature request, but it would be really cool to see some of the stuff that's put into Time Machine apply to ZFS, and particularly cool if that was combined with more robust server features. And I know that... What are you asking for? What is in Time Machine that isn't in ZFS? Nothing.

That's my point. My point is that ZFS and Time Machine, in my mind, seem to fit together pretty well, and it would be nice if... I mean, I'm hoping that that will work together pretty well, especially on the server side. But ZFS won't be on the client. Okay. Yeah. ZFS isn't on the client.

I had heard otherwise, but-- Well, it's not in there easily. It's in there now. It's in there now. The DVD we got has ZFS. We're all under the same NDA, aren't we? It's not booting. Well, if you're on a client machine, and that's all you have. Yeah, you need to boot for it to be practical. I don't think they've said anything about that.

Okay. There's the only server bit, though. In any case, my real interest is actually, I mean, Time Machine offers a lot of possibilities if it was done well for the server. So, I mean, right now you can use Time Machine to backup the server OS, but that's not really that useful.

Time Machine and snapshotting is not an effective server backup tool. No, I understand that. I understand that. But it would be nice if you could apply the Time Machine effects to, say, home directories so that a user can actually benefit from the Time Machine. Well, you can. You can have a network Time Machine volume.

Yeah, but it doesn't work very well. That's a different question. I have it at home. Well, my point is it would be nice if it does work well and if it's actually designed a little bit better instead of generating hundreds of little DMG files. Well, it's always going to be a DMG on the network. All right. Well, I guess I'm asking for it to be a network. And that's actually a good thing because that way you have it. Thank you.

I did not take the mic from you. Sounds like what you would really want is the time machine UI on top of ZFS snapshotting stuff. And that's a good thing to ask Apple for in the future. And that may be the direction they're thinking in the future. If you think that's going to ship in Snow Leopard, I think you're smoking crack. I didn't say it would. I was asking.

and David It's also a third-party opportunity. I mean, putting a GUI on top of a ZFS is Yeah, you want to speak on the mic there? As long as he doesn't steal it from me again. Putting a GUI on top of ZFS is probably not that hard of a thing. I mean, Zpool and all the rest are pretty simple commands.

Sure, but, I mean, Apple's selling a nice polished product for end users with Time Machine. ZFS is not a polished product for end users. I understood that. I understand that. But what I'm saying is people are going to want that feature in IT departments because they have it at home. Sure. And be nice to Tony.

I mean, I'm not disagreeing with you here, but the crux of your problem is that if Time Machine doesn't work for you, Time Machine should be fixed. It's not replacing Time Machine with ZFS or something else like that, but the Time Machine mechanism and those things in there need to work solidly and reliably for you. Yeah, that's essentially what I'm getting at. It seems to me that ZFS would be all right. Don't cloud the water with ZFS. All right. Mark, real quick. Especially since there's mic theft going on now.

Um, On software update, there's a bit of a behavior issue going on with software update right now. One of the things that we thought was going to happen with Leopard was that when you get a software update, say in Tiger Server, it automatically pushes the previous update out of the way and deactivates it and puts it to the side. That's great unless you don't want that to happen.

Say you haven't finished your SQL cycle on QuickTime 7.5, but now 7.5 is suddenly up on software update. This is very personal. This is quite personal. I know that there were some discussions on that. We have two minutes left. I'll try and keep it short and basically in the next rev, be really nice if the software update would be able to just hold it until you as administrator send it back.

I don't disagree with that. Definitely, that's feedback that needs to make it into those sessions and those things like that. I do take issue with people that really try to do a lot more with the software update server than is ever really rational to be done with it. This is just getting a release though, Joel. I just want to gate the release.

I don't want to stop it. I just want to hold it. It surprises me how many people are using the software update server for all of their software updates to their users. If it comes down from Apple, yes. Being that this is a whole technology, you still have to find a solution for all of your other third party technology. Yeah, I got that kind of covered.

Either through Landline or through Casper. So why aren't you running the Apple updates through Casper as well? No, because the previous person who talked about imaging and how they use disks and FireWire drives or whatever, I mean, we have a base image. It's 10.5.1 and then software update gets it to me in about 12 minutes to 10.5.3 or 10.5.2 or whatever. Sure. So that's -- I don't worry about images. I just have a base image and then I have software update do all my work for me and it works great. Sure. But how do you do the third party updates? Casper.

So why don't you do the OS with Casper? Because you -- Trust. You trust the software update server more than Casper that you paid for? No, Casper -- I may be wrong here, but I believe to do a software update of the OS itself in Casper, you need to net boot to do it. I don't think you can do it.

For me, also, it comes down to the software. I don't know if you can do it with the software. I don't know if you can do it with the software. For me, also, it comes down to the fact that when users run software update, they get a UI, they get feedback, they get this is being downloaded, you need to log out now, you need to restart. Yep. I would really love it and have been hassling Apple about this for ages if we could trigger the same user experience with software update from the command line. Yep. So we could go download and install. All right.

With that, we do have to, time is up. Oh, Schoun, I had one last, one quickie. This is in regards to people asking about what, how can you manage to make sure you have an OS release, that one that will work going backwards? What we found over the years at Genentech in our imaging process, best rule of thumb is go to the top of the line hardware. Basically, if you, for example, if you image for PowerBooks or for MacBooks, you do your imaging on the latest, hottest, sexiest MacBook Pro. Yep.

If you're doing desktops, you do the Mac Pro. Basically, MacBook Pro, Mac Pro, and actually, in most cases, if you do a MacBook 17, you cover everything. Yep, and it means your imaging guy doesn't resent doing imaging because he gets all the sexy hardware. All right. Yeah, that's true. Thank you very much. Hope you had a great WWDC.