Hi everybody, I'm Dave Zarzycki and I'm going to be talking to you about launchd today. And although the title may say managing processes with launchd, it's really about helping you get your processes run when you need them and when you want them. But first, and for the first third of this talk, I want to talk big.

I want to talk about Apple and I want to talk about the goals we have for ourselves and how they permeate the system and go all the way down to a low level technology like launchd and how they permeate back to everyone else at the company and or third parties to make really great products for all our customers. So these are three fundamentally different products, but they're all derived from the same core Unix operating system. But only one of them, the laptop, has a terminal. The other two, no terminal.

In fact, one of these doesn't even have a shell, the interpreter. What does this mean? Well, when something goes wrong on your laptop, you can get a smart friend, maybe a system administrator, they can come poke around, maybe fix it. The other two, they're on their own. So we need a system that's reliable, can respond to events when especially when things don't go right, or when unanticipated events happen and when expected events happen. So let's talk about these goals more in depth.

What are our goals? What are our goals at Apple? And what are our goals that we hope that maybe you guys can adopt too? Well, unlike mainframes and computers of bygone eras, we don't want to restart applications or computers after you change a configuration detail. Use... Users expect us these days. They expect things to just work, and can you blame them? We also want the system to be more dynamic.

We have hot-- I like the airport express this way. It's a hot plug hardware, and it's a wireless device. These are low level technology, like a device as far as iTunes is concerned, and it's wireless networking, which we're all very familiar with these days. Finally, it's user friendly. This is really what we're trying to go for in a nutshell.

This is a slide that I'd really like to drive home to. Today's multi-core CPUs that Intel and other vendors are providing are pretty amazing. We've got quad core, eight core. Just a couple of years ago, we were single core. We need to figure out ways, collectively, how to extract the power that this modern hardware provides and take advantage of it across all layers of the system.

Now, this is one of the web pages on store.apple.com for one of our machines. The details aren't important, except for one little part. That memory, 512 megabytes, might seem like a lot, but, you know, when you have a couple thousand developers at Apple and tens of thousands of third-party developers all trying to share that memory, we all get a small piece of the pie.

And what we do at Apple, and we hope you can try to do too to help everyone out, is use as little as possible, and if you do consume a lot to try and solve a problem, let it go as quickly as possible, so that way you can let some other part of the system or some other application use it. So, what we've done with launchd helps us even more, and we'll get into that later.

Now, given these goals, of course, there's going to be obstacles. What are these obstacles? Well, straight back to where we started, life is dynamic. That is an obstacle. Back in those past, when we have those mainframes, well, we could make assumptions. We could assume that the power wasn't going to go out, that the machine wasn't going to go to sleep, network wasn't going to change. That really made life easy for us programmers because we could get away with a lot.

Can't do that as much these days. And what we do inside of Apple, and we hope to help work with you to do the same, is to push everybody to be more dynamic. You know, things will change on you, and you should expect it. When you do expect things to go wrong or things to go different, your software tends to behave better and your customers are happier.

But some more on the way of design obstacles. Back in the old day, again, kind of making the assumptions, you know, things happen very linearly. I was going to do A, then I was going to do B, and then C. And that limited concurrency, because you had to accomplish things linearly. And that was unfortunate. But it was easier to program that way.

So no one's going to blame you. But these days, we have a lot of event-driven APIs. And what we do is we encourage, again, Apple people and outside of Apple to adopt these APIs. And respond to events as they happen. And now we can do things out of order, and possibly even in parallel.

But more to the point, when you adopt these event-driven APIs, you increase the flexibility of your software. And the more flexibility you have, the more resilient you are to unanticipated events happening and not having your software crumble under the bizarreness of the system or the environment. When things go wrong on other systems, and sometimes our system, something like this happens.

You get a blue screen of death, and that's because things fell apart so bad we had to grind to a halt and reboot. And what we'd like to do, and what we encourage, again, everybody to do, is adopt ways to avoid this problem. And most of it's through error recovery.

Now, how did Apple accomplish our design goals? Well, launchd is one part of it, but it's a big part. Launchd manages processes, and when processes, they occasionally fail. And when they fail, what launchd does is it restarts them. In that sense, launchd is a lifeguard. Watches out, something goes wrong, it'll save you, get you restarted again. But also, given that there's a bunch of processes, launchd is also a telephone operator, and it helps these processes find each other. This is really all launchd does, and we'll go into this in detail quite quickly.

First, let me say that launchd is very popular at Apple. There are 164 launchd jobs in Mac OS X just provided by Apple. And this is just Mac OS X client. There are even more on Mac OS X server, and there are unnamed ones on some of the other embedded devices. In fact, we don't even have startup items anymore.

The fact that launchd has been so successful. From airport to x-ray, many diverse and different technologies that are from low-level airport to high-level x-ray with developers, they're using launchd to accomplish their design goals. So I'm glad that for those of you that are here, you're going to find out why.

Well, first of all, fault isolation and recovery. launchd, at a very simple level, launches a process, keeps it running. If it crashes or if it exits mysteriously, launchd will restart it. That's the simplest usage model, but that's powerful. In the previous system, we'd just launch a process and forget about it and hope that things went all right. It's a simple change, but it makes a world of difference.

Launchd, going with the telephone operator analogy, allows for fully asynchronous bootstrap. We can really help processes find each other, even though, and coordinate things in a much more parallel way that feeds back into helping us solve the multi-core problem which we talked about earlier. It also provides pay-as-you-go computing. We try, again, in the low memory scenarios and performance goals that our customers love, try and help our developers inside and outside the company only run software when they need it. And being event driven helps that too.

And finally, security. Launchd provides, and I know this may sound crazy, just a clean environment for your process when it starts. You don't have to worry about pollution by maybe the user set up some strange environment variable that may change the behavior of a library you're using, or all of a sudden a user may launch a program in a working directory that you didn't expect. Nope, same environment from the first time to the last time. It's always clean. Also, in a mechanism that Apple primarily uses, but you can use too, launchd provides privilege separation, which going combined with a clean environment can allow you to accomplish privilege operations in a secure manner.

So let's go into depth. Let's go into those bullets that you all love. Launchd, first of all, is open source. We at Apple think this is so important that we've open sourced this with a license that is the Apache license. So that organizations, such as the BSD projects, are more likely to include and find all around more palatable.

We'd like to work with other OS vendors, such as the BSDs and the Lennoxes, or anybody else, for that matter, who wants to adopt it. We have a URL and a blog and a mailing list and actually all that good stuff on launchd.macosforge.org. In fact, you can see the source live as we update it here at Apple.

Now, what are we going to cover today as far as the details? Well, for you developers in the audience, we'll cover where your code fits in. There's a big system. There's lots of different places it can go. How does it get involved? When is it going to be run? How do we resolve dependencies? You know, we still need to rely on things. We may be making less assumptions, but we still need things.

And if we're the backend of something, how do we check in? How do we get things going? If you're a system administrator, we'll show you some of the things that you can do, and we'll give a demo later showing some of the commands you can do to interact with the system and find out how things are working.

And we'll talk about why launchd is good for you. So fitting into Mac OS X. We have some three basic layers now in Leopard. This is different than Tiger. In Tiger, we just had the per machine layer, and the per session layer was, well, less defined. Now it's much more concrete.

You're going to find programs like Safari, iTunes, System UI Server, running in this collective layer. And to be completely forthwith, I may use different words to describe this layer. I might say bucket. I may say session. I may say context. I mean all the same thing. We're still struggling with terminology for describing these containers.

We also have a per machine layer, and this layer contains things like disk arbitrationd, or kextd, or configd, if you've ever done a PS. But what we've added that is truly new in Leopard is a per user layer. The per user layer is meant for programs like these outlined here, like the GSSD or the C cache agent. You're not necessarily, I'm not going to go into what these exactly do, but it's a new layer for your software can run and provide services on behalf of the user.

So what's the relationship between these layers? Well, what we'd like to encourage, given these programs need to talk to each other, they don't exist in a vacuum, is peer-to-peer or downward interprocess communication. We have these three buckets, as we outlined before, and we have programs. Here's some nicknames we give for them. We have applications, we have agents, we have daemons. These are the nicknames we try and stick to, but we're not always consistent about.

And what we'd like to encourage is that, well, talk to the friends and their direct peers. And it can go either way. One app can talk to the other, or vice versa, or one agent can talk to the other, or vice versa. It's all cool. But what we'd also like to encourage is downward communication. Applications can talk to helper agents or helper daemons, or agents can talk to helper daemons.

On the flip side, why only die with Nord IPC? What's the problem? Well, we have this machine-wide layer. This is where our programs live that help the machine out. And oh wait, we can have more than one user on the system, and they both could be running the same program. In fact, there's a whole bunch of layers on the system.

A user can have a GUI session, they can have command line sessions, or yet future sessions. When we encourage downward only IPC, we encourage well-layered designs. It's unambiguous what the helper process is at that point. A GUI session only has a user session, and it only has one machine that it's running on.

On the flip side, the machine has multiple users, multiple sessions, and at that point, you'd have to have some policy, and policy is the right word, for which user you talk to. And that gets really messy, and it's something the computer shouldn't be trying to do, you know, at that low of a level.

So what is the purpose of these containers? We've been kind of alluding to them. Let's go in depth. The per session layer is for programs that need to interact with the user. That's the closest we can come to a session definition. They come and go with the session. You want that finder to exit when you log out.

The per user bucket, or session, or context, is for programs that don't need to interact with the user, but still want to run on the user's behalf. They are allowed to transcend login and logout, and they're pretty much the lifetime of the system once they get launched. Dave Zarzycki And finally, we have the per machine layer.

We'd really like to encourage the minimal amount of code that is required to arbitrate or share hardware to live at this level. And we do this because of the security risks. Obviously, it's working on behalf of the machine, but what it also is doing is operating on behalf of all the users. Dave Zarzycki And if you get a security bug, maybe a buffer overflow, or some kind of other exploit, you put all the users at risk. Whereas if the per user bucket is a security bug, then you're operating on behalf of all the users.

And if the per user program were just up one more layer at the per user layer, the damage would be limited to that user. So if you're not arbitrating the hardware or helping share it between users, hopefully you can consider moving up a layer. Well, getting started, let's go down into the meat more of this presentation.

Let's install a property list, Apple's favorite data description language, in the right location. There's a whole bunch of standard locations on Mac OS X where you can install a property list. First and foremost, that probably most of you that may be shipping software to a customer, is library launchdamons. Library launchdamons are per-machine jobs that are installed by the system administrator, and they are loaded up for the machine for that system. Now, Apple provides some, obviously, but we have a directory of our own, and that's /system/library/launchd. That's per machine jobs provided by Apple.

Now let's do four more. They're called agents. These can either be the per user or per session jobs. We have a separate directory for those. Firstly, we go look in the home directory for the user. Maybe they want a program only run for them when they're logged into a machine somewhere. Well, they can do that by putting a property list in their home directory in library and launch agents. These are loaded in any of their sessions when they log in, and the user can limit that through a key we'll describe later.

Now, the system administrator can also load jobs into all the user sessions and on a network. They can do that through /network/library/launchagents. And any machine that mounts that, launchd will find that and it will get some jobs loaded up for every user on every machine on the network. Sometimes you don't want such a wide net. You can use /librarylaunchagents to get processes loaded for each user on the system as a system administrator. And finally, the one Apple provides, /systemlibrarylaunchagents.

Well, what if you actually wanted to create one that worked on Tiger and later? Let's go back to that simple case where launchd starts a process and restarts if it crashes or exits. Well, you're going to need a label. This label is really critical. It's a unique identifier for the job. It's typically a string. It's reverse DNS because that's pretty hot these days. And what it allows us to do, which is really different compared to Unix of old, is we have a name for a process, regardless of when it starts.

So we have a label. We have a label that says, "I'm sorry, this is a new PID. It's going to be a little bit more complicated if it crashes and restarts." The kernel, on the other hand, has a PID, which is ephemeral. Every time it crashes and restarts, it gets a new PID. And if you tried to keep chasing it, you're going to go crazy. So we have a label, and we can use that, and we do use that to keep ourselves sane.

I'm going to read this bullet because that's the best way to say it. It's a pre-tokenized array of strings that corresponds to the second argument of main, including argv0, or the second argument of execv, and the first argument will be inferred if not supplied to launchd via the program key. Dave Zarzycki I can just say, see the man pages.

It's a little bit more complicated than it should be, but what it amounts to is the name of your program is actually an argument, and you need to supply that, and we'll use that name to figure out what to launch if you don't tell us what to launch. And finally, the last key. It's just three keys. What to run, a name, and the on-demand key. And when you say on-demand false, we'll keep that program alive, and we'll keep it alive for the life of the system.

This is what it looks like in a plist. It's surrounded by, obviously, your header for a property list with the XML tags. You've got the key labels, and its value is a string. In this case, we have com.example.helloD. And we have the program arguments. In this case, the name is a very explicit path to where it is on the system. We could have just put helloD, and launchd would have found it in user sbin. But this is more explicit and a little more cautious. And finally, onDemandFalse. OnDemandFalse tells launchd, start it, keep it alive forever.

All right. So this schema, I've mentioned three keys. There's actually a lot, lot more. And the man page is the best place to go if you want to see them. But to give some brief examples, the working directory. You're probably used to that in your shell. You can say CD here, CD there. Launchd lets you set that in a property list. Now every time launchd starts that job, it'll be started with that working directory.

Environment variables. If you are at all familiar with some of our malloc debugging library facilities, you will know that the malloc behavior can be changed via malloc environment variables. Maybe you want to do that for your job. Maybe you're having difficulty with it. Well, you can add it to the property list, and now you know every time that program is started, be it the first boot or some crash and restart, the environment variables will be the same.

You can set the program. This goes back to that program arguments key and its relationship. This allows you to skip the path walk of maybe bin or user S bin or whatever and directly execute a program. Username, this only applies to daemons since they run as root by default. Maybe you don't want a particular program that's a daemon to run as root. You can pick a user to run it as.

A really basic example is the start interval and start calendar interval. Start interval, it's pure. If you say 137, we're going to start that program every 137 seconds. It's a little bit different than cron that way, because cron, well, doesn't let you do that level of granularity. The best you could hope for is once a minute on the minute.

But should you prefer a calendar interval, maybe every Thursday at 3:00 PM, you can use start calendar interval. And to talk about some improvements of start calendar interval for Leopard, we now support more of them. So if you wanted to support multiple calendar intervals, you can do that.

Dave Zarzycki And secondarily, we fixed a popular bug to report that if a calendar interval is missed while the machine is asleep, we will now quickly notice that, or hopefully quickly notice that when the machine wakes up and run jobs that should have been run while the machine was asleep.

Finally, an example-- yeah. Sorry about that. Finally, an example, now that we have the whole concept of per user and per session for context or sessions or buckets, we have limit load to session type. Popular examples are background, Aqua, login window. We have a couple more, but these are the ones to mention up front. Background is the per user context, and Aqua is the graphical login context. And login window is what you see when you boot up to a login prompt.

And the reason for this is sometimes not every program you want run in, let's say, the Finder while you're logged into the GUI. You want run while you're at the login window. But some you do want run in both, like the bezel UI. Whenever you adjust the volume or the brightness, that happens to launch a program on demand in the background. And we want that program loaded into both the Aqua and login window context.

Now dependencies, a favorite topic for people, and I can't blame them, because we all have needs that we need to sort out. We looked at things a little bit differently in Apple. We decided that programs, programmers work with contracts. We need to ask what those pro, uh, contracts were. Well, they define expectations and promises on both sides. In fact, there are two kinds.

We have programmatic, which is every API you've ever used, and you kind of not, you don't think about it. But one that we sometimes forget is data driven. And what we've talked about thus far with all the property lists is the fact that we're using a data driven API.

And it's a very nice API, and it's an API that hopefully you all can appreciate. But I do know the system administrators appreciate it, because now they can write tools to introspect all the programs that run and ask questions about them, or maybe change their behavior in a consistent way.

And it makes it just much easier to interrogate and change the system. And it provides for consistent behavior. I can't say any programmer who would rather see the same recipe of code copied and pasted between applications, when they could defer to some other program to implement it for them.

Well, let's say you're writing a program. How are you going to solve your dependencies? Well, we're going to start a trend here. For network changes, we, Apple, recommend internally and both externally that programmers use the system configuration framework to monitor for network events. Remember those laptops. They're coming and going from networks. If you want to write an application that your customers will appreciate, that's the way to go.

Dave Zarzycki Now, you might say, "Hey, I'm writing a server product. That server's not moving, and its network's not changing." And what we would like to suggest is, "Well, okay, fine. It may not move, but if you're not responding to network events, it makes it very difficult for system administrators to reconfigure their network in a way that's transparent to their users." Dave Zarzycki And as we watch more and more server administrators maintain lots of servers, they want to be able to centrally administer them.

Dave Zarzycki Well, we're going to start a trend here. For network changes, we, Apple, recommend internally and both externally that programmers use the system configuration framework to monitor for network events. Remember those laptops. They're coming and going from networks. If you're not responding to network events, they might not be able to centrally administer them.

Dave Zarzycki And as we watch more and more server administrators maintain lots of servers, they're using things like DHCP to assign IP addresses to them. And it's not surprising these days to see system administrators reconfigure the network and like to see those updates happen on the fly. If you're writing a server, we'd really appreciate it if you, too, would consider using the system configuration framework.

Dave Zarzycki Let's say you've got disk changes. Well, the disk arbitration framework is the way to go. And this, again, both client and server, this permeates everything. We have FireWire and USB. We have Fiber Channel. It allows for hot plug devices. Dave Zarzycki If you happen to be in the previous session, we demonstrated basic Unix tools like I/O stab, discovering when devices come and go. Dave Zarzycki It's very important that everyone pay attention when disks come and go to provide a user experience. Dave Zarzycki And a server administrator can do that. Dave Zarzycki And a server administrator can do that. Dave Zarzycki And a server administrator can do that.

It's a server administrator experience that makes everyone happy. Dave Zarzycki Finally, for device discovery, we recommend the I/O Kit framework. Dave Zarzycki We're really dynamic these days between USB and FireWire and CardBus was the old technology. Dave Zarzycki And I believe ExpressCard is the new plug-in slot for laptops. Dave Zarzycki Their devices come in going like crazy. Dave Zarzycki And if you use the I/O Kit framework, you can find out about that.

Dave Zarzycki Other changes are going to require other frameworks. Dave Zarzycki Well, why are we driving home the framework message? Dave Zarzycki In the old days, with System Starter and other similar mechanisms, programmers, and this is just the way we all collectively did things, declared which processes they needed. Dave Zarzycki But what we noticed in Mac OS X is the processes were actually an implementation detail of a framework. Dave Zarzycki And they were already linking against the framework, so that was the dependency declaration.

Dave Zarzycki It was automatic. Dave Zarzycki And what we did is we provided a mechanism for the frameworks to find the processes they need. Dave Zarzycki And should that process go away in the future or a new one comes along, Dave Zarzycki and that new one be added to a framework that didn't have one before, it won't affect the application writers because they're already declaring their dependencies by what frameworks they link against.

So this is a list of some common examples. Some of you may have seen them. Just link against the frameworks. You'll be a lot happier. Well, let's say you're writing a framework. What do you got to do? Well, I'm going to pick a case example. It goes all the way back to the '70s. Simple input, simple output, takes a login, returns a structure. This is getpwnam. It looks up an account.

It previously read a file directly, because that was just the way we did things back then. But later in the Mac OS X evolution, we changed it to talk to a program called Directory Services. But how did we avoid a race condition? Because once that API is called, we need that process to be running, don't we? Well, maybe not.

And the way we did that is by telling LaunchD about directory services and the fact that it vends inter-process communication. This is the change we made to the property list. And this goes all the way back to what I said LaunchD does. It watches out for processes and restarts them, but it's also a telephone operator.

And what this allows us to do-- and if I may stretch the telephone operator analogy-- this is like getting an entry in a phone book that you can then look up and start dialing. But the person doesn't-- or the process-- doesn't need to be home yet and ready to answer.

But you're not going to get a no such number problem when you make the phone call. And what this allows us to do is actually launch directory services on demand, because we know that a process is trying to talk to it. And we can keep that process-- again, stretching the analogy-- ringing until directory services, the process, comes online and picks up the phone and starts answering. This is very powerful. This is exactly what allows us to have a very asynchronous boot up and to be able to launch things in a pay-as-you-go manner.

So the property list. What are we promising you guys? Again, API contracts. We need to make promises to you. Well, if you use IPC, you're writing a framework, you're writing the backend to a framework, we're going to have race-free IPC handle setup. And how do we do that? Well, we read all of the property lists before we run any program. We're populating that directory of possible IPC channels. This is true for per session.

Wait a sec. What are we not promising you? What's the other end of the equation? Well, no network interfaces have been found or configured. No auxiliary file systems have been mounted. Not all hardware has been discovered yet. In fact, we're basically promising that any detail about the configuration beyond IPC is done. But if you're using the frameworks as an application, you'll be OK. And this applies to everyone.

We would really appreciate it, both inside of Apple and outside of Apple, if you just collectively, if we could all make fewer assumptions about the state of the system and use the callbacks from the framework. All right. Let's say you're writing a backend to a framework. What do you got to do? You're going to We have some simple APIs. They're boxed objects in an object-oriented terminology. But they're really, really simple. And their whole purpose is inter-process communication with launchd. Here's a simple example. We have a boxing of a string, in this case, hello world. It returns temp. And we can unbox it with the basic getter.

Well, we've been talking about IPC for a while. How does that all work? Well, you've got to get the IPC handles from launchd before you can start talking to your customers. How do we do that? Well, today, you create a new string with a known constant called checkin, and you send that via one API. It allows you to send an object graph to launchd and get an object graph back.

If it returns null, there was some kind of problem in the IPC at a very low level, it's an error. There is also secondary failures that need to be checked. You might be able to talk to launchd, but the request that was made was somehow interpreted and possibly rejected by launchd. And the way we do this is by checking the type of the response via the get type API. If it's error no, we can extract the error no and then print out why launchd rejected the request.

Otherwise, things are probably moving along. What do we need to do next if we're writing a backend? Well, we need to iterate the results. : This is an advisory idle timeout to help consume less resources. You obviously don't have to pay attention to it. And if you're writing a program that's busy, you don't obviously have to exit. You're busy. But we'd like to see everybody inside and outside idle exit.

And this is a hint about when to do it. You can also iterate the Mach services that were provided to your job and define in the property list. You can iterate the sockets. And you can do this by first extracting the well-known keys using a dictionary lookup. And then you can iterate the results of that dictionary fetch via the dict iterate call. And the next slide will demonstrate what that callback looks like.

It's real simple. Key, value, a cookie, or a reference constant, whatever you would like to call it. that'll help you iterate the results. Now if you're a system administrator or just a programmer trying to figure out what's going on, you can use the launchd tl command to talk to launchd from the shell.

It has many available subcommands. There are a few listed here. We have list. It'll list all the programs loaded into launchd. And in fact, it can pass the label, again, the unique identifier, as an argument to list, and that'll tell you more information about the job loaded into launchd. You can also load and unload a property list. And those commands also allow you to tweak whether or not it's disabled on a permanent basis.

And should you need to debug what's going on in the system, you can say log space level space debug. And that'll crank up the output of launchd. You will need to, of course, adjust your syslog configuration to actually have those messages go somewhere, but that's covered in another talk.

Dave Zarzycki You can also manually kickstart a job. You can stop a job. And all that does is send a SIGTERM. But again, going back to the label analogy, launchd keeps track of the current PID, and it will send it to the correct process. So you don't need to worry about any race between your typing of PS, and if it dies and restarts and gets a new PID. Finally, we have export and help. Export extracts all the environment variables out of launchd that can be useful for debugging. And help will list more subcommands and how to use them.

So what's new in Leopard? Well, we've added Mach IPC support. This may not affect most of you, but it affects a lot of what Apple's implemented. If should you choose to use it, it's real simple. You can add the dictionary as we iterated before. We've also added agents. That's both the per user and per session. If you look on Leopard right now, just in the system library launch agents, you'll see several dozen adoptions of that technology to provide many services on Mac OS X.

We've also-- and this is, I think, kind of fun-- At a conditional keepalive support, and this is logically ORed. So what does that mean? Instead of saying keepalive true, like keep this program running all the time, you can make it conditional. The first example, successful exit. Let's say you want to run a program, and as long as it fails, you want launchd to restart it.

But the minute, or the first time it actually succeeds, you want launchd to give up and just walk away. Well, you can do that now. You can say successful exit, false. That's what launchd does. You could actually do the inverse, too. Maybe you want the program running as long as it exits zero, which is traditionally considered success. Well, just set it to true.

Now, doing the logical OR, the network state can be paid attention to. Network state's ambiguous. It just means that we plausibly might be able to talk to another machine. No guarantees. But if you feel that your software is not very useful unless that is true, you can express that now in your property list.

And then when the machine boots up, or Whatever the case may be, launchd won't start the process until we sniff out a network. Again, oring in the next possible state, we have pass state. There's already several projects at Apple that have adopted this to only run their programs when they're needed. And a popular example is a configuration file. So what we can do is say, well, this program isn't needed unless the user's actually configured it. Well, a great way to express that is say, keep this program alive as long as the configuration file exists.

And this is how we express it. You say pass state, it's a dictionary, and then you can have as many paths as you want to monitor. Again, they're all logically ORed. You can do the existence of a path, you can do the non-existence of a path, whatever makes more sense for your application. Once again, all these top level keys, successful exit, network state, and pass state are logically ORed. So keep that in mind.

So just to rehash our goals, we would like to see everybody recover for more hours and monitor for more events and Pay attention to when things happen around you, be it the devices you need, or the networks you need, or disks. We have the mechanisms. You should use it.

We have our frameworks. We can write our own. You can use launchd to launch on demand, help save resources for everybody involved. And should you be writing a framework, please use launchd to help your backend and your framework coordinate and find each other in a race-free way. And again, to buy off on the pay-as-you-go model.

So I want to give some demonstrations. If we could switch to the demo machine one, I'd like to show two changes to basic Unix technologies that we've made launch on demand. So SSH has its own public/private key cryptography to simplify authentication. And I'm going to create a key right now.

So we have a key. And it has a program to help cache that key so we don't constantly have to re-authenticate. That's the SSH agent. And normally what a user would have to do is say something like eval ssh agent, and it would say this, and this is how you would start it. Well, that's annoying. Why should I always have to start it? Why can't the computer do this for me? And well, we can do that.

What we did in Mac OS X is we made SSH agent launch on demand. So if we actually look, the agent isn't running. But if we look in launchd with list, and I'm going to grep here to sort it out. We have the SSH launching on demand. And this first column is the PID, and it's not running.

But just by-- if we say echo ssh ossock, we see that it's actually ready to be talked to. So we can type ssh add to add that key we just created. And now, when we do launchctl list, we see that it's PID 160, which if we had looked at the PID of SSHAd, that was probably 159.

And now we can kill 160 and say, oops, it died. Launchd has noticed. This was actually the signal, the exit code. So it exited with two. But we can add it again. There it goes. But one other thing to say about this is this SSH agent, the way it's configured, can be accessed by all GUI apps.

So what that means is things like Xcode, when it runs SVN and SVN tries to use SSH to talk to your server, that SSH can talk to your agent. Because the SSH agent was configured with all the other launchd jobs, and that means that every launchd job can find that SSH agent, which was really complicated and really hard to set up right in previous versions of the operating system.

So another example of launch on demand that we've done in Leopard is we've made X11 launch on demand. What that means is you don't need to, as the user, either double click X11.app or somehow ensure that it's running. You can just say, Xterm. And if you watch the doc, There it goes. It's launched. And again, it's done via the same way. We've modified it just a little bit and told launchd about it with a property list. And now any program that uses the X11 library or framework will automatically get the server run if they need it.

To give more examples of what can be done-- well, now we killed the server and Xterm got cranky. To give more examples of launchd and what can be done either as a system administrator or a developer, let's say list. This is listing all the agents on the system. Any, so a lot of examples are all right here. We have things like Spotlight. That's this menu right here.

Or, What's another good example? The pasteboard for when you do copy and paste. But notice how many programs aren't running. This is what Apple is using to help save resources for all your users. This is that quick look process that you saw demoed in the keynote. If I were to start browsing via the finder and go to quick look mode, it's do launchd to list again.

Well, I need to get it to render something. That was probably a generic icon. But this quick look-- Oh, space. Oh, it's an alias. It's on-- it should have-- had I found a document, actually-- Oh, well. You would have seen Quick Look running. And to give you a better example of how this can be used to debug, Let's see what com.apple.quicklook is configured to run as.

Well, we can see their program is right here. We can see that they've been this Mach service, and that's what would have launched it on demand, had I found a document to render. And it has some advisory exit time out of zero. It's on demand is true, and it only wants to load in GUI sessions, because that's only where it makes sense. Well, you can use other examples too, like, well, what are all the environment variables in launchd? Here they are. Here's that SSH OSOC that makes that possible. This is what makes the launch and demand X11 possible. And here's some other basic environment variables that users expect.

You can also-- let's give an example of unloading something. So you can say, unload system library launchagents com.apple.spotlight. So what this is going to do is unload spotlight from the current running session. And what you're going to notice right up here in the menu is this is going to go away, because that's what's drawing that. But we can load it back.

And I'm going to do something here to make sure it works right. So we're only interested in Aqua jobs. And this is the job we're trying to load. Now it's back. So let's say if you're developing that program, you could then maybe unload it, modify the plist, add an environment variable, and then load it back to get maybe malloc debugging or some other feature enabled. Now what else can we do? Let's see. We can start a job. That's real simple. That just kicks it off. I showed export. Oh, debugging. So let's say you need to debug launchd. What you can do, I'm going to demonstrate syslog a little bit here.

I forgot that we can't do that part. Well, what you would do is you can say launchctl log level debug, and that engages it. You can verify by typing log, and that shows you what's being logged. You can obviously go back to log level notice, which is the default.

But what you can do is crank it up to debug and then tail the system log. And that's pretty verbose when enabled. You'll need to adjust your syslog.conf, but it'll tell you everything you need. If you want a more granular level, you can add debug true in your property list and just get debug output from launchd when your launchd is working on behalf of your particular job. I think that's it for what I want to demo. So if we go back to the slides, we can Go into more information. We have documentation and sample code. You can also go to the open source website and get involved and ask questions there.