Hi, good afternoon. This is Core OS Enhancements for BSD Developers. I'm happy to see you all here today. The heckling section is in the front. My name is Marc Majka and I'll be doing a tag team session today along with Dave Zarzycki. He'll be up in a few minutes. We're going to tell you about a couple of topics that I hope you'll find interesting and useful, the stuff that we've added to Tiger. I'm going to start off talking about the Apple System Logger.

Let's just get into it. What is it? This has kind of come in as a fairly low profile enhancement to Tiger, but it's the start of something that I think will be a useful direction for you as developers in the future. ASL is a new approach to creating and managing system log messages. It's a replacement for and an extension to the old BSD syslog system.

There's several new components. There's a new API that I'll be describing a little bit later. There's some behind the scene changes in the syslog API library. There is a rewritten syslogd server and a new syslog command line utility. The goals of this work were to provide structured, flexible, and hopefully more useful system log messages, to try to reduce the proliferation of log files that you find scattered around on the system, and to make it easier to find log file messages and read them.

Let's start with our starting point, which is the BSD syslog message system. BSD log messages had associated with them a priority level, one of a fixed set of different facilities, and a fixed format of message. In the example, a timestamp, a host name, the sending application's name and process ID, and a piece of message text. Pretty basic.

What we've done with ASL is extended messages, or we've made messages extensible key value dictionaries. Keys and values are just null terminated strings. There are a bunch of standard keys in every message. Things like the priority level, a time stamp, the sending process ID, PID, and the message string. But you can add additional keys that are appropriate to your application.

So if your application feels it's useful to attach, let's say, a color key to a message or a language or something like that, this is totally up to you and hopefully to make things more useful to your application. The little example message at the bottom shows a typical ASL message. The content here is not the big thing. The useful feature here is the message structure.

Take a quick tour of the ASL APIs, but before I get into it with the slides, let me just mention that your conference registration includes some complete example code that has a bunch of this stuff in it that may be useful to look at. I'll also direct you to look at the online Unix man pages for ASL and syslog and a number of associated components that I think will really help.

So here's a little example of just a hello world in the ASL API. ASL log is the basic send a message routine. There's a couple of null parameters at the front here that I'll talk about in the next slide. We're just using default messages in this one. And basically we're just setting a priority level debug here and a hello world message.

And off it goes to the syslog server. And once again you can see the message that this actually produces has a bunch of this sort of extra or standard message keys like the timestamp, the host name and so on. All of which are added into the message by the library code.

Here's a little bit more of a sophisticated example of sending a message. In this one, we start off by creating an ASL client. It's a connection handle, a connection to the server. In the previous example, we didn't have one of these, and that's sufficient for code that's single-threaded or that only has one thread that actually logs messages. If your application has multiple threads logging messages, each of them should have its own connection handle to the server to make them thread-safe.

We also create a connection handle in the ASL open call that creates the handle. There's a couple of parameters. There's a sending process name. We've allowed that to be null here, and it'll just pick up your application's name. And a facility. One of the nice changes in this ASL system is that facilities are no longer hardwired into a header file somewhere or other. A facility is just a string. If you're creating a new suite of utilities, application programs that all kind of work together in some way, then you can make up a facility name and use that.

You don't have to go edit anybody's header files, or you don't have to recompile anything. After opening a connection handle, this example creates a message, an ASL message structure. That's actually the key value dictionary. It's an opaque structure. And in order to set keys and values in that dictionary, you use ASL set. Here we're adding a subsystem key and a language key with appropriate values.

Once again, call ASL log to log the message, and then clean up with a couple of frees, a free for the message and a close to shut down the connection handle. Let's look at the output from this. You see, once again, all of this is in the same way. You can see that the output is the same as the output. Once again, all the standard keys and values set by the library, but this message includes the subsystem and language keys that you set in the previous slide in the example code. That's about it for that.

Here's a search example, the search API. Once again, we open up a connection handle using ASL open and once again create a message. This message is slightly different. This is actually a query. It looks a lot like a message. Once again, it's a set of keys and values. But in addition to setting a key and a value, you also can supply an operator.

Operators are things like equal, not equal, substring, prefix, as you might imagine. They're all documented in the man pages. And in this example, we're looking for all messages that have a key flavor that has a value vanilla, exactly. And we run through the data store and search for all of those with the ASL search call. That returns a list that we can iterate over.

With ASL response next, each of those gives us a message structure. See the example code or the man pages to look at how you tear apart a message structure to get the individual keys and values that are in it. You can print them or compare them or look at them any way you want. Finally, some cleanup to free the returned iteration list, the query message and shut down the connection.

It may be a little bit onerous to ask you to write a new application every time you want to go look for log messages, so we also provided a syslog command line utility, which will do exactly the same thing. It has a bunch of different search options. Syslog is a real Swiss Army knife of a tool. It does all kinds of stuff, so I highly recommend taking a look at the man page to see all of the options.

You can also do syslog-help, and it prints a reasonably useful help message on terminal. A few usage examples here. Syslog, all by itself, just prints every message that's in the data store. Syslog-k, flavor vanilla, as in this example, prints out every message that has a keyword flavor with a value vanilla, as you might expect.

Syslog-k, facility user. Syslog-k, time. Syslog-k, time. GE minus 1D, what's all that about? Well, that's actually searching for two different keys and values. We're searching for all user facility messages that were logged within the last day. So GE greater than or equal minus 1D means one day before right now. And that's a useful little convention. A day is just a 24-hour period.

It'll take all kinds of different things like H for hours, S for seconds, and so on, M for minutes. So if you want to know everything that happened since a particular time, you can also actually give it an absolute value of seconds, and it'll use that too, but most people don't have a watch that gives them the absolute number of seconds since January 1st, whatever, 1969. Final example here, syslog-w. The dash-w option means watch the data.

Watch the database. Watch the data store and print messages as they come into the database. It gives you a similar functionality to something like, let's say, tail-f of a log file. So it keeps watching the data store. The dash capital F flag means that you are supplying a format, an output format for the messages. It has a couple of default, or a couple of different formats that it can print messages in.

One's a sort of very much a very much like the kind of format that you might see in a syslog file, or a system.log file, excuse me. It has a couple of different options for printing out the messages. And if you want, you can say, no, I really only want to know, let's say, the time stamp and the sending process ID and the message text, as it is in this example. And once again, in this example, I'm only interested in messages that had a, a level less than or equal to three. So fairly high priority messages.

Let's take a quick look at the architecture of all of the stuff that goes on behind this. syslogd is the ASL server. It provides both the old syslogd functionality plus support for the new API and the datastore. There are input modules that receive messages on a variety of different channels and output modules that sort and file and forward messages as appropriate. We'll take a look at that in a sec.

There's an API for sending messages, both the ASL API that we just looked at a moment ago and the syslogd APIs. Both of those are in the system framework lib system and the syslogd APIs continue to work as they used to. And of course there's the kernel printf which also gets picked up by syslogd.

API for searching the data store is in ASL, the ASL library. And once again, I mentioned the syslog command line tool for searching the data store and monitoring messages. Here's the big picture. At the top, a number of different clients using different APIs, either the ASL API, the kernel using its printf, syslog API, or a remote client sending messages via a UDP datagram from somewhere else across the network. syslogd listens on all of those various ports, pulls in the messages, and hands them to various output modules. In this case, there's a couple, an ASL output module and a BSD module, both of which do whatever is appropriate with those messages.

The ASL module will put things in the data store. It will actually also forward to NotifyD. It's kind of a... This is a piece of work that's in development. We haven't really done a lot of documentation on that. If you're really interested in it, I can tell you a bit more in the Q&A after this.

But you can have syslogd send a notification, post a notification using the notification APIs whenever a particular message comes in that matches a particular search criteria. The BSD output module logs things to all the various log files as it used to. It'll send messages to a terminal as the old one used to. And once again, it will also forward on messages to some network log server.

Let me talk a little bit about filtering messages, because lots of processes are logging lots of messages, can lead to a fairly large amount of messages. So there's some filtering that you can use to control the flow. The client library, both the syslog library and the new ASL library, contain some filtering controls that you can set when you're writing your applications to determine which priority messages actually get sent from your application to the server. If you've used the syslog API, you're familiar with setlogmask.

And ASL has a similar filter to determine which priorities actually get through to syslog. The server, syslogd, also has a filter which controls which priority of messages it saves in the datastore. So by default, it doesn't save everything in the datastore. You can tell it to if you want. By default, it actually ignores debug and info level messages and stores everything else in the database. And what's interesting is that we've added a remote control mechanism on top of both of these that allow you to control the flow of log messages.

So you can tell an application, hey, even though I set the logmask to, let's say, filter out certain types of messages, now I want to see those messages. And you don't have to restart your application or send it any signals or anything. All of this happens dynamically. Let's take a quick look at that.

What we've done is modified the client library, both the ASL actually and the syslog library, so that they listen to... notifications from the syslog command line tool, syslog-c in this case. Once again, see the man page. But you can say to an application, I want a certain set of log messages to be either filtered out or passed through to syslogd server. Likewise, there's a control on the syslogd server for which messages it will actually put in the datastore. And once again, syslog-c will control... which messages get stored in the database.

Well, of course, if lots of messages get saved to the database, the database can get large. So we need some sort of a pruning mechanism to get rid of old messages. And by default, as I mentioned, the datastore doesn't save info or debug-level messages. You can set that as a startup option for syslogd if you wish, or you can use the control mechanism.

So just by not saving info and debug messages, we actually don't, the file doesn't grow that fast, but it still grows. So there's a daily cronscript, the periodic daily cronscript actually prunes the datastore. If you look in that file, let's see, periodic daily 500.daily, you'll see where the datastore gets pruned. You can also use syslogd, syslog, the syslog command line utility with the dash P flag.

Which is all that happens in the daily script to prune the datastore anytime you want manually. You just tell it to throw away certain messages. In the cronscript, we sort of tail off messages. So although we keep all messages above the level notice and above at the beginning, after a day, we throw away the notice message and the warning messages. After a couple of days, we throw away a little bit more. And then finally, we throw away everything. That's a week old or more. So by default, we only keep messages for a week in that datastore.

I should just mention that we're working on some improvements in that data store for Leopard and beyond. Right now it's pretty simple. It's actually mostly a flat file if you go and look at it. I know a number of people have. We're probably going to turn that into a slightly more sophisticated database in Leopard and beyond.

So that we can do a little bit better management of duplicated strings and make pruning a little bit faster and easier. With that, I'll turn the podium over to Dave Zarzycki who's going to tell you about service management. And I'll be available in the Q&A session afterwards. Thanks.

Thanks, Marc. So, service management in Mac OS X. Well, what are services? Services are background processes. We need to manage them somehow on the system. In Tiger, we've introduced launchd. It's our unification of background process management. It is the daemon to end all daemons. It is XML based instead of shell scripts. These are our key points we'd like to bring up, actually. It's XML based instead of shell scripts. It is less work for you as a software developer, as we'll demonstrate later.

It is easy to use for simple scenarios. Not very many knobs you need to tweak just to get your daemon up and running. It has more flexible options than any configuration daemon before it. And it has new kind of launch-on-demand criteria if you don't necessarily need your job running all the time. It has a simplified notion of dependencies that we actually feel can lead to a better system as far as reliability is concerned. And it has support for user-supplied jobs, which some of the previous daemons to manage all other daemons didn't support.

So XML, why XML? Well, structured data is a good thing. It's very easy with XML to quickly introspect every job, daemon, agent, whatever you want to call it on the system, and look at, let's say, well, what user is this job running as? Well, there's a key and a value for that. Is this job nice at all? Well, there's a key and a value for that.

And let's say if we want to bulk modify some daemons. Well, we can script that, we can quickly do that in a structured way. And it's consistent above all else. That's why we can do things like this. In the past, we had shell scripts and with the shell script, it's a language. It's really hard to just run into a language, you know, any kind of file with a language in it and just find where a particular variable is tweaked or there could be a whole multitude of ways those things are accomplished.

And we brought that all together with XML, so it's done in a consistent manner. It's also faster, too, because now you don't need to run through an interpreter of some kind and actually try and run through and do what probably amounts to a few simple things like just, hey, run this process. We can just launch it directly.

Now, on the topic of less work, how can you as a developer find yourself with less work? Well, you don't need to fork and have the parent exit, or what some people call daemonizing. SetSid, another common daemonizing task. Nope, you don't need to do that. You don't need to close stray file descriptors. You don't need to reopen standard IO as dev null.

And you don't need to change the working directory to slash. There's a whole bunch of things. No, you're pre-daemonized. You can hit main. If all you want to do is say sleep for a million years, fine, that's a daemon. You don't need to do anything special to become one now.

And in fact, when you don't do some of these things, like close standard IO, that allows us to actually, before we launch your process, maybe direct those to interesting places. By default, we'll have them go to Dev Null, but if, let's say, an administrator says, no, for this daemon, I want his standard out and standard error to go to this file. Well, now we can do that. Whereas before, the daemon would just say, bye-bye, I closed it, I know what's going on. And that would be difficult to do, but now it's easy.

Now it's easy to use. Here is, minus the header and footer, an XML property list. We have the label, which uniquely identifies the job to launchd. We have the on-demand key, which tells launchd, "No, this is not on-demand. We want this job running all the time." And what program to invoke, in this case, somethingd.

And what launchd will do is when this property list gets loaded up into launchd, it'll say, hey, this is not an on-demand job. We need to have it running all the time. And it'll start something deep. And if for some reason something deep consciously or accidentally exits, launchd will notice and say, hey, this job's supposed to be running all the time and start them back up again. Marc Majka, Dave Zarzycki So now you as a daemon writer cannot worry about, you know, if you die accidentally, the system will restart you.

Increased flexibility. Well, as we talked about, we now have a standardized schema with XML to represent how a daemon started. We also have a lot more parameters that you can tweak than other previous daemon managers have supported. As I mentioned earlier, we have the standard out and standard error that we can set on a per job basis. There's the nice level, the working directory if you want a daemon to maybe have a little sandbox over here.

Even the root directory, if you really just want to root the daemon off and have them not affect other parts of the system. The UMass, you can have per job environment variables that are easy to set up and see in a consistent manner with the plist. There's actually a huge multitude of things you can adjust and where you can look for these variables on the system is the launchd.plist man page. It's in section 5. And you can find where all this stuff is documented and what they do and how they behave.

We also have interval timers. Now some of you may have in the past in cron sat there and calculated out the exact interval you wanted to run for every five minutes. So you say 5, 10, 15, 20. No, no, no, you don't need to do that anymore.

Just say run it every 300 seconds and launchd will do the rest of the work for you. And in fact, if you want to be weird and say every 137 seconds, you can do that. Also, we have still, if you want the more cron-like semantics, you can specify a calendar-based interval timer.

So that way you can run it on the third of every month. Now, dependencies. Let's first talk about the old way. Since we have been talking a little bit about system boot up, it's important to talk about dependencies because we came from a world with System Starter. In System Starter, dependencies were explicitly declared by the Startup Item writer. Well, in reality, this didn't work out well. What ended up happening is that one of the following things would happen. Dependencies were overstated. People weren't sure, so they'd just throw some more in there until it worked.

Dependencies were sometimes understated. Sometimes a developer didn't know and it just happened to work on a system. So he missed something, but it wouldn't work for someone else. And dependency meaning was often vague. Well, I need the network. Well, what does that mean? Do I need a DNS server? Do I just need an IP address on an interface? I don't know.

And that was difficult for people to both decide on what dependencies to export and what dependencies to export. Also, dependencies sometimes were just virtually assumed by other dependencies. Well, I need, let's say, directory services. And there were some StartupItems out there that we found inside the company that assumed that if they needed directory services that the network would already be up, whatever that meant.

Well, that was a problem, and we need to fix that. So how did we solve all this? In Mac OS X, we've essentially inlined dependencies. We made the observation that, well, you don't really need the network, you need the ability to talk to configd. And configd knows when the network's up and can tell you when that changes.

We noticed that some people need to find out when disk came and went, the device, you know, disk were mounted or unmounted. Well, you don't need to wait for all the disk to be mounted, you need to talk to disk arbitrationd and find out when these events happen.

So this observation was IPC was the key. And what we do now is we register all these IPC handles of configd and other daemons with launchd. And we get all those registrations done at bootup. And then we allow daemons to start up. And because the communication handles are registered now with the system, daemons, when they start talking to each other, now find each other because their handles are out there.

And this allows us to boot up really, really fast because what we can do is start up a lot of daemons in parallel. And as they're initializing themselves and start talking to other daemons, it doesn't matter if the other daemon isn't done initializing itself. It won't answer its IPC query until it's ready. And that way, the thundering herd, eventually the dust settles and your system's booted. Yeah.

Now, again, what this means though, as a daemon writer, if you expect people to depend on you, you need to declare your sockets and your communication and your P-list. Because if you don't, people can start up and they'll try and connect to your socket and it'll fail. And then you end up with all these degenerate code paths that probably haven't been tested in a very long time. So what you, yeah, please, please just declare your socket's in your P list and everything will just be groovy.

Now, if we want to talk about a case study of declaring socket in PLIS, we can talk about the SSH agent. In the past, there was an interesting dependency problem that people would have with trying to start their SSH agent. What people would do is they'd put a lot of complicated shell logic in their shell startups to figure out if SSH agent was running. And if it was, use the existing copy. And if it wasn't, start a new one. Well, this is all actually a lot easier with launchd now with a small patch to SSH agent. And I'll demonstrate this later in a demo.

But now it's just you put a plist in your home directory saying here, launch SSH agent when somebody connects to this socket. And SSH agent will then start up and accept whatever connection came in. And it's now a lot simpler for a user to use the SSH agent. And if you want to actually play with this, you can go to one of the OpenDarwin developers, Landon, and grab the patch and the plist and play with it yourself.

Now, this SSH agent that is talked about, it needs to get its socket from launchd and actually then demux and dequeue this IPC. Well, how does it do that? We have an IPC API to talk to launchd. It's a very simple RTTI, runtime type information based object graph system to support message passing. That's all it's designed for.

It's not designed to be core foundation or any of these other stratosphere level frameworks. It's just IPC, you know, hello, goodbye, you know, that kind of thing. Here's the simple C APIs. As you can see here, the very first one, the one and only really one to talk to launchd is launch message.

It'll take an opaque object in. It'll return you an object from launchd. An example of creating an opaque object, as we see here, is I'd like to create a new integer. And you get an object back. And I'd like to get from the object what integer is in there. There you go. It's really simple.

Now, if we want to talk about the semantics of that launch message and launchdataT, here we go. launchdataT just represents an object graph. It could be a single object, like a string or a number, or it could be a more complicated tree. The launch message API is synchronous for the common case. It returns null if there's some kind of IPC failure itself to talking to launchd. And you can check error no if that happens.

You can get asynchronous messages back from launchd if you've requested them by passing null to launch message. And then the first message that was asynchronously sent to you will be returned. And you can keep calling that in a loop until you've drained the queue of asynchronous messages. And at that point, null is returned. And then you can check error no to see if it's not equal to zero to differentiate whether there was an actual error or if there's just no more asynchronous. messages.

Now, let's dive a little bit deeper into launchd. What does it support? It supports dictionaries, key value pair-based entries. It supports plain old arrays, if you don't actually need keys. It supports file descriptors as a unique object type. It's not just a number, it represents a file descriptor. Integers, real numbers, booleans, strings, opaque data. This is all that we needed and actually a little bit more for talking to launchd.

The, again, this is just IPC we're talking about. We have get and set for the basic types. That's all we need. For dictionaries, the only thing we needed was the ability to insert, look up, remove, and iterate. Again, we're just talking to launchd. We're not trying to solve all the world's problems. Arrays, get and set index, and get the count.

Now, stepping back a bit from the APIs, I'm going to talk about the XML plist. There's only two keys that are actually required in LaunchD property list. One is the label, which uniquely identifies the job, and the other is what to launch. And by default, if you don't specify the on-demand key, you will default to being on-demand because we'd just like to push you to be that way because, again, it helps you help us. But the common case is that people will include the label, the program arguments, and the on-demand key with the on-demand key set to false. So that way they can have their daemon running all the time.

But if you do have the on-demand key set to true, we then have some optional keys you can then use to specify how to launch you. As we talked about earlier, there's the sockets or the timers or watching a file. There's also many other optional keys, as we've talked about earlier. There's even things like the username and group name if you're having a daemony-type launchd job.

And let's see. Yeah. Now, sockets. Sockets are actually a complicated thing to set up. But we tried to do a lot of same defaults. We default to example to the stream type, but if because that's the common case, but if you need a datagram socket, you can set that up.

It contains a lot of details of like, I want to connect to this machine. And then a file descriptor will be created. Or maybe you want to listen on this IP address with only the IPVP. And a datagram socket. Well, you can then also specify that. And to repeat myself, since I think it's important for this talk, all these configuration details of how to specify your property list are in the launchd.plist man page. Now, once you have your socket set up, we get to an interesting thing. What we do is we actually take this property list and do kind of a quick transform of it over to launchdata-t, since it maps really well.

The only difference, though, is we take the socket specification and turn them into launchdata-t file descriptor objects. And that has a very interesting and powerful effect. What that does is it makes it so launchd is not aware of socket types at all. It's just an opaque file descriptor. This future proofs launchd such that you could have a long running system. And if you wanted to have a new file descriptor type as maybe a kernel extension.

You could do that, load the kernel extension, create this new socket, and hand it over to the running launchd, and now he can launch your job on demand. It also means since you can programmatically talk to launchd, you could also come up with a new file descriptor that launchd hasn't heard about and just hand it over and say, hey, yeah, when this becomes readable, launch this job. So we're very excited about this particular aspect of launchd and how potentially people might use it.

But other than that, we take the XMLP list that's now transformed and we can do something like a submit job as an example message. And messages are simply dictionaries with the one key and the value is the message. So the key might be submit job and the value is the now launchdata T that represents the XML property list.

You can also do something like a remove job. That's again a dictionary with one key saying remove. The value is the string representing the label. You can do other things like get jobs and get jobs is so simple it doesn't even take an argument. So the way we do that is we just throw a string at launchd saying hey, this is the message.

It's a string. It says get jobs. If you want a specific job, again back to the dictionary, hey, here's the key, the value is the label and you can get a specific job and find out about its attributes. Checking in is how a native launchd job gets its configuration parameters and its file descriptors back from launchd and then goes about its business.

There's also things like start and stop, which if we rewound a few slides and how I pointed out that a label and program arguments is all that you need. Well, if you don't specify the on-demand key, your job's just going to sit there not running. And if you really want, you can manually poke a job and say, hey, launchd, start that job.

And launchd will start it. Or if you want to say stop it, launchd will stop it. But that's primarily meant for testing purposes. If you have an on-demand job and you want to manually start it up to make sure that it works, your keys and your criteria should ideally launch your job on demand.

Now to go into an actual programming example of how you would talk to launchd. Again, we're going to allocate a dictionary for the message. We're going to insert a string, com.example.helloD, and start job is the key. And here we go, launch message, send the message and get a response. And now we can free the message and the response. It's really that simple to talk to launchd.

Now to rehash. Launchd, we see this as the future. It has less work for you. You're pre-daemonized out the get-go. If you just want to sleep forever, you can do that. There's nothing else you need to do to become an official daemon as far as we're concerned. If you've gone native with us, just check in and go. That's only one message you really need to send and get back. Other than that, you're done talking to launchd. There's automatic restarting if you're worried about your daemon staying up. Again, a very powerful feature that many people will appreciate.

There's more flexible criteria than ever before to get your job running. We hope that you can take advantage of it to help us help everybody save system resources. We also have the ability to monitor multiple, lots of file descriptors if you want. So if you want half a dozen connections scattered around the universe and be able to launch on demand whenever any of them become readable, we can do that. And finally, user agents, like I talked about with the SSH agent.

That is a powerful concept that we haven't ever had before with daemons like INETD or INIT. So let's jump into a demo about this SSH agent and some of the other powerful things launchd can do. So system two. Okay, so just to show you what can happen with the SSH agent, we've dropped a property list describing it into the system. And when we log in now, we have an SSH agent. Oops. Ready to launch on demand. And I'll show you with a quick PS output that it's not running, but when we say SSH add, it'll get launched on demand.

Let's see, that's readable, right? All right. No SSH agent, but if we do a set pipe grip SSH, This is the font of demos. Library launch agents. Yeah. Apparently it wasn't, oh, I'm sorry. One of the things I haven't done that I was going to demonstrate is how this environment variable gets set. It's actually loaded, as I can show you right now with launchctl list. But the environment variable that SSH add uses to talk back to the agent isn't set.

And if you remember before, I said that people needed a lot of shell logic to figure out whether the SSH agent isn't running or what, or whether to start it. Well, we don't need that anymore. In fact, the only thing you need to do at the top of your shell startup scripts now is say launchctl export.

And what that did, just to run it again and show you, is it did a quick shell script output of all the environment variables in launchd. And now we've set them in our own local shell. And the important one would be this SSH authsoc that is using launchd. And now we can say SSH add.

And now, the SSH agent is running. And the -l flag is the flag we created to say launch on demand. And just to prove that that is the case, we can say list, there's the key, and if we kill that, and then list, now there's no more identities. And it's, obviously a new version is running. and David Zawadzka, are the main developers of the OS Enhancments for BSD.

This is what we used. Here's the program arguments to launch SSH Agent on Demand. As you can see, they're pre-tokenized, which saves everyone a lot of work. Because that way we don't need to launch a shell just for every daemon we launch. Now we can launch the process directly. And for the sockets, again, I mentioned earlier that we have a lot of defaults like SOC stream.

In this case, a secure socket with key implies a Unix datagram socket. And the key is the environment variable we want set. And this is all we need, oh, this and service IPC, which contractually obligates that this guy will do a check-in with launchd. But other than that, this is all we needed to get the SSH Agent running. To now move on and show some general launchd, launchctl stuff that you can look at on your own Tiger system, we can move to the stuff that Apple provides.

So if we were to look at the Postfix example, we can see that we launch post, in fact, we specify where exactly the program is with the program. But the actual arguments are right here. And what this -e says is, and 60, is that after 60 seconds of nothing to do, Postfix will exit.

And that wasn't even something we needed to add to Postfix. They already actually had that built in. And again, here's the Q directories. If we just watch this directory and anytime something shows up in there, we launch Postfix. Another interesting example, actually using a combination of launchd criteria is the cron example. Now we, oh, that's right.

We have cron, the thing we're running. And if we notice the lack of the on-demand key, it now defaults to on-demand. So this is an on-demand job. What we did tell launchd to do is try running this at least once. And what that does is when cron is run at boot, it looks it around, particularly its Etsy cron tab.

And if it finds that it has no work to do, no user cron tabs, no system cron tabs, it exits. And then we can take it back to the system. And we can take advantage of this launch on-demand criteria to watch this file and to watch this directory to see if it goes non-empty. And if either of these criteria fire, we then run cron again.

It goes out and looks around and finds some work to do and then sticks around and does its cron thing. But again, if it finds no work to do, it'll exit and it'll launch on-demand the next time particularly this file changes. Another example that I'd like to demonstrate.

is the, oops, Oh, the NNB, I believe, oh sorry, SMBD. So this one's a little bit interesting. The front part isn't as interesting as much as the next part. So to demonstrate the Sockets dictionary of how that works, we have two keys here. Notice net bias and direct. These keys are up to you, the programmer, to use. You can use these keys to describe whatever you want. Ideally, it should be a protocol. In this case, these are two separate protocols.

And what happens is when you check in, you get the Sockets dictionary back, and you can look up these keys and find out the file descriptors that actually correspond to that protocol. So this dictionary will end up allocating some file descriptors, probably just one, since we're only doing IPv4.

But again, here's the service name. We'll look this up, translate it to a port, bind to a file descriptor, and that'll get passed along to when this guy checks in. Direct, again, we'll look up a different port. It represents a different protocol, so we stuck it over here. And we bound it to IPv4 specifically. So that's an example of potentially having one daemon handle multiple protocols using the configuration file syntax.

Is there any other fun examples to show here? Oh, an example of the periodic, the calendar based stuff. We have actually a few extra keys here, which are interesting. One is the low priority IO. That's a feature we have in LaunchD. It tells the kernel, well it gives a hint to the kernel that this job is not terribly important as far as its disk IO is concerned.

So try and make it a second tier as far as disk access is concerned. And the nice does the same thing for the CPU. Now we have the plain old cron stuff before on the, at 3:15 in the morning, let's run this job. And finally, let's see. Oh, I should at least show the simplest example as I demonstrated earlier.

Here's a job that just runs on demand. Here's the label. Here's the program argument. On demand, false. And this is how the kernel event agent starts. So I think that's enough plist to show. The only other fun thing to show is you can use launchctl to talk to launchd.

And there's a whole bunch of things you can do. You can say load to load a property list. So for example, You could say load system library launch daemons nmbd. And that's how you would load up a property list into launchd. If it were disabled, which is a key, you can use dash w to remove the disabled key and then load it up.

Because at boot up, what happens is this whole directory of system library launch daemons gets evaluated to decide if we want to load a job. And if the disabled key is true, we won't load it. And the dash w flag removes the disabled key. But some other fun things you can do, for example, is you can list all the jobs. You can look at its environment-- whoops.

Yeah, export is one way of doing that. But if you wanted to, for example, set mv foo bar, now you can get mv foo. And that came all the way from launchd. We set it over there, it came back, and you can even unset it, which is useful for any job launching out of launchd, if you want to just have it set globally, not have to set it in every plist.

You can do things like adjust launchd's limits if maybe you want to turn core dumps on and not have to do it on a per job basis. You can use the launchctl command to adjust launchd itself. Here's what launchd is logging, the following log levels. If you wanted to adjust its logging, you can use the log command, log level debug. and now it's going to log debug messages.

Let's see, help. But all these things here are adjusting launchd to kind of a global level, not at a per job level. You can also, oh, one of my favorite things to demonstrate is if we use the lsof command to look at launchd, you can see the standard out and standard error are dev null, whereas now we can use launchctl standard out var log launchd dot launchd.

We can see that those files are now indeed opened up to that. And now any daemon that launches at a launchd that doesn't manually override its standard out and standard error will inherit this log file, which can be useful sometimes if you just want to get the standard out and standard error of every background process on the system. So I think that's it for doing a demo. If we can get back to the slides. So for more information, we have documents, sample code, and other resources available at the following URL. and we should talk to Craig Keithley at, there's his email address.