Network Configuration & Mobility Architecture on Mac OS X. In particular, we're going to be talking about the particular pieces that we have and how they interact. And I'm not going to be talking about APIs today, unfortunately, because what we have today is all private. And it's a work in progress. So what we're looking from you guys today in particular is some feedback on the kinds of configuration questions you'd like to answer. We have some APIs and I'll talk about that a little bit later.

So what are some of the goals that we have? We'd like to support dynamic network configuration changes. On Mac OS 9 everybody was used to the idea that you don't have to reboot your computer to make a change, so Mac OS X has that now as well. The other thing that we wanted on top of what we have in 9 is the ability to configure multiple network interfaces at the same time and actually make use of them in a sensible way. The other goal that we have is to avoid user intervention when it's possible.

So, a couple of examples. You've got a laptop that you're using for work and at home and you're working on some project at home and you're using the network. You put the system to sleep, you bring it to work, you plug it in and it just works because you're using DHCP and it figures out a new address and it just works.

Or, if that's not possible, you switch locations and you pick up your new config, just like on 9. Another config that uses a multi-homing is the LAN and airport configured simultaneously. So, your system prefers LAN over airport while LAN is connected, but when LAN disconnects, the system switches over to using airport automatically.

So to be able to support this kind of thing, we have a set of requirements. The first thing that we need is some sort of low-level network events. So when a link status changes, we need to be able to find out about it and do something. We need some sort of persistent storage to be able to store the network configs.

We also knew we needed some sort of dynamic storage to be able to store temporary information or dynamic information as it's generated, such as information from a DHCP server or a PPP server. The other thing that we need is some mechanism to be able to apply some configuration to the system. The other requirement we have is some sort of flexible schema in both the persistent and dynamic store to be able to allow for future expansion.

So what we've come up with is a system configuration architecture and we've decided to call it system configuration architecture, not the network configuration architecture because we wanted to generalize the concepts. We knew we were going to solve some general problems and wanted to make the services general. Our solution leverages core foundation pretty heavily using run loop source technology and all the property list types and objects.

The other point about the network configuration architecture is that it's open source. All the source code we have today that's shipping in Mac OS X is available in Darwin. You can check out the projects and see what it does. Unfortunately, as I mentioned before, it's all private API, so you can't technically really rely on it, but it's there for you to look at and understand. So the system configuration architecture contains three main pieces: the dynamic store, system configuration preferences, and the configuration agents. The dynamic store stores the current system configuration information. It's a snapshot of the state of the system. The other thing the dynamic store provides is notification services.

The System Configuration Preferences provide system persistent storage for configuration information. This is where the sets in different locations are stored. Configuration Agents apply the configuration that the user has chosen to the system and interact with the dynamic store. So here's a little diagram that illustrates the relationship between the agents, the dynamic store, and the preferences.

and where each of the individual agents are interacting with the dynamic store in some configuration source. So what is the dynamic store? The dynamic store contains the current system configuration information. It's a snapshot of basically how the system is configured. It's not persistent across reboots, so every time the system starts up, the contents are regenerated.

It's a dictionary, so it's a set of key value pairs where the key is a CS string whose format is a hierarchical namespace, much like a URL. We've encoded it using a domain prefix and a path, so in the example there, there's a state colon prefix and the path is network global DNS. The value is any CF property list type, normally a CFDictionary, so that you can contain multiple properties within a specific key.

So the way the dynamic store notifications work is a client decides that it's interested in a particular key and registers to receive notifications. And the notifications are level triggered rather than edge triggered. What we mean here is when a key changes, you aren't provided a key and its value. You're provided that the key has changed and then it's up to you to go and verify what the value is.

We're trying to avoid creating big long streams of event buffers that you know when the system gets really busy may overrun. The dynamic store is hosted by the system configuration server, which is actually a daemon in the system called configd. And I call it the heart of the mobility architecture because everything that happens with respect to networking goes through the dynamic store.

So the second major piece of our architecture is the system configuration preferences. It's a persistent storage, as I mentioned before, for the system setup information. Like CFPreferences, but it actually has notifications when things change. The other thing that it has is that it's accessible, easily accessible from system daemons, and the source code for it is in Darwin, whereas CF preference currently is not in Darwin. We use CF property list types for just about everything, and when the file is written out to disk, it's serialized as XML. And the default preferences for the system that are used to store all the network configurations is in var db system configuration preferences XML.

So what do the system preferences contain? Currently we've defined four main preferences: the current set, network services, sets, and system. The current set contains a link to the currently active set. It's pretty much all it contains. The network services contains a complete list of all the network services that have been defined across all sets. And the sets preference is just that, it's a list of sets. And the system contains items that are not location sensitive, such as the computer name.

So what's a set? A set is the complete configuration for a single location. It contains the list of network services that are defined for that location. It also contains the order of the network services for that location. So here's where you would prefer LAN over built-in airport. A network service, what we mean by that is a collection of network entities which are just a set of properties for a specific area of interest which I'll mention in the next slide. But it's the network services, the configuration for a single network connection or interface. And it's identified by a unique network service ID.

Network Entity is a list of properties for a single protocol or area of interest. An example there would be IPv4 or PPP. There's a set of properties that are associated with the protocol or the area of interest. So here's a diagram of what we have in terms of the preferences and the relationships to each other. You can see the current set contains a path to a property inside sets that points to which set is currently active.

So when you go into the location pop-up under the Apple menu, the thing that it's manipulating is the current set. So you notice there we're pointing at set K. Set K is a dictionary that contains... a sub-dictionary called network. And... the sub-dictionary network contains two sub-dictionaries itself. One called service and the other global.

Inside service, we have links to the individual services that are active for that set. So you notice IDX points to IDX in network services. And inside IDX in network services, we have a set of network entities. In this case, the configuration method for IPv4 is set to DHCP. And the interface entity has a device name of EN0.

So we've identified the device and the configuration method.

So, the third main category of the architecture is the configuration agent. Configuration agent manages some aspect of the configuration. It's associated with one or more configuration sources, basically things that can be configured or provide configuration information. And it has a well-defined role.

So from the original diagram, the relationship is that the agent interacts with the dynamic store and it manages some configuration source or multiple configuration sources. So here's a list of a number of the agents we have in Mac OS X today. The first one is the Kernel Event Monitor and the second is the Preferences Monitor, the IP Configuration Server, IP Monitor and PPP Controller.

I'm going to go into more detail on each of these in the upcoming slides. So we'll start with the Kernel Event Monitor. Kernel Event Monitor is an agent whose sole purpose in life is to maintain the link status and the interface state of all the interfaces in the system.

So what happens is it opens up a kernel event socket and receives kernel events that interfaces are added, the link status has changed, addressing has changed. And when it receives those events, it populates the dynamic store with the values. In this case, the link status on EN0 has changed and the kernel event monitor has updated a key in the dynamic store that says that the link status has gone active.

So as I mentioned before, it maintains the network interface state and it manages the kernel event socket. It maintains a list of network interface names in a key that's encoded as state colon network interface. That's actually a dictionary that contains an interfaces array that contains the name of each interface that's defined in the system.

So if you want to know if an interface has changed, has been added to the system, you would receive notifications on that key. The other thing that it maintains currently is the link status of each interface. And we've encoded that as state:network interface interface name link, which is a dictionary that contains inactive property, which is a Boolean.

By updating the dynamic store in such a way, the Kernel Event Monitor provides notifications for other agents, in particular the IP Configuration Agent, as we'll see in an upcoming slide. The second configuration agent that I'd like to talk about is the Preferences Monitor. The Preferences Monitor's sole purpose in life is to take the current set dictionary and load it into the Dynamics Store.

So it's configuration source is the system configuration preferences. It maps the current set dictionary into the dynamic store. And the mapping process amounts to a flattening process and the next couple of slides try to illustrate that. So here's the dictionary before we flatten it. So if you look at it, we've got a dictionary that contains property that's a string called UserVisibleName and it's set to work LAN. It also contains a sub-dictionary called DNS. The DNS contains server addresses and a domain name. After flattening, that'll become two dictionaries.

The first dictionary is encoded as the dictionary at that root node and the second dictionary is encoded as just the dictionary at that second node with all the other elements removed. So if you look at it again, we go from that to that.

So the mapping amounts to a dictionary tree traversal where we start at the root and at each dictionary node we compute a key and a value.

The key is the path of the dictionary within the preference and the value is the node with all the sub-dictionaries removed. Also part of the mapping process, we resolve all the references and links. So there's a possibility that there are links inside the preference. We'll resolve those and fully populate them.

The other thing that it does is it eliminates dictionaries that are inactive, that have been marked by the user interface or some other entity, and it also eliminates empty dictionaries. So if you have a dictionary that doesn't contain any keys, effectively the entire key is just destroyed. And before putting it into the Dynamics store, It adds a setup colon prefix to the key. That's its namespace within the dynamic store. So why do we do the mapping? The reason why we do the mapping is to provide finer grain that is per dictionary notifications. The other notifications that we would get without it would just be simply that the preferences had changed.

So instead of doing that, we can actually generate fine-grained notifications by keeping track of the current mapping and the new mapping and computing a difference between the new and the old and triggering notifications on the keys that have changed. So the benefits of doing this are that a single agent understands the preferences layout and other agents only have to know the dynamic store layout. So it provides a layer of abstraction.

So I'm going to talk about the next agent, which is the IP Configuration Agent. IP Configuration Agent configures Ethernet devices for IP using Emanuel, DHCP, and Bupi. The way it works is it's triggered off of notifications out of setup. So anytime someone switches locations, it reads the configuration out of setup that the preferences monitor loaded and applies the configuration to the interface that's associated with the particular config.

Anytime the user goes into the control panel and changes a key, the IP configuration agent wakes up and applies the new config to the system. Once it's configured an interface, it writes the dynamic store keys to indicate that it's done its configuration and the configuration is active. In particular, it writes the IP address and subnet mask and for BOOP and DHCP, it supplies router and DNS info.

The other feature of the IP Configuration Agent is that it detects the link status. So the Kernel Event Monitor is providing notifications that the link status is going up and down. The IP Configuration Agent notices when the link goes down and stays down for some period, currently set at 4 seconds, and deconfigures the interface when that period is expired.

And when the link becomes active again, it knows that it's going to retry the configuration method. So you can take a computer and plug it into one network and then plug it into another. It'll retry the configuration method. And if you're using DHCP, you can grab a new address automatically.

The fourth agent I'm going to talk about is the PPP Controller Agent. PPP Controller does a similar task to the IP Configuration Agent except that it manages PPP interfaces for IP. So it does dial-up modem connections in PPP over Ethernet. Just like the IP Configuration Agent, it reads its configuration from the setup keys and it's notified when the user makes changes as well.

And once the interface has been brought up and configured, it writes dynamic store state keys when the configuration is applied. So it's going to supply IP address, destination address, and a router and when the PPP server supplies it, it's going to supply DNS as well. The other thing it does is it creates PPP interfaces as it needs it and interacts with dialers.

Which brings us to the fifth agent. The IP Monitor Agent is actually the thing that decides which interface or service is going to be primary. And what we mean by a primary service interface, we mean the one that's associated with the default route and the default DNS for the system. Since the system currently can only make use of a single DNS in the system, you have to pick some interface. And so what we do is we allow the user to select which one they want by ordering them.

So it consumes the information provided by the IP configuration and PVP controller agents and waits for changes to appear. And when a service comes online, decides that, okay, that's the interface or service I'm going to use, and it elects it primary and takes the configuration that's supplied both in the back end, the permanent preferences and the dynamic information, and applies it to the system.

As I mentioned before, there's a service order array that decides which service is going to be primary when you're in a multi-home situation. So now I'm going to call my colleague Allan Nathanson up here to do a demo for you and show you some of the things I've been talking about.

So Dieter went over and sort of gave a good overview of what we've got and my job is to sort of show you a little bit of the stuff under the covers and So the first thing that I like to do is crank up System Preferences and I've already picked the Network pane here. And we have pre-configured a couple of different locations. Automatic is the one that you would get first time you go over and crank up your system.

I've got a service called None and if you go look at the Advanced tab you can see that all of the three ports have been turned off. This is the one that I use when I'm flying from the east coast to the west coast and don't want to go over and disturb the airplanes networking.

And then I've got a couple other configurations here. The WWDC Air and LAN. And this configuration has got all three network services enabled with the airport interface having a little bit of priority over the built-in Ethernet, which has priority over the internal modem. And the idea here is that if the airport interface is available, we'll use that. And if not, we'll try going over and using the built-in Ethernet.

And if that's not there, off to the internal modem. And a very similar configuration is the WWDC LAN and Air. And if you go over and look at the difference, this one's got the built-in Ethernet being on top. So these are the configurations that I've got. A couple other things to look at. In the built-in Ethernet configuration, it's set up for manual IP addressing.

IP address, a subnet mask, a router, some DNS information. This is what you'd have to do if you didn't have a DHCP server available to provide all the information for you. In the airport case, we're set up for using DHCP and nothing else has been provided. So we're sort of like relying on the DHCP server to provide the configuration information.

This is what we've got from the top side of the view. This is what you as a user would go over and set up for your configuration preferences. All of this gets stored in that configuration store, the preferences store, which is the next thing that I'm going to go over and show you. It's all stored in an XML file.

happen to have a handy dandy reference. The root dictionary of the preferences file has got four preferences that we've currently defined. The current set, the network services is a dictionary of services that we've got. The sets is a dictionary of the sets which have currently been defined and then the system.

The current set says that, well let's go over and look at sets for a second. Four different services were configured. Set 0 is the automatic set. You can tell that based on its user defined name. Set 1 is the WWDC LAN has priority over airport service. Set 2 is the airport service has priority over the LAN. And set 3 is my none.

So, if we go over and chase down current sets pointing to sets 1, we can open this up. And within the network configuration of this set, we've got a couple of pieces of information. The global information, which Dieter alluded to. Within the IPv4 configuration, here's the service order array. And it goes over and says that the relative priority of the services for this set is service ID 1, followed by service ID 2, followed by service ID 0. And these are all strings.

So that's the global state. And then within the services, again, we have the three services, service zero, one, and two. And one of the things that Dieter mentioned was that We have the ability to put in some links. So Service ID 0 is really going to be grabbed from Network Services 3.

and David are the main speakers. The first two are the main speakers. The second is the main speakers. The third is the main speakers. The fourth is the main speakers. The fifth is the main speakers. The sixth is the main speakers. The seventh is the main speakers. The eighth is the main speakers. The ninth is the main speakers. Network Services, 3, 4, and 5. 3 is the configuration for the internal modem. 4 is the configuration for the built-in Ethernet. Five jumps around a lot. Here's the configuration for the airport.

Within the airport configuration, we go look at the IPv4 and how it's going to be configured. It's set up for DHCP. If you remember the airport configuration, we didn't have any information for DNS, so there's an empty DNS dictionary. One thing that you'll see if you go over and look at the preferences as they're stored right now, there's a couple of dictionaries which are sort of like, why is this here? And the big reason behind that is we're trying to go over and remember things for the UI.

There's an inactive flag that will go over and show up which the preferences monitor looks at and says, ignore this for now. It's not relevant to the configuration. So the airport configuration is set up for DHCP and the built-in Ethernet configuration is set up for manual addressing and the addresses that will get applied. There's the IP address, there's the subnet mask, and there's the default route. And then up here in the DNS dictionary, you have the domains that are being searched and a list of addresses for the DNS servers.

This is what's sort of at the very bottom of the food chain is the stored preferences. So the next thing to do is to go over and look at what's in the Dynamics store. When putting all of this stuff together, I wrote a little command line tool. to go take a peek into the Dynamics store and exercise the APIs and it's just something that comes in real handy. So this is the System Configuration Utility.

gives you a little prompt and you go over and open up a session with ConfigD with the system configuration server. And there's a whole bunch of commands that you can do and I'm going to go over and show you a small handful of them. There's a list command which you can go over and say, "Show me a list of keys in the dynamic store that match this prefix." So here's keys that relate to the current state of the system. Here's a list of keys that relate to the stored preferences. And what you end up doing is do a get of a given key.

and then we go over and show the results of what that information got fetched out of the dynamic store. And you can see that we've got the subnet masks, we've got the addresses, the config method is manual, the default route is here. So this is really just the flattened version of what's stored in the configuration preferences.

If you want to go over and look at things like what the current link status is, Get the state. It goes over and shows the active flag and a Boolean value. This version of this command doesn't split out things as well as I'd like. There's a couple of other things you can do as far as notifications. What I'm going to do here is I want to go over and say, Let's go over and watch for a notification of something changing. So I see do a notification add. And I want to go over and watch for changes to the built-in Ethernet.

So at this point, the command is sitting here. It'll go over and it's told the system configuration daemon to let it know when something happens to the link status on EN0. And if I reach behind here, and Yank out the cable, an event just happened. And it turns out everything else is going on behind the scenes and it's going over and reconfiguring things. You plug it back in again and link status comes back up.

It's flexible in that you can go over and specify which keys you want to go over and watch for, and you can use it for one configuration or another. So if I want to go over and say, let's watch for another key, and now we're watching for this key also, let me go over and bring up another window.

Open a connection to the server and we'll go over and create a new dictionary. This is a key value. Hello is the key and the value is everyone. So the contents of the dictionary, so the key is hello, the value is everyone. And then when we go over and say let's go add this key to the dynamic store, if you go over and watch the top window, when I go over and add the key, it sees the change.

So you go over and tile this together with the link status coming up and down and what's been configured and everything sort of just puts itself together. So the last thing to go over and show you and I will be presenting the Mac OS X system in a few minutes.

Left side is a list of the services as they've been configured. So this is the information that comes from the network setup, from the system preferences network pane. It's the information that's stored in the preferences.xml file that was loaded into the dynamic store. And the right side is the current status of the network interfaces. So here we've got the built-in Ethernet. It's config method was manual. There's the addressing information. And the airport was configured for DHCP. The built-in Ethernet state goes over and has the same addresses that were requested. You can see the link status is up.

The airport interface also is up because we've got an airport within range. These are the addresses that were picked up from the DHCP server and the router that came with it. Also up on the top here we go over and see which location we're currently set to and the service order. So service order ID 1 is going to be the primary interface if it's available. And service ID 1 is the built-in Ethernet and you go over and see that since the Ethernet is up it is the primary interface.

If I go and actually change the location, I'm going to switch this to be, instead of being the land having priority with the airport following, I'll switch it to... The airport interface having priority with the LAN. And you notice that the primary interface changed to EN1. The nice thing about watching this is that if you go over and say Atlanta's got priority over the airport, and you go over and unplug the cable, if you watch the primary interface, which is currently the Ethernet, when I disconnect it, It sees the link status go down, waits four seconds, and the primary interface switches.

And if you plug it back in again, it goes over and reevaluates what needs to be primary. Interface comes up, you've got new addresses, it goes over and switches state. and right before you get on the plane, you go over and turn off all your network services, everything gets turned off. So, that's what I've got. Thanks, Allan.

So what do we have in terms of API today? Well, what we have is we have a system configuration framework. It contains mostly low-level APIs to access the dynamic store and preferences. We have some higher-level APIs. We have reachability APIs that parallel the OT will dial APIs that I'm sure some of you are familiar with.

As I mentioned when I started out, all of these APIs are currently private and we are working very hard to make some APIs public. What can you, what can, so one of the things that we realize is that Since we have mostly low-level APIs, there's not much in the way of isolation between you and the preferences format or the dynamic store format. So you're kind of running pretty bare to the metal in terms of knowing exactly how we've implemented things.

So we'd like to implement some more higher-level APIs to provide some isolation here so that you avoid having schema-specific information embedded in your code. The other thing is to avoid duplicating code. So we're definitely viewing this as something that's going to evolve and grow as needs become apparent to us and we can provide some common APIs that are useful to many developers.

So, the networking, some related sessions that actually one of them has already happened, that was this morning, the network overview. We're talking about network protocols and APIs. The other one, if you want to get more information on core foundation, there's a core foundation overview over at the Civic Auditorium across the street Friday morning at 9.

So who can you contact if you have questions about this? Thomas Weyer is the man. He'll be back up on stage in a couple more slides. If you want to take a look at the APIs that we have today, you're welcome to go to Darwin and check out the source code. There are four projects that we have that are implementing pretty much everything that I've talked about today.

ConfigD contains the System Configuration Framework, which contains APIs for the dynamic store and the preferences. And it also contains source code for ConfigD, which is the configuration server. The other project to check out is ConfigD plugins. This contains most of the agents. It contains the IP monitor and the preferences monitor that I talked about today. The third project is BootP. BootP contains the IP configuration agent. And PPP contains the PPP controller.