My name is Wilfredo Sanchez Vega and I am here to talk about the Darwin Calendar Server which is also known as the iCal Server. And I'm pretty jazzed up to be up here in WWC talking about new products again and open source things so hope you're as excited as I am.

The Darwin Calendar Server is Apple's calendar sharing solution. So we've been getting a lot of requests for years and years now about why it is that Mac OS X Server fills almost every need that you've got and wouldn't it be swell if it had some kind of calendar solution and finally we've got something. Thank you.

So to speak about CalDAV and iCalendar and so on is Cyrus Daboo who is a colleague of mine on the Mac OS X server team. And Cyrus is one of the authors of CalDAV specifications so he's pretty well suited to take it from here.

Okay, thank you Wilfredo. I'm Cyrus Daboo and I'm here to talk to you about two main topics today. The first of these is going to be a brief introduction to the calendaring and scheduling standards that Wilfredo mentioned a short while ago. And then I'm going to go on and give you a little bit more detail about the CalDAV specification.

So first of all, iCalendar. Well, iCalendar is a data format that's defined by the Internet Engineering Task Force RFC 2445 specification. This was actually originally published back in 1998, so it's been around for quite a while now. And there are many products that actually implement this standard and they allow you to import and export data in iCalendar format, etc. And of course, iCal has been able to do that since day one.

The ITIP specification, ITIP stands for iCalendar Transport Independent Protocol, is a specification that defines how you can use the iCalendar data to actually carry out scheduling operations between multiple parties in a calendaring system. So ITIP defines a set of methods and types that define how people actually do scheduling processes. So it defines certain roles for people, an organizer, an attendees and actions that they can perform.

So an organizer can send out a request to the attendees, it's an invitation. The attendees can look at that, can decide whether they want to attend or not and they can send back their replies to the organizer who can then aggregate and update all of that information for the other attendees as well. Together, these two standards basically provide the overall standard for calendaring and scheduling. And again, a number of other products already support ITIP through the email mechanism that Wilfredo mentioned, which is the IMIP profile.

So it's a very simple way to do that. But what has been lacking until now has really been a formal way to do an access protocol to actually access the data from the client in a typical client-server architecture. So that's going to be CalDAV, which I'll talk about a little later on.

So let's look a little bit at what iCalendar gives us. So iCalendar basically defines a set of events, tasks, and journal entries in the iCalendar data format. The start and end times which you need in any calendaring specification can be specified in various formats. So you can have floating time events, you can have times that are fixed to UDC and you can have times that have full time zone support.

And since a lot of us these days travel across multiple time zones, having full time zone support in your product is actually very important. And we also need to take care of things like daylight savings times, transitions and so forth. So time zone information encodes all of that for us in iCalendar. Another thing we need to be able to do with calendaring data is to be able to have events that recur over a period of time. So in iCalendar you can actually define a set of recurrence rules which actually can be very sophisticated.

And what you also need to be able to do with recurrences and recurrence patterns is to define exceptions to those. So sometimes you may have a meeting which normally takes place on a Wednesday but then one particular week you may need to cancel that for some reason. Or another week you may need to actually move it to the Thursday. So the iCalendar data format needs to be able to allow you to specify exceptions to recurrences as well.

Since we want to do scheduling, we also need a way to specify attendee data so that we can invite people. We can determine the time zone. We can determine what their status is. So an attendee will be able to specify whether they've accepted a particular invitation, whether they've declined that invitation, or they may want to delegate their presence in a particular meeting to another attendee. So all of that can be done within the iCalendar data format.

So let's look at iCalendar in a little bit more detail. I'm going to show some of the syntax that you will see in an iCalendar file. So iCalendar defines a set of top level component types. So there's a VEVENT component which defines events. There's a VTODO component for tasks. There's a VTIMEZONE component that encodes time zone information for the different time zones across the world. And that also includes daylight savings times rules. And don't forget that those are changing next year in the US, so be careful of that.

We also have VALARM components which are usually embedded and attached to VEVENTS and VTODOS. And these let you specify alarms that can be triggered a certain time before or after an event is due to start. The VJOURNAL component, well back in 1998, blogs were not being used or were not really around.

And VJOURNAL was kind of designed as an equivalent to a kind of a blogging mechanism. It's just a way, you know, typically if you have your, you know, the old days when you had, you know, your diary that you'd scribble in, this is what the journal entry was supposed to be. It's a way of taking notes during a meeting and just putting those on your calendar and tagging them with a date and time. Journals have not been particularly implemented widely in products, so that particular aspect of iCalendar hasn't been used very much.

Though there are some specialized uses of it that are actually beginning to appear recently. The VFREE BUSY component is a way for you to specify busy time information or to block out time on your calendar without actually specifying exactly what it is you're doing during that time. It's also a way for you to transport free and busy information to other people just to let them know that you're busy for a particular block of time, again without actually giving away information about what it is that you might actually be doing at that particular time.

So let's take a look at iCalendar syntax. I mean, these days you can pretty much go to the Internet and pull down an iCalendar file from many different calendar sharing sites. So you'll pull down a file which will typically have this .ics extension. And if you open that up in a text editor and look at it, you're going to see something that looks a little bit like this. So usually at the top of the file, you're going to have this begin vCalendar statement. And at the very end, you're going to have an end vCalendar statement.

Within that, you're going to have begin vEvent, end vEvent, begin vToDo, end vToDo. So the top-level object, the vCalendar object, can contain many of these different component objects, vEvents, vToDos, vJournals, et cetera. Within each of the actual components themselves, there are a set of properties that actually define the specific information about that particular component. And properties have a name, a set of parameters, which are optional, and a value. So there's an example of a property. Here, which is actually the start time property. And the name of this one is dtStart.

It has a parameter which includes a name and a value, the name being tzId and the value of that parameter being usEastern. This happens to be the time zone that this particular time is relative to. And then there's a value for this property as well. So if we take a look at an entire iCalendar file example, this is what you would typically see.

So the entire iCalendar object is specified between the begin vCalendar and vCalendar. Within that, in this case, we have two components in this particular file. One is a vTimeZone, the other is a vEvent. And within each of those components, we have the properties that actually describe the component information.

So let's go on and take a little bit detailed look at recurrences because recurrence is usually the biggest problem that people run into when they're actually looking at the iCal specification. It's one of the harder aspects and has been an area where interoperability has been somewhat dubious of late.

But there are efforts underway to actually clarify the standards and fix some of the problems that have actually been observed in the wild between products interoperating and so on. And right now the IETF actually has a Calcify working group that is actually looking at revising these standards that have been around for quite a while and producing updated versions of these documents that fix most of the known interoperability problems.

So you can make events occur more than once through the use of recurrence rules. There are two ways of typically specifying recurrences. One is as basically a set of fixed dates and the other is through a use of a pattern. Now patterns can either be bounded, so I can have a fixed set of dates set to occur for five times or they can be unbounded as you would typically want to do for anniversaries, your birthday or holidays, etc.

Simple patterns might be every Monday or every day for five times. There are examples of the iCalendar syntax for both of those. But let's look at some examples in more detail. So let's say I want to specify a recurrence pattern which is every day for five times. The way we do that in iCalendar is we have this RRule property.

Within that we specify a frequency that says daily, so that's our interval. And then we limit that to five instances with this count property. Another example, every Monday in March. Again, we're going to specify this as a daily recurring meeting, but now we're going to say only on Mondays and additionally we're going to say only in March. So that gives us every Monday in March.

Now with recurrences, you usually need to specify a start time, which is the very first instance of the recurrence set. So in this case, I'm going to specify a birthday. So the actual start time is going to be the very first birthday you might have. And the R rule is just going to say repeat this every year.

The other way of specifying recurrences is as a set of fixed dates. So if you have a recurring event that's maybe too complex for the actual recurrence rules that iCal specifies, you may need to set that up as a specific set of dates. So an example of this would be the Easter holiday, which has a very complicated set of rules for determining when it actually occurs.

And the iCalendar R rule syntax isn't actually able to cover all of those possibilities. So in that case, you could actually use R dates. And in this particular example, we have just a simple set of three dates, which are three days during August that we wanted to specify.

Now you can do more complex things with iCalendar recurrences. So this is an example of a complex pattern that iCalendar enables. So I'm going to use the example of payday, which I'm going to define as the last weekday of the month. And hopefully everybody has at least one of these on their calendar every month.

So with this particular rule, we're going to let it occur monthly. But what we're going to do is specify that the rule only occurs on the last Monday, Tuesday, Wednesday, Thursday and Friday in any particular month. And then we're going to use this by set pos element to actually pick the last one of those in the set that's generated by those instances. So the recurrence rule expansion process is actually quite complicated. There are a whole set of elements that can occur.

Within a rule, in some cases, those elements can actually expand the set of recurrences for a particular frequency interval. In other cases, they can actually restrict the set of occurrences, as we saw with the by set pos, which restricts you from a possibility of the five weekdays down to the one that actually matches the recurrence that you want.

So actually calculating these is not a trivial process and actually takes a fair bit of CPU power and effort to do. It's good to know that there are quite a lot of There are a number of open source iCalendar libraries out there that actually let you parse and generate and interpret iCalendar data. So if you need to incorporate this technology in your own products, you can certainly look at those.

In addition to being able to specify just the plain recurrence patterns, as I mentioned before, you also need to be able to specify exceptions. One type of exception is an exclusion. So in this example, we're going to specify every week on Monday for five weeks, except that we're going to eliminate the third instance of that. So in this case, we just specify the starting Monday for the recurrence set. We have an R rule which specifies weekly for five times, then we use a special X state property to exclude the third one of those from that set.

Another type of overridden recurrence is the exception. This is a situation where we actually going to, rather than eliminate a particular meeting, we're actually going to change it from one time to another. So in this case, I have an example where I've got a meeting that's taking place on Monday, Wednesday and Friday. Now what I want to do is move the Wednesday meeting to a different time on Thursday.

Now in iCalendar, the way we do this is to actually add a new VEvent component that defines this overridden instance on Thursday and link it back to the original component that defines the original set of occurrences. And with the iCalendar semantics defined by the specification, the iCalendar clients will know that that overridden instance overrides the original one from the master instance and will give you the correct view of your events with that overridden instance in the appropriate place.

But it's worth noting that particularly when it comes to CalDAV, a CalDAV server, as we're going to see, needs to understand recurrences and iCalendar data because it has to do a certain amount of iCalendar processing on the server. So in addition to the client having to understand this information, the server needs to do so too. So let's actually move on and take a look at CalDAV. CalDAV So why did we need CalDAV? Well, there were previous attempts to define a calendar access protocol within the IETF.

The original CAP protocol, which stood for Calendar Access Protocol, was a completely new protocol that was designed to do as much of calendaring and scheduling as possible. As a result, it actually turned out to be way too complex and way too difficult for most people to want to implement. So there was very little interest and buy-in from vendors to actually implement this specification. And of course, if you have a standard that no one implements, well, it's not really useful to anyone at all.

The other problem with this is of course calendaring and scheduling itself is actually complex. I'm only touching on a very limited view of what's actually here. If you actually go and read the specifications, you'll see the level of complexity that's there. Particularly things get very complex when you start dealing with multi-party scheduling where you do have recurring meetings over a period of time. You do have exceptions to those meetings. You need to invite different people to certain instances of a recurring event and not others.

There's a lot of complexity there. What we wanted to do was to try and remove from the access protocol the complexity of the protocol or transport layer itself. Why not build on an existing transport protocol? Obviously, HTTP is a good transport protocol. It's used for a number of different applications. HTTP was a good choice, but we needed to be able to do a little bit more with that. As Wilfredo mentioned, we need to be able to author calendar events online.

We wanted to be able to do a lot of that on the server as well as just retrieve them from the server. WebDAV is the authoring specification, and we wanted to utilize that. Then, of course, we want to stick with iCalendar and iTIP itself so that we can leverage the knowledge and the libraries that people have already developed for iCalendar and iTIP.

So HTTP, again an ITF standard. Most people will see HTTP because that's the way you browse the internet. But it's also used in other applications. Web dev, as I said, SOAP, XMLRPC, et cetera. WebDAV adds a structured and editable data model to HTTP specifically to enable collaborative authoring of documents on an HTTP server. So one of the key things that it adds is this concept of collections, which is the equivalent of a directory in your file system. It allows you to group resources or documents into subfolders on the server and organize the hierarchy of resources on the server.

So WebDAV added this MKCOL method to allow you to actually create collections. It provides a way for managing resources by adding copy and move methods, which lets you move or copy resources between different collections. It adds a mechanism of allowing you to add metadata to resources that are on the server. So typically the server will store metadata on resources such as the last modified time, the creation stamp, the owner, HTTP specific information such as the email.

It adds a way for you to add tags to a resource, which is the e-tag of the resource, which is a synchronization token that you can use with HTTP. But clients can also define their own properties, which they can store on the server for information that they need for their own purposes. So if you need to have a property that adds tags to a resource, you can add that through a WebDAV property.

In addition to allow multi-authoring of resources on the server, you need a way for one user to lock a resource to prevent another user from coming on and coming in and trying to edit that and maybe overwriting their changes. So again, WebDAV defined new lock and unlock methods to help with that.

In addition, there's another specification that adds access control to WebDAV. And access control is really the key to allowing end users to manage their own documents. Now, typically with an Apache server, you can manage access control by going into the server and editing your .htaccess file and giving people privileges that way.

But typically, it's only the system admin who's going to be able to do that. What we want is a way for users through the HTTP protocol to actually manage sharing of the documents themselves. So that way, I can go in and grant Wilfredo access to a particular document I have on the server. And I can do that directly through the WebDAV HTTP protocol. So that's what the WebDAV access control protocol gives us.

There's also the version control specification, which allows resources that are stored on the server to be put under version control. So a full history of all of those resources is stored on the server. You can look at previous versions of a particular resource. You can update resources and do merge. You can do merge and so on. And in fact, the version control features are used by some source code control systems, such as Subversion today.

So let's actually look at CalDAV itself. So CalDAV is a calendar access protocol. It's designed to address real calendaring requirements. So one of our primary goals when going into the process of writing a specification is to actually take a look at how people are using calendaring and scheduling today in the proprietary products, custom systems that have been built and to try and leverage some of that knowledge to build the specification in a way that's actually going to address a lot of the use cases and requirements that people have. in the real world.

Right now we have two specifications. There's the calendar access protocol which allows you to read, write and search calendar data on the server. This particular specification is in the final IETF approval process and we expect to see it published as an RFC in a couple of months or so.

Then we have the calendar schedule specification. This enables the process of scheduling on a CalDAV server. So it allows you to exchange scheduling messages between users on the same server. This particular document is close to IETF last call and again we don't expect there to be any major changes in this and in fact we, our server and the iCal product have both implemented the current version of that specification.

In addition to this, we're expecting to have another specification that defines how you do server-to-server scheduling operations. So we need a way to be able to schedule between different organizations that may have different servers, different types of servers even. So the third document in this series will be a document that defines how to do that. One of the things we have to be very careful of with the server-to-server protocol is ensuring that we do this in a secure and safe manner.

Obviously, we want to avoid the horrible threat of having calendar spam appearing on people's calendars. Unfortunately, in this day and age, it's more than likely that that's going to happen. So we need to take precautions up front to help prevent that type of thing. A lot of the work on CalDAV and the requirements for server-to-server protocol, etc., is actually being carried out under the auspices of the Calendaring and Scheduling Consortium, which you've heard a fair bit about this week already.

So if you're interested in a lot of this work, you know, please try and join in in the CalConnect Consortium and help us define the requirements and needs that you may have. And in particular, we're not just looking for developers here. We actually want end users who have their own requirements and needs to actually come and give us input as to what they need in a protocol such as this.

So who's actually doing CalDAV right now? Well obviously apart from Apple, there are a bunch of other vendors, both commercial, open source, and education sites that have been developing and have released CalDAV products in various forms. These range from clients and servers to a free busy aggregator that sits on top of a proprietary system.

So let's look at CalDAV in a little bit more detail. So in CalDAV, calendars are actually represented by WebDAV collections. CalDAV actually defines a new MK calendar method that actually allows a calendar client to create a special calendar collection on the server. The events and tasks in the calendar are stored as individual resources within that calendar, with the exception that any events that represent a set of recurring events-- so in that previous example I had of exceptions where you added a particular event that was related to the master recurrent set-- those two events, the master and this exception, would actually be stored in a single resource on the CalDAV server. And that's just there for convenience, because typically you're going to want to download both of those together when you actually want to process that event.

So let's look at a typical server repository layout. So on a WebDAV server, in this case a CalDAV server that we have here, we're going to have two main areas on the server. The first of these is going to be the principles hierarchy. This is the location where accounting information about users, groups and resources is going to be stored. Then we're going to have a calendars portion of the hierarchy which is where the actual calendar data lives.

So with the WebDAV Access Control Extension, every user on the system is represented by a principal resource. So in this case, we have principal resources for Cyrus, Chris and Wilfredo. These principal resources are the identifiers that you actually use in the Access Control Extension to specify who gets access to which particular resources.

In addition to be able to specifying individual users, of course you want to be able to specify groups of users. And again, the principal resources that define groups allow you to use those in access control so you can grant rights for a particular group to read a resource.

And all of the users who are members of that group obviously inherit the ability to read or write based on the privileges. Now in calendaring, we also need to include non-human resources in the calendaring process. So rooms, printers, all of these need to be bookable as part of the invitation. So again, we will have principles related to those particular types of resources.

Within the calendar data hierarchy, again, we will have three subgroups for users, groups and resources. Within a particular subgroup, we're going to have the calendar home directory for each particular principle. Here we have calendar homes for Cyrus, Chris and Wilfredo. In this example, Chris has actually got two calendars within his calendar collection. CalDAV does allow you to have as many different calendars as you want under your own calendar home directory or you can create calendars in public hierarchies if they're set up as well.

In this example, Chris has two events in his work calendar. It's worth remembering here that we're still dealing with an HTTP server. So every one of these resources and collections is individually addressable by an HTTP URL. So at any time, you can just even pull up a web browser and just browse this entire CalDAV hierarchy assuming you have permissions to look at the various portions of that hierarchy.

One of the key components of the CalDAV protocol is the other reports. The reports are the way that you actually get access to the calendar data in a smart way. So for reports, we use the WebDAV report method and this gives us a way to do calendar specific queries on the data on the server. Reports typically take an XML request body and return an XML response.

CalDAV defines three types of reports. There's the calendar query report, which allows you to query calendar objects on the server that match certain criteria. And I'm going to go into this in more detail in a minute. There's the calendar multi-get report, which allows you to get a whole set of events or tasks from the server in one single request.

So rather than having to do multiple requests for each of the individual events in a particular calendar collection, you can ask the server to give you all of those in one go. And that just saves you. So you can do a little bit of bandwidth and round trips, which is particularly important for mobile devices.

Another report is the free busy query. With the free busy query, it's possible to grant someone the right to get free busy information about your particular calendar without giving them the right to actually see the data in that calendar. So I can grant Wilfredo the right to run a free busy report on my work calendar, but he won't be able to, if I don't give him the right to actually read that calendar, he won't be able to see. The details of the events, he will just get information that says, I'm busy from this start time to this end time, et cetera.

So let's look at the query report in a little bit more detail because this is the report that most clients are going to want to use for their day-to-day usage of CalDAV. So with the query report, clients can specify a set of search criteria. You can match the text in a particular value in the calendar data on the server.

So you can find events, for example, that contain WWDC in their description. You can match by a time range, so you can find all events that are on your calendar this week. You can match by particular component types, so you can find just events or just to-dos or events that only contain alarms and so on.

In addition, the client can actually specify what gets returned by the server. So the client can request that the server, in addition to returning the calendar data, it also returns WebDAV properties that may be necessary to help interpret that information. The client can also request either the entire calendar data be returned or only parts of the calendar data. Now, it's worth remembering that with the iCalendar specification, you can include attachments in your calendar data that you store on the server.

Of course, these attachments these days can be very large if you're storing large keynote presentations or pages, documents, and so on. So from an efficiency standpoint, you don't want to have to keep downloading attachment data when all you really want is the start and end time of a particular event.

And CalNav allows you to do partial fetches of the data on the server. In addition, it's possible for the client to ask the CalNav server to actually expand recurrences on its behalf. So again, we might have mobile devices that are on the server, but we don't want to have to do that. So again, we might have mobile devices that are on the server, but we don't want to have to do that.

Let's actually look at what one of these reports might look like over the wire. So this is again a typical HTTP request response type process. So here we're going to be sending an HTTP report request. The report, as I said before, is going to take XML data in the request body. The request itself is going to include XML that says find events on my calendar. And in this particular case, we're looking for events that occur between a specific start and end time range.

So the response from the server, this is going to be a multi-status response. Multi-status is a special response defined in WebDAV that allows the server to return information about multiple resources in a single response. So in this case, we actually have a single matching component where we have XML data being returned and within that we have the actual iCalendar data that the client requested and that matched the search criteria. In this case, one event was found.

Let's look at the second part of CalDAV, which is the scheduling portion. What we wanted to do with CalDAV scheduling is support all of the modes that the ITIP specification defines for carrying out scheduling. This requires, amongst other things, this requires the server to really understand the concept of calendar user addresses, which is the means that you actually use to identify users on the calendaring system and target invitations at those particular users.

In addition, iCalendar and ITIP define the roles of an organizer and attendees. So there's going to be a certain amount of asymmetry in the protocol depending on what your role is. If you're an organizer, there are certain processes you need to go through and tasks you need to do to initiate an invitation. If you're an attendee, there are certain steps you need to take to process that invitation.

In addition to this, we definitely need access control and access privileges for scheduling. You want to be able to control who is able to invite you to a particular meeting, whether you reject invitations from certain people. You also want to be able to delegate the ability for someone to schedule meetings on your behalf. So if you have an assistant who is able to book meetings on your behalf, you want to be able to grant them the right to do that through the protocol.

And again, we've leveraged WebDAV access control to allow you to do that. So let's look at this in a little bit more detail. The CalDAV scheduling specification defines for each user on the CalDAV server an inbox and an outbox collection that are actually used during the scheduling process. And to set the access control privileges for determining who can schedule with you and who's able to schedule on your behalf, you actually set access privileges on the inbox and the outbox resources.

Wilfredo Sanchez, Cyrus Daboo In addition to this, we have a lot of different ways to schedule meetings. We have a lot of different ways to schedule meetings. We have a lot of different ways to schedule meetings. We have a lot of different ways to schedule meetings. You can schedule meetings with your friends, you can schedule meetings with your family, you can schedule meetings with your friends, you can schedule meetings with your family, you can schedule meetings with your friends.

So scheduling requests or replies are deposited by the organizer into their own outbox. That triggers the server to actually send the requests or invitations to the intended recipients. The organizer gets back an immediate response from the server that says, this particular invitation was delivered successfully to the recipient's inbox.

The recipients themselves will actually see those invitations delivered to their own inboxes. And in their own time, they can pick up those invitations, they can look at them, they can process them, they can determine how to respond, and they can respond by depositing their response in their own outbox, and that gets routed back to the organizer.

There is one exception to this, and that is the case of free/busy information. When we were looking at the calendar in the scheduling, it was very clear that one of the biggest pain points in the whole scheduling process is actually determining free/busy information because that's really the first thing that people do when they're actually putting together an invitation.

You need to know, you know, when can I actually book this meeting? What is a suitable time? And the ability to look at people's free/busy information in almost real time was a very important key requirement. So in order to enable that, we defined the CalDAV scheduling specification to return free/busy information immediately to the end user. So the server calculates this on behalf of the organizer and returns that information immediately. And we'll see an example of this in a minute.

So just graphically to represent what happens in this process, we have an organizer and an attendee. The organizer adds the scheduling request to their outbox. The server sees this being delivered to the outbox and takes copies of that and puts that in each attendee's inbox. At some later point, the attendee will see that in their inbox.

They'll be able to pick it up. They'll be able to process it. And if they need to send a reply, they can add that reply to their own outbox. Again, the server spots this new reply in the outbox, picks it up and delivers it to the organizer's inbox.

And the organizer can pick up that reply and update their local copy of the event with the new attendee information that was provided. Another way to look at this is what happens when you have more than one attendee. So again, we have an organizer that's sending an invitation to the server. In this case, the invitation actually lists four different attendees. So again, the server is going to pick that up and deliver four copies of it to each attendee's inbox on the server.

Each attendee in their own time is going to be able to see those invitations and they're going to be able to send them back to the organizer. And the organizer will get four copies or four status, four replies from each of the attendees. They'll be able to update their own copy of that. They'll be able to push out updates to other attendees to notify them of the status change of other participants in the event and so on. This whole process is typically called fan out in the calendaring and scheduling terminology.

So let's take a quick look at our scheduling request on the wire again. So for schedule requests, we actually use the HTTP post method. And in this case, again, we're targeting the outbox of the organizer. The request itself is actually iCalendar data. In the HTTP headers, we include addressing information. So this is the information about who is the originator of the message, which may differ from the person who's organizing the message.

So if you have a delegate, someone who's acting on your behalf, they will be listed as the originator, though you as the organizer will be listed in the iCalendar information as the organizer. So in email terms, you know that there's a difference between the from address and the sender address, et cetera. This is an equivalent type mechanism.

Within the actual iCalendar data, we have an ITIP request. In this case, it's an ITIP event invitation. And the invitation itself is for dinner on August the 9th. The reply that the organizer gets back from the server, again, it's going to be hopefully an okay response. Again, it's going to be XML data. And encoded within that response is going to be information about whether the event was successfully delivered to the attendee's inbox or not. If the attendee had disabled my right to schedule with them, I would get a negative response back at this point.

Luckily, in this case, it was successfully delivered, so you get a success response. Let's take a look at the FreeBusy case, which differs from the invitation process. So again, with FreeBusy, we do a post request on the HTTP server. Again, we provide iCalendar data. Again, we provide the addressing information. This time, however, we have an ITIP FreeBusy request. And we're going to be asking the server for the response to the freeBusy request.

And we're going to be asking the server for the response to the freeBusy request. Now, the response we get back from the server looks similar to what we got back before, except that now the actual iCalendar FreeBusy information is included in that response. So we don't need to wait for the attendee to hit reply in their client. The server automatically determines the FreeBusy information for that attendee and returns it in virtually real time to the organizer.

And in this case, the attendee was just busy for an hour. So where are we with CalDAV today? Well, there's been a lot of interoperability testing carried out between the various people that have CalDAV products. This actually started back in January 2005, and we've been holding about three interrupts a year since then. Again, it's been carried out under the auspices of the Calendaring and Scheduling Consortium. The next event, interop event, by the way, is actually being hosted by Apple at the end of September down in Cupertino.

There was a recent free busy demonstration that was done, again, put together by the CalConnect organization. And that actually showed several different, in some cases, proprietary calendaring and scheduling systems actually leveraging CalDAV to do cross-organizational free and busy lookups. So CalDAV was effectively acting as an aggregator for proprietary systems, aggregating free busy information from these and providing a standard CalDAV interface for clients to actually access that free busy information. That demonstration was done about a month and a half ago and was very successful. And again, we're going to be building on that in terms of the server-to-server protocol to actually build out this new server-to-server specification. Now, it's fair to say that CalDAV is interesting.

It's really still in its infancy. And there are certain key elements of a true calendaring scheduling system that are missing right now that still need to be specified. Obviously, as I discussed before, we need to do this server-to-server protocol. In addition, HTTP does not really have a way for the server to notify the client of changes on the server. So we need a way to do asynchronous notifications so that the client doesn't have to continually poll the server to look for changes. So that's another element that we need to specify. And that's more important.

And we're going to be looking at more of an HTTP issue as well. So it's going to be done within the HTTP community rather than just the CalDAV community. We also need a way to do smart and efficient collection synchronization. So again, trying to minimize the bandwidth and the data that clients need to exchange with the server so that we can be efficient on that. And again, this is looking at not just to benefit desktop clients but also mobile clients. But we really want to see a good adoption of CalDAV in the mobile arena as well. And the other thing that we would like to see.

Is the ability to do partial updates of resources. Before when I described the partial fetching of resources. You could download part of an event and avoid having to download the attachment. But one thing we're lacking is the ability to change, say, just the start time of an event without actually having to upload the entire attachment along with it. So again, a way to do partial updates on an HTTP server will be good as well. So with that, I would like to actually turn back to Wilfredo and have him come up here. And give you a. an in-depth look at our CalDAV server.

Thanks Cyrus. So now that we know all about how to implement a calendar server, fortunately you don't have to go off and do it because we did one for you and you can use it. Again, the Darwin Calendar Server is an HTTP server because you have to be an HTTP server in order to be a WebDAV server, which it is also, and you've got to be that in order to be a CalDAV server, which of course is the end goal.

Our server is written in Python. It is based on the Twisted Framework and it's configurable using Open Directory. So let's get into some detail here. Again, the server is written in Python. It's a dynamic, flexible, object-oriented programming language. It's been very convenient for us to code in this and hopefully you'll be hearing a little more about that tomorrow when Bill talks about how you can write Cocoa applications using Python and so on.

Leopard includes Python version 2.4, which we're actually using in our server, although it probably wouldn't be too hard to backport it to 2.3. A common question I'm getting when I mention Python is what's the performance of this scripting language, right? It's not C, so it's not going to be super fast, right? And it turns out that actually Python's not a bottleneck. There are plenty of examples of high-scale Python implementation, such as like eGroups from way back when and a bunch of things like that.

The particularly CPU-intensive work is actually still done in C, such as SSL handling, which talks to the OpenSSL framework. But we fully expect our server to scale to moderate to large organizations. We are not building something that you could deploy as a Yahoo or a Google, because, wow, that's hard. But for your business, we should be just fine.

Again, this is built using the Twisted framework. Twisted is an open source networking framework and that's of course written in Python. It provides for us a lot of functionality that we already needed in order to get started. And it implements a bunch of protocols such as IMAP and POP3 and NetNews and SSH and particularly of interest to us, HTTP. It's got a very active development community. What's interesting about Twisted is that it's, again, asynchronous, but it does this without using any threads.

So all the headaches of thread management and the bugs and the endless debugging that you do with threads, we don't have to worry about because we're not using threads. But at the same time, it can field multiple requests at the same time. So when you have a blocking operation, such as you're going to write to the network or you're going to read from a file or you're going to do something that's going to take a while, rather than writing a method that does that and then returns an answer, what you do is you schedule that operation and then you return a promise to give the caller an answer. So that's called a deferred object.

And a deferred object doesn't necessarily know what data you are going to read right away. So as the caller, you get back this object and then you chain onto it a callback, which once the answer is available, that callback gets called and then you can continue your processing from there. So it's a somewhat different way to write your software, but it's actually fairly effective at doing this asynchronous operation without a whole lot of grief. - Yeah.

It's actually pretty cool. So Twisted Web 2 is a submodule within Twisted that implements HTTP. In particular, it does it by modeling a resource as an instantiated object so that whenever you're going to field a request, the server gets a request for some URL. It figures out which resource is responsible for handling that URL. It instantiates that resource and then it calls a method on that resource which corresponds to the HTTP method which was called in the request.

So in a typical GET request, you're going to create some resource and that resource is going to get called with a method called HTTP_GET and that method will get a request object with it and it's going to do some processing, probably read a file and render out some HTML and then it's going to return a response object or very likely a deferred response object that then the server, once that deferred fires, will actually give a response back to the client.

Twisted WebDAV is a submodule of Twisted Web 2 and that implements WebDAV. This was actually written by Apple, me mostly, and contributed to the Twisted project as part of open source stuff. What's cool about Twisted WebDAV is that it leverages all of the stuff that was in Web 2 to do all the HTTP stuff and then it adds in WebDAV functionality.

It supports WebDAV level 1, which is all of your typical get, copy, move methods, put, which is actually an HTTP method, but only WebDAV client servers tend to implement it, and all your regular WebDAV methods. It supports WebDAV ACLs, which is important for the scheduling stuff, as Cyrus was talking about.

You need to be able to grant permissions to have certain kinds of access on specific resources and it implements that specification. We're starting some work on... WebDAV level 3, which is the next revision of WebDAV is actually in... being written right now in its final forms and it's just a small update to WebDAV level 1.

It does not support WebDAV level 2, which is the lock and unlock methods. So it's not really suitable for, say, mounting in Finder, but lock and unlock are not required for CalDAV, so we kind of skipped over that for the time being in order to get the calendaring implementation done. And it does not support all of WebDAV versioning. It does support the report method, which is specified in the versioning spec because CalDAV needs a report method.

So Twisted CalDAV is a module that builds on top of Twisted WebDAV, which builds on Web2. So you can see how this stuff starts to layer fairly nicely. You just start subclassing a lot. And in this case, we add all of the CalDAV functionality. In particular, we're implementing CalDAV access, which means we have to implement the make calendar method in order to let you create calendar collections, which are somewhat unique from regular WebDAV collections.

It has a lot of business logic behind it for doing calendaring operations. So it has, of course, an iCalendar parser and an iCalendar generator so that when you edit stuff, it can send you back useful data. It supports all of the CalDAV reports that Cyrus was going through in his examples.

It supports the read-free-busy ACL privilege, which says that on a calendar collection, I can grant somebody permission to see whether I'm busy, but without giving them permission to actually read why I'm busy. And that might be particularly important. For a lot of us. And it enforces good behavior from clients.

So you can't like have some Yahoo crazy, not Yahoo the website, but some crazy client come in and then start writing random data that then confuses iCal or whatever other clients that are well-behaved out there. So if you try to write JPEG information into what you're claiming is an iCalendar file, it's going to say, well, hold on, that's not iCalendar data and refuse the request. It also implements CalDAV scares.

So it's a great tool for scheduling, which means it does all of this inbox and outbox fan out that Cyrus is describing. You can park data into it. It'll automatically forward it along to other users and then they can pick it up from there. It supports all of the scheduling requests and it knows about itip and some of this logic. Again, it's all calendar and business logic. So it's not just webDAV, it's webDAV with some calendaring smarts. It returns free busy responses right away.

So it actually will when you schedule and I want to know whether Skip is busy. I look at his calendar. The server actually reads all of his scheduling information, aggregates that into a free busy response that's useful. Again, if you have that privilege and it shuttles it back to the client that's requesting it.

And it has scheduling specific access control privileges. So you can, again, the concern being that you don't want calendar spam. You don't want anybody to be able to propose meetings with you. You want to restrict that to some subset of people that you know are sane. So you can do that with ACLs.

So looking at the implementation, you'll see that our data storage is actually all file based. There's no SQL massive database that stores all this calendar information. We're just writing information out to file. So the resource layout that you see in your web server is actually what you see on disk when you look at the data store.

And so it's a lot like managing a typical Apache install or regular web server. What's nice about that is that it's simple. If you want to just see what's going on, you can see the into the documents directory. You can kind of poke around in there and use tools like CAT and CP to manipulate the data in there if you need to do that. And you don't need special database specific tools.

and that also has the advantage of giving you like some flexibility in how you do your backups and restores and the like. Properties are stored in files as well. They're stored as extended attributes. So if you think webdap properties kind of look and smell like extended attributes, well, in our implementation, actually, they are.

Now, in order for the performance to be good, because, again, we want this to scale to a fairly large organizational size, we use SQLite databases to index that information so that when we do a lookup, rather than having to open 1,000 files that might be in your calendar, we just say, well, let's look up where the event with this given UID lives.

And then we can just go straight to that file or actually keeps all of the time spans in there. So if I want to find out all of the events that are happening this month, I can actually use a SQLite database to kind of find which data is likely to have that and narrow my search down and then go straight to that set of files.

Um... Account information, that is which users exist, which groups exist and which resources such as rooms and projectors and the like exist, that all lives in open directory. So you manage that using regular open directory tools, your work group manager, your team's directory and all of the great tools that come out of the open directory team. And that of course has the advantage of being able to integrate with the rest of your authentication and user management systems throughout your business.

Again, we use SQLite to cache some information off of open directory so that we're not making quite so many network requests every time somebody comes in and says, oh, I'd like to schedule with somebody and I have to figure out, oh, I don't know who that is, let me go to open directory. Well, we remember some of that information for a while so that we don't have to keep doing these lookups. Again, the SQLite databases are there only for performance reasons.

So if for some reason they're annoying you, you think there might be a corrupt file, you don't know what's going on, you can just delete the file and next time around it'll get regenerated. These are all things that we can do. And if you're not familiar with them, you can just delete them. And if you're not familiar with them, you can just delete them. And if you're not familiar with them, you can just delete them.

Again, the server is configured using open directory so you can use your standard open directory tools. The way this works is if you remember the diagram that Cyrus had where you have principles and then you have calendars, the principles that are HTTP resources that are actually backed by open directory and not really primary on disk.

So we look up which users are around on your directory system and that's how we populate the users, groups and resources parts of the principle hierarchy. But at the same time, we figure that any user who's got an account on the calendar server also needs a calendar collection so we also provision out the calendaring information. And again, not all of your users in open directory have to have calendar accounts.

You might want to only have some subset of users. You might create some users that don't need calendaring. And you can enable or disable that in. So you can do that in the open directory using one of the properties there. Again, user authentication is done via open directory so your regular HTTP authentication mechanism is hooked into that. There's no crazy forms based authentication, fortunately.

And Kerberos authentication is in progress. iCal and the Leopard seed doesn't support Kerberos auth, but we plan to have that for Leopard and so that'll fit into your single sign-on strategy if you're using Kerberos. So let's talk about the actual implementation a bit. First thing is you're going to probably want to get some source code. And the way you do that is use Subversion. In the Leopard Seed, Subversion is built in. Yay.

If you're using Tiger, you can download it from some websites. We have some information on our site on how to do that. But once you've got Subversion, you can just do Subversion Checkout of that URL there, which is on Mac OS Forge, and get your own handy dandy copy of the calendar server as it is in development.

Once you've got the source code, you can change the directories into there, copy one of the example Director of Configuration Files in your development environment. In this case, we're going to suggest you use a static one so that you don't have to also set up open directory in order to test your server. You can actually, in XML, specify a set of users and just create an admin account with the test password.

And then you run this shell script that's in there called run. And that will actually finish up the rest of your downloads. It's got to download a few dependent libraries that we use. And it knows whether to fetch the right ones on Tiger versus Leopard. And then it should get you started. Now we've got a couple of kinks in this I'm finding out from emails, but this will be that easy by the time I get home next week.

And then you've got a running server so you can get Safari out, you can connect to local host port 8008 which is the default HTTP port we're using and then you can actually poke around in it using your web browser or you can launch iCal and you can poke around at it using that.

So I mentioned that we're dependent on some other software. If there's anything you take home from all of the stuff that Cyrus was telling you about iCalendar and so on is that it's pretty complicated to implement all of the iCalendar logic and you probably don't want to build that yourself and neither did we, so we stole a lot of code.

First of all, like I mentioned, we use Twisted for our core engine that drives the server. We use a library called VObject, which is written by one of the fellows at the Open Source Applications Foundation named Jeffrey, and that deals with all of this iCalendar parsing and generating, which we fortunately don't have to build ourselves.

Twisted requires a small bit of the ZOAP framework called ZOAP Interface in order to do some, declare programming interfaces, which is a feature that Python doesn't provide. We use PyXML to do all of our XML generation. If you're a web dev person, you're doing a lot of XML processing, so that's our engine for that.

We use DateUtil to do a lot of the recurrence and date processing magic, and VObject depends on that. We use PyOpenSSL, which is a thin shim on top of the OpenSSL library, as I mentioned before, which of course is written in C, so we're actually binding to that. Bobby Pulido wrote a small shim to the extended application. We use Attributes API that's in Mac OS X and on Linux and on FreeBSD, and so we use that to store our web dev properties. And we use PySQLite to store those SQLite databases I was talking about.

So we're kind of a player in a lot of little buckets. And in order to implement our open directory magic, which stores all the data in LDAP and so on, and to do the Kerberos authentication, we had to write a couple of more modules, and that's PyKerberos and PyOpenDirectory. both also on the same source repository as the calendar server.

Hopefully they're useful for things outside of the calendar server as well. So just briefly I'm going to go over some of the classes that are in the source code so that you don't quite have to figure out the architecture of it from zero. This is actually a fairly simplified version of what's going on but it should give you an idea of how we built the server. So again, Twisted Web 2 implements HTTP as a collection of resources which are instantiated. So you've got a resource class that's kind of a generic class and knows how to do some basic HTTP.

operations. And we have a static derivative of that which knows how to do things like generate an e-tag on the fly, get you the last modified time, just for resources that are not like your CGI type dynamic resources. And from that, there's a file implementation. So the file implementation, of course, you send it a get request and it's going to read some data off of the disk and just shuttle that back up to the protocol. And that's a fairly simple, typical web server right there.

In order to implement WebDAV, we subclassed the static resource into a DAV resource class and that knows how to do a couple more things that are specific to WebDAV. In particular, it has a property store. And that property store is where the WebDAV property information is going to live. And our implementation of that uses extended attributes, as I mentioned.

That in turn is subclassed in a class called DAV file. Now you notice here there's multiple inheritance. So if you're terrified of that, close your eyes. But it's actually fairly straightforward. We derive from the file class because we want to read data off of the disk and the file implementation does that and we derive from DAV resource in order to get some common web DAV logic.

And then of course, subclassing that for CalDAV resource and CalDAV file, we get the calendaring logic in there. So you can see how this stuff kind of layers very nicely from regular HTTP processing down to WebDAV processing and down to calendaring processing. And there's some other classes in there which do some WebDAV things such as like there's a DAV principle resource which knows how to instantiate a principle and what that means and you can authenticate against it. And there's a file-based implementation of that as well. So you can see how all this stuff kind of ties in together.

So hopefully like you guys are excited about some of this stuff and might want to tie it into something that you're doing. You might want to tie it to some calendaring backends that you've got already. You might want to implement it to control some other stuff. Maybe you want to implement a public event system, some Evite type thing without all the ads.

Or you want to run your sprinklers at home or change your thermostat settings while you're at work. You can imagine that a calendar server can kind of just bind to these information systems and kind of drive that stuff. And so there's some cool stuff that I think could be done there.

But also you can extend the protocol which there's a lot of interest in Atom right now because Google Calendar is doing Atom for subscribing to calendar information. So you can imagine that we could just put an Atom front end on top of that. And then you can put a calendar on top of the server in addition to CalDAF so that you can use those tools to talk to it as well.

We've obviously been getting a lot of questions about legacy systems and how you're going to tie into those because it's open source. You can actually replace the file classes that I was showing you there. Just swap those out with some data store that instead stores to some database or an existing database or speaks network protocol to some other calendaring system.

You can imagine a map back end that, for example, speaks to Exchange. change. And then you might have other directory systems that are not in open directory. You're porting to Linux, for example, and you don't have open directory deployed. So maybe you want to replace the directory back end to use your particular single sign-on strategy.

and the source code hopefully enables you to do all of these things. So if you want to learn more about this, the website is collaboration.macosforge.org. That's a track installation so you can actually file bugs in there. You can kind of browse around some documentation that we're starting to bring online. Hopefully the server is a little happier now. We were on Slashdot a couple days ago and had the associated downtime that comes along with that.

But the server seems to be behaving pretty well lately so you should be able to connect to this and get some information out of there. If you register on the thing you can actually file bugs. If you do not register you cannot file bugs and that's because we want a little less spam in our mailboxes. And you can actually browse the source code in there and see how we're progressing. There's a cool timeline feature in track that lets you actually see what wiki pages got edited, which source commits changed.

And what tickets have been filed. So you can actually see our progress in a very useful view. So if you want to learn more about all of this information, again if you want to learn about the calendar server in particular, you want to go to collaboration.macosforge.org. If you want to learn about Chandler, which is OSAF's client, you can go to chandler.osafoundation.org. There's another CalDAV implementation which is open source called Cosmo, also from the OSA foundation. Learn more about Twisted at twistedmatrix.com. And of course if you want to learn about Python, go to docs.python.org.

There's a lot of specifications in play here. Again, iCalendar's got some RFCs up in there right now and there's edits going on to updated drafts. If you want to learn about HTTP and WebDAV, there are other RFCs to check out there. And CalDAV is, if you go to CalDAV.org, that'll redirect you to the proper website for some CalDAV information. Also, if you go to WebDAV.org, you can learn about WebDAV.

And then, of course, if you're at all interested in this space and you want to be able to drive some of these standards, get involved with the calendaring and scheduling consortium. Again, Apple's hosting the next interoperability event in September on our campus. Hopefully, that's a sign that we're pretty committed to the standards process and take it from there. And all of these URLs are on the Mac OS Forge site, so you don't have to keep scribbling.