Hi, everybody. Welcome to session 520, Deploying and Scaling the iCal Server. I’ve got a good crowd. My name is Wilfredo Sanchez Vega. I’m the engineering manager for the iCal Server, and we’re gonna be talking about how you can fit this into your environment back at the office, maybe even at home.

So, iCal Server is Apple’s calendaring solution. We introduced it with Leopard just at the end of last year. We talked about it last year at WWDC and the year before. It’s all based on open Internet standards. So, we’re not trying to lock anybody into any kind of proprietary protocols.

We’ve got a very strong commitment to doing everything using open standards and we’re actively involved in driving some of these standards because they are emerging. we interoperate as a result of that with a variety of clients, including, of course, iCal. But as you’ll see, there are actually a handful of other clients that are starting to enter the space that interoperate with our server as well.

And the entire project is open source. So today, you can actually go out there and get our source code, fiddle around with it, and add whatever you need in order to make it work for you. And ideally, contribute something back. So what we’re going to talk about today is-- Some of the features of the iCal server, so you can kind of see exactly what you’re getting into if you deploy it.

Talk a little bit about what’s new in some of the software updates that we’ve been shipping since we launched Leopard. This being a brand new product and a new part of the server product, we’ve been doing actually quite a bit of work in the software updates to address the comments that we’re getting back from our users. And we’re going to talk a little bit about what you’re going to see in Snow Leopard.

Cyrus is also going to update you on the status of all the relevant standards that we’re working with. There are many. And then we’re going to get into some of the nitty-gritty about how the iCal server deploys. And the whole talk will be actually hopefully not too long so that there will be plenty of time for some Q&A at the end.

So just first up, quick review of what kind of standards I’m talking about. First up is iCalendar, which is just a text file format that describes events. It’s simply this event happens at this time. It maybe recurs every Monday, Wednesday. And when it ends, you can also describe to-do items and that sort of thing. We’ve got another format called iTip, which is an iCalendar transport independent protocol.

And that enables you to talk about scheduling a meeting. So if I want to invite you to a meeting, I can actually send you a message. It looks a lot like iCalendar. The only difference is that it adds a little bit of glue to tell you that I’m inviting you, I’m accepting a meeting, and that sort of thing.

We’ve got HTTP, which probably everybody here is fairly familiar with. But this is the protocol that drives the entire web and a great deal of the internet. And in order to actually make that useful for authoring, you’ve got to have some more stuff, and that’s WebDAV, which is distributed authoring protocol for HTTP.

That allows you to edit content on a server. We’re using WebDAV ACLs, which let you control who has access to which parts of your server. So, for example, I can read and write into my own calendars, but I can’t read and write into Chris’s calendar. And, of course, WebDAV is a heavy user of the XML standard.

On top of all of that, we build what’s called CalDAV access. And that is a protocol that actually just got standardized last year, which defines how you store data on a WebDAV server that’s useful to calendar clients, and that servers can do something to kind of enforce some consistency and make sure that the data you’re putting up there is actually valid calendar information.

And it just tells the client exactly how to find all the things it needs to find. And then the final bit, which is actually still a standard in progress, is CalDAV schedule, which describes how I can look up when you’re available, how I can invite you to a meeting over this DAV protocol. And that leverages the ITIP bit that I just talked about as well as all of this client server protocol that’s underneath.

So with that, I’m going to turn this over to -- oh, no, I’m going to keep going. One sec. So let’s talk about where we are. So the calendar server is implemented in the Python programming language using a framework called Twisted. Twisted enables asynchronous event handling within a single thread. And so it’s very good for implementing network protocols on top of. And so they already had HTTP support built in there, and then we added WebDAV on top of that, CalDAV access, CalDAV schedule.

We have a fully compliant CalDAV server. We’re actually fairly up at the lead of the pack in terms of implementing all of the different standards, which includes HTTP and SSL and a lot of things that you ought to be familiar with. And as IT folks, it ought to make you a little more comfortable with deploying it because you can use a lot of the information you already have about how to deploy this sort of thing.

It also binds to a directory service. In the Mac OS X Server product, that means open directory. And from Open Directory, we can get information about users, rooms, locations, resources, and groups. So that’s how the server finds out all relevant information about those entities. Multiple directory service implementations are possible within the server, and we ship a couple of example ones within the open source project. So you can, for example, provision all of your users out of an XML file or out of an Apache-style user file. And then the authentication and authorization steps are the same that you’d expect from any HTTP-based service using a basic authorization or digest or Kerberos using SPNEGO.

Again, the calendar server is open source. It’s using the Apache 2.0 software license, which most people find quite amenable. And it’s available on the calendarserver.org website. Over there, you can find not just the source code, but a wiki that has some information about the server and our bug tracking system. So we actually track our bugs out there.

You can file bugs through bugreport.apple.com, which ends up in our radar system, and that works as well. But if you want to directly be able to follow and track what progress we’re doing on your bugs, this is actually a very good resource. And you can, of course, keep commenting on what you’re seeing and whether something worked and that sort of thing.

All of the changes to the source code are sent to an email list, so if you want to review what’s new as it happens, you can do that as well. All of the changes to the bug tracking system are also sent to the developer’s list. We have two mailing lists that are relevant to most people, a developer’s list for those of us that want to actually work on the software itself, and a user’s list for those of us that are just deploying it. And this is in addition to the regular lists.apple.com mailing list which exists in the ADC space.

And of course, new developers are welcome. We’re actually in the process of trying to get the wheels turning on actually giving other people access to edit stuff on our wiki and commit code that aren’t actually Apple employees. So we’re trying to model this very closely to what WebKit’s doing in their space and actually be an open development project, as well as just making the source code available for viewing.

So some of the stuff that’s coming up in, or that has come out in the Leopard updates, we’ve had a large number of bug fixes. Again, it’s a 1.0, so we learn from you guys when you send us feedback what’s working and what isn’t, and a lot of things got fixed in software updates.

We’ve added a couple of small features that were important for some key users, and one of those is private events that lets you actually, within iCal, they actually added support for this as well. There’s a little checkbox that lets you mark an event so that your delegates who have read access or read/write access to your event don’t actually get to see the details of it and can’t modify it.

We’ve done a lot of performance work, in particular in the past few months, but a lot of that has actually already come out in the past couple of software updates. And the iCal Client has done actually a lot of work to help us with scalability. And one example of that is that their polling their polling is actually far more efficient than it was when they first shipped it.

Now coming up in the future, much of this you’ll see in Snow Leopard and there are a couple items in here that are kind of a little more long term. But again, performance improvements are the big thing. We’re doing an awful lot of caching in the server now so that we’re doing less activity that is expensive or takes a long time, such as file system I/O.

We’ve got a shared cache in the system now so that rather than having separate caches for each of the processes because again, the server is single threaded and if we want to leverage all of the CPUs on your computer, we’re gonna launch multiple processes in order to do that. We wanna make sure that there’s only one cache for all of those as opposed to N caches, which uses more memory and makes things slower.

We’ve changed the directory layout a little bit, and that’s actually in a software update that just came out. But we used to just keep everything like there’s a user’s directory and a location’s directory and so on, and everything was split out in there. We actually now sort everything by GUID, and that means that if you delete a user, say WSanchez goes away and a new WSanchez shows up, it doesn’t confuse the two. So this is highly dependent on your directory being correctly administered, but the idea being that it’s a little more secure this way.

A big thing that’s coming up is what we call implicit scheduling. This is being worked out in the CalDAV specs area right now. And the idea here being that right now it is iCal’s job or whatever CalDAV client is talking to us, it is the job of that client to manage all of the scheduling operations on the server. If I want to invite Cyrus to a meeting, I edit my copy of the meeting in my calendar, and in addition, I do a separate request, which is, hey, tell Cyrus that I changed my thinking about when I’m going to have this meeting.

And what we’re finding is that if one of those things happen but not the other, that can turn into a bit of a problem. A lot of the special processing involved with the ITIP part of the specification is fairly complicated and there’s a lot of logic in there. And if it has to get implemented in every single client, then, well, first of all, it makes it harder to write a client. But second of all, it makes it really hard to write a correct client.

And even correct clients are going to have some interoperability issues. So we’re trying to clean that up and make it a server’s responsibility to deal with that workload. And in Snow Leopard, we expect to have that. We are simplifying the open directory schema. We’re finding that it was way too complicated. If you know what a service’s locator is, you probably aren’t too excited about it. So that is going to go away.

If you don’t know what it is, just smile and move along. We’re going to add server-to-client notifications. The buzzword is push notifications, and that’s the functionality you would get from ActiveSync in the Exchange space. But again, we’re doing this with open standards, so ours is going to be based on XMPP and a Pub/Sub mechanism in there, which is going into Mac OS N Server as a notification service, which you may have heard about. And that’s going to enable us to tell the client, hey, now there’s some new data for you, rather than having the client ask us, are we there yet? Are we there yet? Are we there yet?

And reduce the load on the server greatly, and reduce the power efficiency of the device. So if your laptop is polling all the time, that means it’s got to keep the network stack up. It’s also not going to find anything out if the client’s not running. So we want to make sure that the client only asks for information when it’s necessary, and we let it know when it’s necessary.

We’ve got group calendars in Leopard Server, and the calendar UI is actually pretty nice. A lot of people are very happy with it. Of course, everybody now wants that for their user calendars as well, so I can see not only in iCal what my calendar looks like, but I want to see it on the web. And we’re going to add that functionality as well.

We’re doing a lot of refactoring under the covers, which doesn’t necessarily affect you, but if you’re monitoring the open source side, you’ll see all of this in order to accommodate the fact that the address book server which is leveraging a lot of these same technologies, the twisted framework and some of our authentication management, that sort of thing.

And we’re going to share a lot of code with them, so we’re going to try to clean up how we lay out our software just internally. Also, by popular demand, we’re working on email invitations so that if you invite your friend at some completely different company that does not have an account on your calendar server and iCal says, hey, I want to invite them to a meeting, the server will will turn around and assuming you gave us an email address that’s valid, it’ll send an email out to that person with the request. And that should interoperate with a lot more software.

Although I’m going to maintain that the best way to look at your calendars is in fact through CalDav using iCal, but not everybody has that and this is a good solution for that. And then finally we’re working on a server-to-server protocol. This is again being worked on in the standard space, so Cyrus will cover it in some more detail, but that’s going to enable us to have a set of users over here, a set of users over there. They may or may not be in the same organization or company.

They might not be in the same network. But better than the email solution is rather than email the other user, which may go through junk mail filters and end up in a client that’s like, I don’t know what this site calendar thing is, but actually notify the other calendar server off somewhere else and let them know about what’s going on so that they can do all of this within the CalDAV space rather than going out into email. So now I’m going to turn it over to Cyrus to tell you about what’s going on with all of these standards.

Thank you, Wilfredo. I’m Cyrus Deboe. I’m an engineer on the iCal server. Today I wanna talk about, I’m gonna talk about four things. So we’re gonna take a brief look at where we are with the iCalendar standards, talk a little bit about the status of the CalDAV standard and where we’re going with that, some of the future enhancements we’re looking at. I’m gonna give a brief update on the status of the calendaring and scheduling consortium and our involvement in that. And then we’re gonna have a brief demo.

So first of all, iCalendar, RFC 2445. This is really the core specification for calendaring on the internet. It defines the semantics and the syntax of calendar data to be exchanged over the internet. It’s, as Wilfredo showed you before, it’s a fairly simple text-based format right now. It’s a way to describe events, to-dos and task information. You can attach alarms.

You can attach time zone information, which is critical for calendaring. But in addition to that, there are key semantics in calendaring that you need to describe. So, for example, recurring events are a very common thing that you see in an organization. So iCalendar has support for describing recurring events through a set of rules or as individual instances that may occur at various times.

And often with recurring events, you need to be able to override one particular instance and move it to a different time or a different location, for instance. And again, iCalendar has a way to allow you to override instances. So all of that rich expressiveness that you need in calendaring, that’s available through the iCalendar standard in the semantics and syntax that it defines.

In addition to that, we have itip, which is RFC 2446. And this describes how you exchange iCalendar objects in order to accomplish scheduling between different calendar users. So the key thing here is that it, first of all, it defines a special method property that describes the type of message being exchanged between the different parties involved. So typically in a scheduling operation, you have an organizer, the person who’s setting up the meeting, and then you have a set of attendees, and those are the people who are obviously expected to be participating in the meeting.

And the organizer composes the scheduling message, and then they will send that out to each of the attendees. When they do that, they’ll use the request method. When the attendees get the scheduling message, they’re gonna look at that, determine whether they want to participate or not, and then they’re gonna want to send a reply back to the organizer indicating their participation status, accepted or declined, or maybe they’ll say just tentative or maybe.

So, ITIP defines additional semantics on top of iCalendar that describe how all these scheduling operations need to take place. And it’s a fairly complicated protocol that’s involved here because most, particularly when it comes down to recurring events, because in recurring events, when you have lots of exceptions, you can have situations where, for example, one particular attendee is only invited to one instance of a recurring event, or an attendee who’s normally at a full set of instances of a recurring event gets uninvited from one particular one.

So the calendaring system has to be quite clever about ensuring that only those attendees who are attending any particular instance of a recurring event gets that particular data, because you don’t want to expose information about meetings that are going on for which an attendee is not actually explicitly invited. So there’s a fair bit of complicated logic there that needs to be dealt with, and as we’ll see later on, the issues in terms of dealing with that can better be solved by having the server sometimes handle most of this work.

So right now, iCalendar and iTip, they really do, together, they define the standard for calendaring and scheduling on the internet. And both of these are supported by many products. I mean, in the past, iTip scheduling operations have typically been done using what’s called IMIP, which is basically an email profile of iTip. So sending calendaring messages via email, and the expectation is the email client somehow hands this off to the calendar user agent, which then knows how to process this and send back the replies through email.

The specifications themselves are actually more than 10 years old now, but there’s actually work going on in the IETF to revise these, bring them up to date, fix some interoperability issues that have been discovered, clarify parts of the specification that people have had problems with. So right now the iCalendar revision efforts, the actual revision of 2445 is pretty much complete. It’s just going through the final stages of review on that. And the ITIP revision document is also in the IETF last call within the IETF working group. So hopefully we should actually see new versions of those specifications published later this year.

So let’s talk a little bit about CalDAV. So as Wilfredo showed you before, it’s based on WebDAV, based on HTTP. We have a picture of the CalDAV stack, if you like, all of the technologies that go into the CalDAV server. One of the key things that the CalDAV specification itself does is define how the calendar data is managed. So again, we want to make it easy for clients to know where they can find users’ calendar data or key information about users, like what are their calendar user addresses, where are their scheduling inboxes and outboxes for carrying out these scheduling operations.

So there are a lot of WebDAV properties that the CalDAV defines that help clients find this information. And the data structure that we have and use in our server is one way of describing a calendar store. Other types of server, other vendors implement it in slightly different ways.

But again, the key thing is that these WebDAV properties give clients an easy way to find the common core of a user’s calendaring information. In addition, CalDAV also defines how ITIP is used within the calendaring service. So the process of an organizer sending out their scheduling message to the attendees and the process by which the attendees send their replies back to the organizer.

The specification itself creates a set of one or two new methods, HTTP methods. There are a set of reports that are used for querying the server, and it introduces some new access control privileges which are specific to calendaring operations. So. Right now, as Wilfredo said, the specification is split into two parts. The CalDAV access specification defines basically how to read and write calendar events on the server, and that’s already been published as RFC 4791 just over a year ago, I think.

The CalDAV scheduling specification, we’ve been working on that for quite a while, and right now, based on feedback we’ve had, interoperability tests that we’ve been doing, and deployment that we’ve had with the actual CalDAV protocol, we’re looking to make some revisions to that to correct some of the problems that we’ve seen there before we actually go ahead with publication as a standard. So let’s actually look at the specifics of what we want to do in terms of revising the CalDAV scheduling specification. And this is work that’s actually been going on in the calendaring and scheduling consortium that I’ll mention a little bit later.

So right now, the responsibility of scheduling is left to the clients. And we know this is bad because, you know, First of all, it requires that the client application be running in order for updates to get propagated. So if I send an update, if I send an invite to a bunch of attendees and then hop on a plane, and while I’m in the air, those attendees all reply, well, none of them are gonna know about the status of the others until I get off the plane at the other end and fire up my client, and it spots all of these updates and processes them and sends updates back out to the attendees. So this isn’t an ideal situation to be in, particularly if the event is just about to occur. So we need to remove that latency, remove that latency in the update process.

The other thing is that it’s quite possible, there are situations where both the organizer and the attendees view of the same event can get out of sync. So again, the key thing we wanna avoid here is people turning up for an event that was moved to some other time or moved to a different location. So we really need to make sure that the data is consistent for both organizers and all of the attendees.

And as mentioned before, ITIP is a complicated protocol and mistakes are made and it’s hard to get it right all the time. So given that there are usually many more clients talking to a single server, it seems like the best approach might be to move as much of that logic back to the server.

And again, the other thing with ITIP is that sometimes there are several different ways to achieve the same end. And again, if different clients from different vendors do the right thing, but do it in a slightly different way, we can end up with, again, data inconsistencies that though right, are problematic in the long term.

So what we want to do to solve this is, as mentioned before, try to move as much of the scheduling logic to the server. So the server takes care of handling all of the notifications that need to be done in order to keep everyone’s view of a particular event up to date. So when an attendee changes their participation status, right now the client has to do a post operation to the outbox and the message has to get passed back to the organizer, their client has to be on, none of that needs to take place.

As soon as the attendees client writes that event with a new participation status to the server, the server spots that change and automatically does the updates on all of the other attendees copies of the event. And that’s all taken care of by the server. It maintains the data consistency. It helps us eliminate the implementation differences. We know that the server is doing it one way and it’s the way that it’s consistent for itself.

This not only helps the server, but it also helps us in the future if we need to, at a later date, update our calendar store. We’ve got a much, you know, we now know we have a particular consistent state of how scheduling messages have been handled, so it’ll make it easier for us to do upgrades if we need to in the future.

And the other key thing is that the server is always running. So these updates go out immediately. Everyone’s data is kept consistent. We don’t have to wait for the organizer to get off their airplane and turn their device back on again. And again, helps to detect possible inconsistencies.

So, a lot of this work to change the CalDAV scheduling specification, a lot of this work has been done in the calendaring and scheduling consortium. It’s been done by all the vendors of CalDAV server implementations and client implementations. So everyone has agreed that this is the right way forward to fixing the few problems that we know we have.

Some additional work that we’re doing with CalDAV, taking it to the next level. With CalDAV Access and CalDAV Scheduling, we have kind of two pieces of the story that we really want for calendaring as a whole. The third piece that’s missing is this server-to-server protocol. Basically, calendaring between different organizations, between different domains, federated free/busy, cross-domain scheduling. This is really the holy grail of calendaring. This is what everybody wants to see today.

And again, CalConnect is working on putting together the specification to define this new protocol. And over the last year or so, we’ve actually had many of the current CalDAV server implementers start work on developing this protocol. And we’ve had technical demos and technical previews of all of this working together.

And at the last CalConnect back in February, in fact, we had five different server vendors and several different client implementations actually scheduling between different domains and working the way we would like. So work on that is moving ahead and hopefully we’ll actually start to see some specifications coming out later this year on that.

The other thing we’re looking at is a process for simplifying the access control management. One of the things we noticed when we were looking at iCal Server was we needed to support a mechanism for delegation. We wanted to allow one calendar user to be able to manage the calendar of another calendar user at the permission of that other calendar user. Now, WebDAV ACLs provide a way to manage access control on WebDAV resources. One of the problems with WebDAV ACL is that it’s a very generic specification. It provides for a great deal of flexibility and extensibility in the privileges that you can assign on different resources.

And this is a little problem. I mean, this is fine for the server because it can implement most of the logic in a fairly straightforward way. But for clients, it’s a little problematic because there are so many possible choices and combinations of privileges, it’s hard to actually expose that in any sensible way to the user. You don’t really want the user to have like half a dozen checkboxes that they have to figure out which ones to check in order to give a particular set of access privileges to a user.

So what we want to do there is to try and put a layer on top of WebDAV ACL that simplifies that whole process and simplifies the protocol between the client and the server so it’s easier for the client implementers to do. And it makes the process of exposing that information to the end user much easier, too. We can provide a much simpler view for end users.

So with the delegate proxy support that we’ve already introduced in iCal Server back in the initial release of Leopard, we actually have a simplified layer to do that. Now, what we want to do is actually extend that delegate proxy capability, which gives access to your entire account to a particular user. We want to, in addition, support a per-calendar capability. So you can share on a per-calendar basis with other people.

So again, the goal here is to use the similar process that we’ve done with this layer on top of WebDev ACL to support that. But in addition to that, a lot of the other CalConnect members are also interested in supporting this technology with this abstraction layer on top of WebDev ACL. So work again is ongoing using our original specification as a basis. Work is now ongoing to extend this to a more general and useful standard. And hopefully we’ll have a standard for that, specification for that coming out fairly soon as well.

The other thing is, as Wilfredo mentioned, server-to-client notifications. The key impact on CalDAV here is that, there’s gonna be some kind of notification service, whatever that is, but what we really wanna be able to do is expose to clients where to go to use this notification service. So the CalDAV protocol itself has to support some means of discovery of the notification service being used by the server. So we’re going to need some additional properties to help the client get to that information.

And the other piece is private events. In 10.5.3, both iCal Server and iCal Client now support a private events capability. Again, the specification for that, we published on our open source site, so that’s available for anyone to see and review. And again, our goal with that is to take that to CalConnect, gauge the interest of the other members there, and actually most of them are interested in this particular issue too. So hopefully we’ll be able to take that forward as a full standard.

So where are we with adoption of CalDAV itself? I mean, right now there are a fair number of CalDAV servers. One of the key things to note here is that many of these servers are actually proprietary calendar stores that have had CalDAV added as a layer on top of them.

And that was one of the key goals of CalDAV, one of the original design decisions for choosing HTTP and WebDAV, which was we knew that it would be easy for people to put this protocol on top of existing calendar stores. So those people that have a proprietary system now have a way to provide a calendaring solution that can interoperate with other calendar products. And that’s really attractive to many of these other vendors.

On the client side, iCal is leading the way in terms of support for CalDAV access and CalDAV scheduling. There are various web clients that are built into some of the other server products. And obviously the key thing for a lot of our users, I know, is support across platform support. And the Mozilla with their lightning and sunbird projects is kind of leading the way on that. And we’ll be able to have a quick peek at what they’re doing later on when I do the demo.

Next, I want to talk very quickly about CalConnect. CalConnect is the calendaring and scheduling consortium. It’s a venue for both users and vendors to come together to talk about problems in calendaring and to discuss the issues, discuss the problems, try to come up with resolutions to these, and generally further the development of the calendaring standards themselves.

One of the key goals is obviously to improve interoperability of the existing standards. So CalConnect actually has three face-to-face meetings a year, And at each of these meetings, there’s a two or three-day interop event where the vendors of all the products get together and developers bash on each other’s products, find bugs, fix them, and generally progress their products and progress the standards. We find issues with the standards. We find issues with products.

And it’s a great environment for anyone who’s working with either CalDAV or iCalendar to actually come together and talk with other like-minded developers who may know more about calendaring, may know less about it, et cetera. So it’s a good venue. So if any of you are actually developing CalDAV or iCalendar products, I would strongly urge you to take a look at calendaring and scheduling consortium interop events.

One new thing that they’re doing this year is they’re actually holding a mobile interop event in early November in Europe. So again, we’re looking at trying to improve interoperability for calendaring across mobile devices. Again, a key area where calendaring has started to be used a lot more. So that’s a new event that they’re holding, and hopefully there will be further mobile interrupts occurring following that.

Now, CalConnect itself does most of its work through various technical committees that produce recommendations and specifications. CalConnect is not itself a standards body, so typically the specifications we produce ultimately end up being submitted to another standards organization, and the IETF is usually the one that we go to.

So the active areas of CalConnect right now, there’s a technical committee on FreeBusy. This is actually looking at standardizing FreeBusy URL format. So over the last couple of years, there have been a lot of websites, web services that have started up that offer users the ability to publish their FreeBusy information and allows them to share that with other people.

And that’s typically done through a URL that sometimes has various parameters in it in some form with some values that may be encoded dates and times or just an arbitrary token. What we want to do here is to actually standardize the formats of these URLs so that clients and users of this can actually interoperate in a much more straightforward manner.

The real-time technical committee is looking at the server-to-server protocol. The time zone technical committee is trying to define a new standard time zone registry and service. Now, last year in North America, there were a lot of problems with getting calendaring systems updated to the new time zone information for North America. And the problems that arose from that are things that we need to address. So the time zone registry and service is one way to do that. It will define a standard set of timezone names and a service where clients can very easily update their timezone information.

Mobile Technical Committee, in the last year, they’ve developed a very comprehensive test suite and a set of test scenarios for exchanging calendar and information between mobile devices, and that’s actually forming the basis of the interoperability testing they’ll be doing later this year. CalDAV Committee is discussing implicit scheduling and all of the extensions that I had mentioned before.

The XML Committee is working on an XML syntax for iCalendar data. So we’ve had a lot of requests from people who are developing web services that want to embed or use calendar data within their applications. And whilst they could go ahead and try and parse the iCalendar syntax, they’re much more comfortable parsing XML.

So one of the things we’re doing here is developing a one-to-one mapping between iCalendar and XML that you can round trip. You can convert from iCalendar to XML and back again without losing information. and that will provide a way to make web services folks a little bit more happy. So CalcNet itself, there are currently about 42 members, and in the last year since we were here, there are actually been nine new members join. So with that, I’d like to actually go to a demo.

Okay, so what I’d like to demo for you today is iCal on one side and a recent nightly build of Sunbird on the other. So Mozilla, they have two calendaring products, Lightning, which is a plug-in to the Thunderbird email client, and Sunbird, which is a standalone client. It’s, again, a cross-platform client, so it runs on Windows, Linux, and, of course, Mac OS X. So I’m going to run it on Mac OS X, and I’m going to fire it up here. And it’s talking to a server I have on my machine right here.

So up it comes. I’ve got a calendar already configured on my server. And I see a set of events, just as you’d expect. Now I’m gonna switch over to another space here and I’m gonna fire up iCal and it’s also pointing to that same calendar server so it sees exactly the same set of events just as you’d expect. Now we wanna see that, we wanna make sure that these two clients interoperate. So I’m gonna start doing some changes here in iCal. So I’m gonna have a really early lunch and move that to 10:00 a.m.

And I’m gonna go over to, to Sunbird here and refresh its calendar view, and we’re going to see that event move. So just as expected, we’re seeing two clients writing to the server, changes are seen on the other client, it can pull those changes down, and everything looks the same across these two clients, two different clients running on this machine.

So another thing you can do, let’s go back to iCal, we want to see, you know, one of the areas where things have, interoperability has been problematic in the past has been with recurring events. So let’s go ahead and create a recurring event here. So I’m going to create an event. I was going to have some drinks down here. We’re going to make this repeat every day.

Well, okay, not really every day. Let’s just do every day for this week. So five times this week, we’re going to have some drinks in the afternoon. So let’s go back and refresh Sunbird and see what we see. Sure enough, those events pop back up in Sunbird, so it’s again picked up the changes from the server.

Another thing you often do with recurring events is you need an exception. So we’re going to have a big two-hour session drinking tonight. So let’s change that particular instance a little bit longer, and we’re going to go to iCal and refresh that as well. And sure enough, it picks up that change as well. So we have, you know, for the calendar access, reading and writing calendar information, we actually have very good interoperability between these two clients, and we’re certainly very pleased at that.

The next thing is CalDAV scheduling, and this is the piece that the Sunbird folks are still working on. What I can show you right now is their free busy lookup capability. So I actually have… Let me switch here. Okay, so this is a calendar that belongs to Wilfredo.

You see there are, again, some events on Wednesday, and David’s calendar as well I can overlay there. He has a few events on Wednesday as well. So what we’re going to do is pull up iCal’s availability panel, which hopefully most of you are familiar with if you’ve been using iCal with iCal Server.

So iCal went out to the calendar server and requested the free busy information for the three attendees in this meeting, and it got the gray blocks of time of the busy time for these people. The green dashed areas are free time for everyone. So very straightforward process, fairly quick. Let’s go to Sunbird and have a look at how it does this.

So in Sunbird, I can create a new event, and I can go to the invite attendees tab here, and I can just type in a calendar user address. And as soon as I hit tab, Sunbird is going out to the calendar server doing exactly the same free/busy request and pulling down the free/busy information for the users. So if I continue and just add a few more in here.

you can see fairly responsive, free, busy information coming back from the server. It’s basically the same information you see in Sunbird. You also see in iCal. So again, the benefit of having the centralized calendar server, you can see all of the information you need in multiple places. Now, the one thing that, you know, when you click on the list of attendees in Sunbird, you see their statuses in a similar way that you do in iCal, and it has an invite attendees option, but right now, that does nothing.

But the good news is that Calendaring and Scheduling Consortium actually had its roundtable meeting last week, in fact. And at the interop there, the Mozilla folks were hard at work on adding their scheduling support. And we expect to actually see that coming out in a nightly build in the next few weeks or so. So it’s really progressing very well.

And we hope to see some good progress on that in the future. And hopefully I’ll get to a good stable release. And we’ll have another client to do scheduling with. And with that, could we go back to slides, please? OK, so that’s the end of my talk. And now I’d like to bring up David Reed to talk about how we can deploy and scale the iCal server itself.

Hello, everyone. I’m David Reed. I’m an iCal Server Engineer, and I’m going to talk about deploying and scaling the server. First off, the basic architecture of the server. Pretty standard client server model. You have iCal, our preferred client. You have the iCal server. Both of those talk to open directory. And then iCal, of course, has a data store.

These four key components are all designed to be modular. We’ve got some interoperability issues that we’re working on, but it’s all supposed to be modular, which will let us multiply them at will independently of each other. the benefits of modularity, we get some flexibility in our deployment options.

We can have multiple open directory servers, multiple data stores, et cetera. Work can be split up and we can move things around to make data closer to the users and improve availability. Obviously, if you want to scale, you’ve got to have multiple clients. If you don’t have multiple clients, well, then my job is done. We can all go home.

After you get multiple clients, you’re going to want to start thinking about adding multiple directory servers. This is going to -- Reduce latency, improve availability. Open Directory natively supports replication. So you can have a write master and it’ll replicate to a bunch of read slaves. Writes always have to go to a master, but they’re very rare and the iCal and iCal server never write.

You also can do multiple storage pools. This is useful for distributing your data across, say, different NFS servers. which will allow you to use more network throughput, also move your data onto actual physical disks so everything’s not waiting for that one spindle to spin around. This will improve scaling and give you some partial availability. The iCal server does have to write to multiple pools when it’s doing the scheduling operations, of course, and we do write fairly often.

After that, you get to multiple iCal servers. Now, the core iCal server daemon is a single-threaded, asynchronous, event-driven process. All the network I/O is non-blocking, and it runs on one CPU. File I/O does block, but we can mitigate that blocking by having multiple processes. By default, the calendar server will start up one process per logical CPU. So you’ve got four cores, you get four processes. You have eight cores, you get eight processes. We can utilize all those CPUs effectively, and this will help mitigate the File I/O blocking.

The basic deployment of iCal Server is a single XServe. You have the iCal Server, you get some clients coming in, what happens? Well, we start off a master process, which is going to be in charge of managing all these slave daemons, which are actually handling the network connections. You have a shared memory cache for all of these daemons. So responses to coffin operations get pushed in the cache. You also share your storage pool. You have a load balancer in front. And as clients come in, they make requests. They get evenly distributed throughout the slave processes.

This architecture also extends to multiple XSERVs. As you can see, we have a single iCal server. This is a physical machine. And we’ve now moved our shared cache and our data store onto several machines. So we have a machine for the cache and a machine for the NFS server, data store, et cetera. And suddenly we can plug in multiple XSERVs.

put a soft load balancer in front of it, and we can again distribute connections evenly throughout all the systems. Little bit later, you’re gonna wanna reduce latency by switching out a hardware load balancer. And after that, you might even consider getting a hardware load balancer with an SSL accelerator so you can do all the crypto stuff in something that’s, well, good at that.

And then multiple storage pools do even more distribution of load across more pieces of hardware. You have a lot of options for storage. Obviously, single XSERV case, direct attached storage. You have local disk. It’s fast, but there’s not a lot of reliability. So you move to a local disk with RAID, two drives mirrored. After that, you can improve your throughput, get faster access to your data by switching to a more expensive Fibre Channel solution with a promise raid box with a bunch of drives in it and use Fibre Channel to get really fast access to your data.

You also have the option of -- when you want to go to multiple XSERVs, you have to go to a storage area network. Here you’re trading off some performance, but you’re getting a lot of reliability. You can distribute your data across multiple physical machines. One goes down, you’ll still have access to some of your data for a lot of your users. As Wilfredo said, the new onDisk data format makes it easy to partition this data using the GUIDs across multiple physical drives or mount points.

So we’ve done a lot of work since Leopard Ships to improve server performance, a lot more efficient use of databases, put indexes in some places we needed indexes. We also reduced the amount of work that the server’s doing. Sometimes it was doing too much work. We’ve gotten rid of that. And we have lots and lots of caching.

caching improvements. Okay. So we decided to use memcached for the caching. I’m sure a lot of you have used memcached, deployed it, comfortable with it, you know. It was written by Denga Interactive to scale their live journal service, and now it’s used all over the place. Our goals with the caching were to optimize some of iCal’s most frequent operations, reduce lot contention on certain shared resources such as various databases, overall reduced file system I/O, hit the disk a lot less, which all gets us higher throughput, better memory consumption, and better CPU usage.

So how did we use memcache to the best? First off, all the processes, whether they’re four processes on a single machine or four processes across multiple machines, all use the same cache. This makes the implementation a lot simpler with cache and validation and such. So the things we cache are, When iCal pulls the server, since we don’t have push notifications yet, every time iCal pulls, they’re getting a static response in memory unless the data somewhere has actually changed. We also put the delegates database, which is used during a lot of ACL operations, into memcached, so all the processes share it and they have really fast non-blocking access to it.

We also plan to cache all the open directory data that the iCal server needs among the entire cluster, so you’re hitting OD a lot less, which will just remove a potential bottleneck and point of failure. We also use it to share authentication information across all the processes, so if you suddenly get rerouted to a different physical machine, your credentials are still valid and you don’t have to generate unnecessary traffic logging in again.

We also put a lot of things, a lot of locks for making write operations atomic into the Memcached database. Again, shared resource, very fast, non-blocking network access to it. The best part about Mooncache D is that it’s a proven technology. As I said, lots of people use it all the time, all over the place.

The iCal client team has also done a really great job improving performance. They’re making a lot fewer requests when they pull the server. And when doing things like delegate-related operations, they don’t do any unnecessary -- they don’t do any unnecessary expensive work. They also started -- they’re also going to be supporting persistent connections, which will reduce TCP overhead and -- help with connection to the network. And now I’d like to hand it back over to Wilfredo Sanchez to wrap up.

OK, so basically we’re going to scoot on into Q&A shortly. But in summary-- iCal Server came along with Leopard Server. We’ve had some pretty good reception from our users so far. We’ve had some issues, and we’ve tried to address as many of them as we can. And we think that in Snow Leopard, you’re going to see a pretty nice polished product. A lot of cool features are going to show up in Leopard Server.

Again, the push notifications are a big item, and the email messaging is a big item. We’ve done lots of performance work. We’re still working on driving the standards. A lot of this work with the implicit scheduling and so on, it’s all kind of bleeding edge. We shipped the thing that implements CalDAV scheduling before the CalDAV scheduling specification was finalized.

So obviously we’re trying to stay within that space as well as we can so that we remain in the standard space. And again, a lot of options exist for how you deploy it. As you can see, a lot of the bits are in fact modular and you can put things in different places.

How that fits in your particular environment is going to be very particular to how you deploy other things. So that’s what the Q&A is gonna be all about, I’m sure. But for more information, talk to Matt Drance. He can help you out, get a lot more information about all of our sharing services.

We’ve got a lot of documentation, sample code on the ADC site. And then again, remember the open source website is actually a really good resource because that’s where we have everything that we’re working on. So we’re trying to be very open with our process. And if you want to get involved, that’s a very good way to do it.

You probably want to check out the performance session that’s coming up on Friday. And that’ll help you with tuning a lot of specific things about your server environment. And that’ll help with your deployment. Also, if you want to corner us for one-on-one questions or smaller group questions, we’re going to be in the Collaboration Technologies Lab downstairs at 3:30. That’s actually pretty soon. We’re also going to show up at the iCal and CalDAV Interoperability Lab.

That was meant mostly for the client team, but since our lab is right after this and you might have some conflicts, we’re going to give you an opportunity to talk to us there as well. And again, the Leopard server performance team, they’re all going to be at a lab on Friday at 3:30. And they can help you out with tuning your server, getting the best I/O out of it, and that sort of thing.