and Chris Apple. We're going to be talking today about data synchronization. Hello. Oops, Okay, this is the second talk on Data Synchronization. The first talk covered the architecture, kind of the core concepts and the API. Talked a lot about basically what Sync Services is, all the different parts and what it means. This talk's a bit more of a how-to.

How do you take an application and get it to synchronize data? Now, there's going to be a little bit of overlap from the first talk, so I don't know if anybody was at the previous talk. If you were, feel free to doze off a little bit as I go over things. But I do want to make sure that we frame all the context and all the concepts so that it all makes sense.

So let's talk about Sync Services, what it is. Initially, we had three applications that you could synchronize, or three data types: contacts, calendars, and bookmarks. You could also synchronize these to devices, phones, iPod, and you could synchronize them up to .Mac. Now what we've added is the ability for you to synchronize your own custom applications.

So you can take your app and you can synchronize your data. You can either synchronize with the existing types of data we already provide, so you can kind of join in on the party, or you can sync your own data and you can get it up to .Mac and across to other machines.

So what are we going to show you today? We're going to go over a review of the syncing concepts. This will be a bit of a refresher if you saw the previous talk or have read about some of this before. But I want to make sure that I cover the basic concepts so you know basically what I'm talking about.

I'm going to talk about what you need to do in an application. I want to make sure I cover the what. What is it that you're doing? We'll give you a demo that will highlight and illustrate what we just told you. So first I'll tell you what to do, then we'll show you what that looks like, and then what I'm going to do is cover how to do it. We'll show some small code snippets and we'll talk about exactly how you do all these things.

When you walk away from here, this is what I want you to take with you. I want to make sure that I'm covering enough of the overall architecture that you understand what Sync Services is, and that you understand what it means for your application to be synchronizing. That consists of three main things: how to work with schemas, if I can get that to highlight. Data schemas are used to identify the different types of data that you'll be synchronizing. So we're going to cover how you can make your own schema and how to use schemas that are already in the system.

How do you manage a sync session? This is the core of the talk. What do you do when you're actually synchronizing your data? We'll go into that in pretty good detail. We're going to talk a lot today, tonight, seems dark in here. We're going to talk a lot about how you trickle sync.

Trickle syncing is a way to basically push small bits of data very frequently so that from the user's perspective, their data is just constantly trickling out. When you make changes in your application, you want to trickle the changes out as soon as you can. And when changes are made in other applications that you're syncing with, you want to be able to pull those changes in transparently to the user.

I'm going to cover a little bit of terminology. Clearly, we need to define our terms, so we'll introduce some of that as we go through the talk. And I'm also going to talk a bit about best practices. By that I mean things that you can do in an application in order to provide the user with the best experience. Things that you can do so that syncing goes smoothly, fairly transparently, and so that when problems occur, the user has choices on what they can do.

Each user on each machine will have a truth database. That means if you, as a specific Mac OS user, have two different machines that you're syncing, you'll have a truth database on each machine. And if you have more than one user on a machine, each one of those users has their own truth database. And the truth is an aggregate. It's an aggregate of all the data that's synced by every client.

So although your client might be syncing some data, you might not sync every field or attribute that's in a certain set of record types. You might sync contacts, but you only sync a few of the fields. There might be a richer client that syncs many more fields. So the aggregate is the combination of everything, even if your client isn't synchronizing it.

We have three main sync modes: Fast Sync, Slow Sync, and Refresh. Fast Sync is the most desirable. That's done when what you want to push is just the changes from the last sync. If you're able to keep track of deltas between sync operations, then when you synchronize, you can just push up changes. Added this record, deleted this record, made a mod to this record. So you push up little changes, and then you pull down from the engine only what's changed there.

If you don't keep track of the deltas, or if it's the very first time you're syncing, you'll need to do a slow sync. When you do a slow sync, you push up all of your records. What the engine does is it looks at all the records you're pushing up and compares them to the last set of records that your client had pushed. If you don't push a record up that you had previously sent, that's treated as a delete. It also compares all the fields so it can tell what records have changed.

This is clearly a fairly expensive operation, especially if you have thousands of records. So it's always more desirable to do a fast sync when you can. But there are a few cases where you'll have to do a slow sync, and I'll talk about that in a little more detail as we go through the slides.

Finally, you can do a refresh sync. This is typically done when you've lost all the data on your machine or you've lost all semblance of state regarding your sync sessions. If this happens, you can tell the sync server, "Forget anything you ever knew about me. I'm just going to refresh. We're going to start over as if we've never synced before." Typically, you'll be doing that to pull everything down from the engine so you can reset your state.

But there are cases where you might have added a few records and you want to also push those up to the engine but still do a refresh. For instance, you may have lost a battery or power in a phone and lost all the data. But you're traveling and you've added a couple new contacts, so you've just got two contacts on this phone.

You tell the engine to do a refresh sync. You just push up your two contacts. They're added and all of the contacts that had initially been in the sync engine are pushed back down to your phone. So with these different modes of syncing, we give you the performance when you need it, we also provide security that you'll always be able to recover all your data and that you'll always be able to sync every record. Most important thing when you're syncing is correctness.

Second most important is speed. And most users are going to think the most important thing is speed, especially when things are slow and take a long time. But trust me, the minute you lose some of their data, they're going to realize what was most important, and that's being correct.

Oh, I'm sorry. I was supposed to show you that before. I'll be doing that throughout the talk, so get used to it. There's one other mode of syncing, which we call Trickle Sync. And it's not really a mode in the same sense as the other three. It's more like a way of life. For your application, if you Trickle Sync, you routinely Fast Sync. And there's a number of different places, opportune moments when your application can do a Fast Sync. You try to do it in the background, make it transparent to the user.

And one important point, an application probably shouldn't ever synchronize unless a user's told it to. So when your application starts for the first time and the user starts to use it, you should provide them some configuration so that they can say, yes, I want to sync my data. Now, at that point when they do, they've given you carte blanche to sync it whenever you want.

So you should go off and sync small changes frequently. We talked in the previous presentation about how the more frequently you can sync, the less data you have to sync each time. So it's more like a continuous flow that doesn't put much load on the system. It doesn't seem to take a long time for the user, and it keeps things synchronized more expediently.

If you make changes in your app and push them out right away, they're available to other applications and available to get pushed up to .Mac right then and there. Similarly, when changes are made in other applications, you'll be notified and you want to pull them in as soon as you can so you always have the most up-to-date representation for the user.

So let's talk about what you need to do in an application to sync it. What are the five most important things, the five main things? I've grouped what I think are the core fundamental pieces that you're going to have to put into an application to sync. The first one is setting up a data schema.

A data schema represents the type of records that you'll be syncing in a canonical format the engine can understand. This is important for a few reasons. Most applications have their own way of representing data. You could use objects all the way down. You might archive them or store them in a graph. You might just have a simple text file, comma separated values. You could be using a database. And all of these things are going to be different for your application from other applications. But the engine needs to know one specific way of representing this data.

So you'll define in a data schema the set of entity types that you're going to be synchronizing. An entity is kind of like class for objects and instances of classes. An entity is the type of object and it will have a set of attributes. So when you sync contacts, you'll define a contact entity, a phone number entity, and you'll define all of this in a schema. And you'll define a schema that can be used by the engine.

Another important point about this, when you sync, you're not just syncing your application to the sync engine. You're syncing with every other client that synchronizes. Those clients don't want to know how you represent your data and you don't want to have to know how all these other clients represent their data.

You don't want to have to know how data is represented on a phone. You don't want to know how it's represented on a server. And you don't need to. You just need to know one way to represent it and that's the way that's defined in the schema. Amen.

Second thing you have to do is configure your application. You could have the greatest sync client in the world. If you don't configure it, nobody's going to know about it. It's never going to run. Configuration is actually fairly simple. It's a combination of a static file that defines some properties and a very small bit of API that you use. By configuring your client, you're registering it with the engine, making sure that the data schemas that you want to use are registered, and specifying any alerts that you might want to get when other clients synchronize. It's pretty straightforward. The fun stuff is the actual syncing control flow.

We have a syncing state machine API that you use. We have a fairly object-oriented API in general, but the actual act of data synchronization is a state machine. It's a procedural API with methods you invoke on a sync session object. We went with a state machine with a procedural API for flexibility. By providing an API like this, there's a lot of different junctures and points where you can opt in and out of the sync, and you have a lot of flexibility on how you want to do the sync.

If you get involved in a sync operation that starts to take too long because other clients are also syncing and they're taking a long time, you can opt out of the sync. You can cancel a sync if something goes wrong at any time. You can complete it in the middle and take up where you left off on the next sync. You have to do the sync steps in the order we're showing you here, but other than that, you can leave after you push your changes. You can stop syncing before you pull anything down, or you can go all the way through. It's up to your application. application.

I talked before about having a data schema to represent your data in a canonical format. That's kind of the class. This is the object. When you take any of your data and want to transform it over to the Sync Engine, you'll be creating essentially instances, which we call records, of the schema types that you've defined.

When you are pushing records up, you take the records or the objects in your application, you transform them into records based on the data schema, and these are essentially NSDictionary's. You push these dictionaries into the Engine, and when you get changes back, you'll be taking dictionaries or sets of field changes that are represented in something we call an iSync change, and you'll be transforming that back into your own record types for your app.

Finally, for fast syncing, you need to do some state management. If you want to fast sync, you have to keep track of the differences between the first sync or the previous sync you did and the current state of your data. This is for both adds and deletes, modifies being somewhat of a special case of add.

If you've modified a record or added a record, you can put it on a list. When you delete a record, you can put the ID in a list. When you go to sync, you just consult these lists, and that's all you'll have to hand to the Engine. I added this, I modified this, I deleted this. If you don't do that, you'll have to give the Engine every record. So there's a few different ways that you can maintain a state.

You could keep time stamps on all your records. You could keep a dirty bit on the records. You could have a list of what records you've changed. What you do is going to be something that you'll choose that's most appropriate for your application's data model. I find typically keeping a list of what's changed is very straightforward. You'll also want to save that list in a file if you quit out of your application without syncing. That way, when you restore your application the next time, you'll still remember the delta. so you'll know what you can sync.

There's a few other things that you can do too when you're syncing. You're an application. Whenever an application does anything, you shouldn't just go off and do something without the user knowing what's going on. So you need to provide some sort of feedback. Most sync operations will hopefully be very brief. Some of them might be so fast that the user couldn't even detect them. But others may take a few seconds or longer. So if you're engaged in any kind of sync operation, make sure that you put some animation up so the user knows what's going on.

You can use a spinning bar animation. You could put up a status pane. You could just put some status text somewhere so the user can look and see what's happening. Users tend to get very annoyed if an app becomes unresponsive. If you're syncing in the main loop of your application, the user might be clicking in a text field trying to do something.

Even if it's just a second or two seconds, it'll get kind of confounding to someone that's trying to use your application if it doesn't respond. But if they see something spinning, that's kind of a cue. Okay, something's going on here. And they'll get used to seeing that, and it'll be less troublesome for them. I mentioned before about trickle syncing. Sync often and then try to sync fast. If you can do that in an application, it's gonna be a much smoother experience for the user.

So I just told you what you have to do. But there's a lot of things you don't have to do. You're probably thinking, "Boy, I have to do all these things. What did you guys do?" Well, we actually did quite a lot. And I think we did a lot of the heavy lifting so that we're going to make it fairly straightforward for an application to synchronize. And these things that you don't have to do are fairly complex. First one is conflict management.

You don't have to worry about conflicts between different sources. In fact, you can treat syncing almost as if you're in isolation. Your application just syncs into the engine. If there's conflicts with records for other clients, we'll take care of that. We'll notice the conflicts, we'll keep track of them, we'll present UI to the user to resolve them, and then we'll handle merging in the conflicts correctly.

Also, you don't need to present what will change to a user. If you're going to make a hundred changes, if you're going to remove a thousand records, we'll pop up something we call an airbag. It's an opportunity for the user to opt out of that sync. We'll tell them, "These changes are about to be made by this sync client." So you won't have to worry about doing anything to notify the user about changes that you're making due to a sync. You don't have to detect duplicate records.

Sometimes when you sync two sources for the first time, in a very common case, is a phone, an address book, or a phone with calendar data, and a calendar, you'll have the same record on both devices, or on both clients. We'll detect if those are duplicates, and we'll handle merging them together to present them as one unified record. So you won't end up duplicating every contact you have just because you synchronized a phone that happened to have all the same contacts on it already.

You don't really have to pay attention to other clients or worry about .Mac. We've got a decoupled architecture regarding all the clients. Your client just syncs to the engine. You worry about that. If you get that right, we'll fan all of your changes out to the other clients. We'll get the changes from them into you. We'll take care of everything.

You don't have to do anything special to sync to .Mac. .Mac will automatically be able to sync your data types up and down to other machines as long as you've defined a schema for them. Okay, now I'm going to bring Nancy Craighill up to do a demo. She's going to illustrate many of the things I just talked about.

You wired up? The clicker, please. : Sorry. I like it. Then we'll go to the slides first. So we want you to feel confident when you leave the session that you too can write syncable applications. So we did select a little more sophisticated example, not a trivial one, so you can get the most out of this session. And also when you're watching the demo and the rest of Gordie's slides, I think it's time to begin to think about how you might modify your existing apps to sync and how you might create a new application that's syncable.

Okay, so what you're going to learn from the demo and the rest of the talk really is how to sync your custom objects. You can sync, like Toby and Gordie said, you can sync all the contacts and calendars and bookmarks, but we think it's a lot more exciting if you create your own object models and you sync those objects. You're going to learn how to sync relationships in your object models.

and you're going to learn how to sync your application simultaneously. If you combine that with syncing your applications often, then you'll learn how to trickle sync. And the good news is that these demos that you're seeing today are available now on your Tiger Seed DVD, so you can go to Developer, Example, Sync Services, and if you're gung-ho, you can open up your Xcode project now and you can follow along because Gordie's going to actually show a lot of the details later that relates to the schema files and the client description, et cetera.

So because it's a sophisticated example, I'm just going to take a moment to just tell you what the architecture is and the object-oriented model behind these apps. So of course you have the Sync Engine and the Truth database at the center. We have one app we call Events. It's just going to import iCal files and create custom event objects. The second application is Media Assets and it's just going to go to any old iPhoto library year folder and parse it. And each of these applications has their own local database store.

So what does the object model look like? There's an event object. It corresponds to a wedding or a birthday party and a media object, and it corresponds to a photograph that was taken at an event. So naturally there's a 2-1 relationship from media to event and a 2-many relationship from event to media. So this is a special relationship because if you set the 2-1 relationship from media to event, you would expect that media object to be added as one of the destination objects of the 2-many from that event to the media. So we call this an inverse relationship.

And the good news is that Sync Services is supporting inverse relationships in the Sync Engine and will maintain the integrity of inverse relationships even if you don't. Okay, and then this is an example of an event. We went to Mendocino to the beach, and here's the photographs. In retrospect, this slide was probably not needed, but I just wanted to show off my photography. So can we go to demo one? All right. So this is the events application. It's a simple master detail interface. I've already loaded the iCal file, by the way.

For those of you in the back row, I'm just going to zoom in. Can you see that? Event has just the title attribute, start date, and end date. So I'm going to point out a couple of other things. Down here, you see the record ID. and below that is the client ID. So each application has its own client ID. Okay, another area I want you to look at.

Over here. Now, when you do your applications, you're not going to have a big, old, ugly Sync button here and a trickle checkbox. But we added that there because we're going to first show this demo slowly, step by step, to show you the process of what's happening between the two apps. And then we'll speed it up later by turning trickle syncing on.

And we also implemented a calendar view, because you typically want to view your events on a calendar. So at this point, the events application has the local event objects. I'm going to push the sync button. There's going to be a progress indicator that runs here, and it's going to push the local event objects out to the truth database. And that's the end of the demo. No. So this must be a tiger bug because I can't hide it. So let me hide it like that. Let's bring up the Media Assets app.

Okay, here's the Media Assets app. Same thing, it's a master detail interface. I'll import some and I have an iPhoto library of 2004 photos. Down here you see, I don't know if it helps if I zoom in, but basically there's a title is one of the attributes, the date of the media object.

The image is just a URL and it's being shown here below. Now there's an event pull down menu. You can't tell, but I'm actually pushing the mouse on there and nothing is appearing because this application doesn't know anything about event objects right now. Again, it has a record ID under here for each record and has a client ID.

Okay, so if I now push the import button, it's gonna push the media objects to the truth database and it's gonna pull out the event objects. I do that all the time, sorry. There we go. So as the events just got pulled over and populated into the menu.

So I happen to know that's Chinatown and this picture was taken at Chinatown, but I'm not going to make you sit there while I set all of these relationships. So what we did was we created a Smart Events button, and it's just going to run down these media objects and assign them to the most logical event matching the dates up.

So if you push that button, now all my media objects have events. So again, just to review, we've just created two one relationships between the media and the events object and too many relationships from all the event objects to the media. Again, it's just local and I haven't pushed it yet.

So if I push Sync, it's going to push it out. And then we'll bring up the events application and the calendar view. And when I push sync here on the Events app, it's going to pull both the media objects and the too one and the too many relationships. And hopefully we'll see them on the calendar there.

So let's turn on trickle syncing on the Events app. And then we go back to Media. We'll import some more photos. February, March. OK, they're down at the bottom. Put Smart Events, create some more relationships. Let's move this up to February. OK, now when I push the Sync button, the Events app is set to trickle sync. So it's going to get an alert that Media Assets is syncing, and it will begin syncing simultaneously. So you have to look quick. There'll be a progress indicator over here. There'll be another progress indicator over here. And there's the photos.

Okay, now we'll go back to events. This is a multi-day event, Tahoe skiing trip. Let's say we want to change the name of that, but I want to show this off, so I'm going to find Tahoe skiing over here too. And let's change that to winter. Now when I hit the tab, it's going to modify the local objects. It's set to sync about every five seconds. So there'll be a moment delay. It will sync. When it updates the local changes, it will update the Tahoe skiing down here. And then when it pushes the changes out, it'll update media assets if I turn trickle syncing on.

All right, so I haven't hit the tab button yet. Now I'm going to hit the tab. There it There it goes. All right. Good work. Slow. Okay. So let's have a little more fun. This is a multi-day event, so let's move some of the pictures. Let's say that this picture was actually taken on the 16th.

And it should appear over here on the date of the 16th. Hit Tab. You're waiting for the sync. There it goes. Now it moves. For those of you who missed it, we'll do it one more time. We'll move this photo to the 18th. Hit Tab. It should sync up here.

OK, you're all probably wondering a little bit how it's implemented, so I'm just going to cover that briefly, especially if you're looking at the code. So it does use a model view controller paradigm, and the models are the syncable objects in the design.

There we go. There we go. OK. And we use Cocoa Bindings, of course, to update all of the changes that are done locally in the app. But in addition, when you're pulling all of the changes and applying them to the local objects, that's how the displays are being updated.

We also used Key Value Observing, which is the underpinnings to Cocoa Bindings, and we used that to record all of the local changes. As Toby and Gordie were saying, that's one of your jobs, to record all the changes you make locally for pushing later. And we also found Transformers, that is the NS Transformer class, useful for converting your models to records before you push, and then when you pull the changes in from the Sync Engine, you need to apply them to your models, and sometimes when you get additions, you need to create models, so we used Transformers there. We also used Transformers for resolving the relationships.

So Gordie's going to show that in more detail, but the relationships that come from the Sync Engine are not what you expect. You have to convert that to actual references to your objects. And I think that's it. We'll go back to Gordie for more details on how to.

Let's talk a little bit about how we did many of the things you saw in the demo. Nancy actually wrote the demo, so she gets all the credit, but if I get it wrong, you've got to forgive me. But I do promise to do better with the clicker now.

Okay, I talked about the five main things you need to do to sync. Let's just recap quickly. You'll set up a data schema. You'll have configuration for your application. Then we'll have the main sync loop. Before, we mentioned that's the meat of syncing. For any vegetarians out there, that's the tofu of syncing. You'll have data transformation, and Nancy just touched on that a little bit. And then, of course, keeping track of your data so that you can fast sync. So let's go over that now.

Let's look at a schema. So what goes in a schema? Essentially, you're defining entities. So in the example we just showed you, we had two entity types. MediaAssets object, which had a picture, a title, and a date associated with it. And we also had an event object, which had a title and a date. And we mapped the media objects to the event objects.

We have attributes, such as the title and the date. We also have the relationships, such as the relationship from the media asset to an event and the relationship back from the event to the media asset. Now, I'm being a little bit redundant, but I want to make sure that we didn't skip over anything here. So this is kind of a pictorial representation. I'm just going to run through it from the top just to tell you all the different parts of a schema. You start off with a data class.

A data class is actually somewhat of an informal construct. It's used to present what you're syncing to the user. So if your application has a number of different entities that you want to sync, you can sync them to the data class. You can group them together in one data class. An example of that is contacts and calendars. Rather than specifying every entity type to the user, providing them with way too much information, you can sort of summarize it by naming it in a data class.

As I mentioned before, you're syncing entities. So a data class consists of a number of entities. And then deconstructing further, we can see attributes. These are primitive types that you use, basically what you would put in an NSDictionary. They represent the different attributes of each entity. In a contact, you would have name, first name, last name, and clearly we just saw an example in what we showed you. You also have the relationships. And one other thing that we didn't mention before, identity properties. The first time you synchronize a new object or a new record from one source, what we'll do in the engine is compare it to all the records we have from existing sources.

If we see that it's the same record, we won't duplicate it. That way, and I mentioned this before, when you sync your phone for the first time with address book, you're not going to duplicate every entry that you've entered dutifully into both. The way we do that is by having schema specify the identity properties. These can be attributes or relationships. You might want to scope the identity of something through a relationship.

For instance, for a phone number, you might scope its identity through the enclosing contact by specifying the relationship from that phone number back to a contact, as well as the type and the value, which would both be attributes. So you tell us dynamically, this isn't something that you're stuck with, it's not a static description, but it's something that you put in your schema that we can use that can be different for each data class. For each entity type, you tell us what the identity of an object is, how to notice that, and we'll take care of mapping duplicates.

So let's look a little bit more at a Sync schema. It's a plist, straight up, it's very straightforward. You'll have a name for your schema. That way, any introspection tools, any UI can be used to look at it. We'll be able to determine the exact name of the schema. Also, the engine has to be able to identify schemas uniquely. You don't want two schemas with the same name, so we recommend that you use a DNS style name. Here we have com.apple.sync_examples as our name.

You have a set of data classes. Usually you'll just have one data class. For instance, with the contacts schema, we just have the contacts data class. But depending on the complexity of your application and the choices you make as to how you want to organize your schema, you could put more than one data class in one schema. It's up to you. You have a list of entities and that's really the main thing that you're going to be putting inside of a data schema.

So let's look at an entity. Each entity also has a name. It's DNS qualified as well, so that it doesn't conflict with other entities. Entity names are in a global name space. They're not just mapped within the schema that they exist inside of. The entities are treated as global. That way you can refer to an entity in another schema if you want to extend something or if you wanted to refer to a data class in another schema that you're adding an entity type to. So you have to make sure that you use a unique name.

You specify the data class that your entity's in, and you give it a display name. The display name would be used, again, by any user interface, something so that the user doesn't get stuck looking at really long, weird, disambiguated names. In this case, media makes a lot more sense to the user. Then you have your attributes, relationships, and identity properties, and let's look at those.

Attributes are very simple. I've just included two here. I put ellipses at the bottom because this isn't everything. It's kind of hard to fit things and I hope you can see this. Actually, I've noticed in some of the presentations it's hard from the back to be able to see when we put code up or any kind of text like this.

But in this case, I'm specifying two of the attributes, the date and the title. This is for an event object. You specify the name and the type. The name is the field that you're going to use in a record dictionary that represents one of these entity types. And then the type is just simply what it is.

Here's a list of the attribute types that you can use. I just put it here quickly for completeness. Standard stuff that you can put into a property list. Also, you can use an array or a dictionary as a primitive type, but you need to be careful. We're doing field level differencing. If you have a record and it's got five different fields, we'll difference those fields independently. So if one record from one source changed field one, another record from another source changed field two, that's not a conflict.

We'll merge it together. But if one of your fields is an array or a dictionary, that entire collection is going to be considered the atomic unit for that field. So if you make one small change in that, in one source, and another change in another, that's going to cause a conflict.

There are some cases, though, where it's very convenient to be able to use a collection, but wherever you can, it's best to split up your attributes into separate, or use separate attributes. for each one of your semantic fields. There's a few additional types. Calendar date, just because it's so useful.

You can't put calendar dates into property lists, but you can put them inside of a record. We have NSData in case you want to take something like an image or something that's your own object type that's not represented by one of these. You can just sort of stuff it into an NSData.

You can also specify an enumeration of strings. This is useful to have a bounded set of strings, and the engine will actually do some consistency checks for you. So if you want to have weekdays, Monday, Tuesday, Wednesday, and so on, you can specify those, rather than just saying "string" and then possibly mistyping something. You can also have a URL, which is very useful to reference things elsewhere.

So let's look a little bit at a relationship. Relationship starts off with a name just like an attribute does. It has a display name. Now I didn't show this for the attribute because it wouldn't fit on the slide, but both attributes and relationships have a display name. That way as tools are developed that can do introspection to these things, you can display something a little bit more meaningful than the normal name that you'll pick. Notice that the names of the attributes and relationships don't need to be DNS qualified. Their scoping is local to the entity that they reside in.

For a relationship, you specify whether it's one to one or one to many. We have a one to one relationship in our example from a media asset object back to an event. Each media object corresponds to one event. However, the events can have many objects. They can have many media objects. So in one direction we're specifying a one to one relationship. In the other, it's a one to many. We're showing the media right here. This is the relationship. I probably should have mentioned this. This is the relationship in a media record back to an event.

You specify the target type. This is the fully qualified target type of event. So here we're just simply specifying we've got a one to one relationship to an event. I did a lot of talking. That was actually something that's fairly simple. You can also specify an inverse relationship.

These are very useful when you want the engine to do some consistency checking for you. If you've set up a media asset to put a link to a specific event, you can specify the inverse relationship. You can also specify the inverse relationship. These are very useful when you want the engine to do some consistency checking for you. If you want a media asset to point back to an event, you want that event to contain that media asset.

Similarly, if you move a media asset's relationship from one event to the other, you want to make sure that it's unwired from that first event and wired into the second one. You can specify an inverse relationship. Now, this is a little tricky to look at outside of context. If you look at the examples that we've provided and you look at the entire schema, you'll be able to see how this is wired up a little more clearly.

If I could say that clearly, there's a metaphor in there somewhere. We have the entity name for the inverse relationship and the name of the relationship that's back. So we're saying a media object has a relationship to an event, and then their inverse relationship is from the event's media relationship field.

This is just how you specify identity properties. It's very simple. It's just a list of attributes and relationships that are being used for the identity for that record. In this case, we're using the date and the title of an event to identify it uniquely. Okay, so let's talk about what you're syncing. I just described the classes. Now let's talk about the instances of those or the records.

When you're syncing an object, you have to push up two things: a record dictionary and a unique identifier for it. Now, the identifier must be unique across all of the entity types that you're synchronizing. So if you have contacts and you have phone numbers, you can't use the same identifier for a contact that you use for a phone number just because they're different entity types. You always have to make sure that all of your identifiers are completely unique. However, you don't have to worry about other clients. Your client has its own namespace for all of its identifiers.

You need to put the entity name in a record. That's essentially like the is a pointer back to a class inside of an object. By specifying that, the engine now knows what kind of record it's dealing with. If you didn't put that in the record, we'd look at this NSDictionary, it'd be filled with all kinds of great fields, but we wouldn't know what it was. So you always have to make sure that you put the entity name in.

Now, everything that goes in that record is just a set of key value properties. For an attribute, it's one of the types I showed you before. So it's just a set straight up dictionary. For a relationship, if it's a one-to-one relationship, you'll have an array with one element in it. If it's a one-to-many relationship, you'll have an array with zero or more elements.

The reason that a one-to-one relationship still uses an array is for consistency. So you don't have to have code that's doing is kind of all over the place to see if this is an array or just a singleton object. Relationships are specified by use. using the unique record identifier of the target.

Now one thing about record identifiers. Often when you're using a relational database, you can construct a unique identifier with some combination of your primary key and your record type in the database. So you might want to use what you have for a primary key as part of the record identifier. If you do that, you may not want to put those fields into the dictionary for the record or into your schema because it's redundant. You'll be using them for the record identifier. There's no reason for you to also put them inside of the record itself.

So let's look at what an application sees. A user will look at an application and they'll be presented with some kind of visual representation of your objects. In the application, you've got your own objects internally. Like I said before, these could be structs, they could be objects, they could be constructed out of strings, it could be whatever you want.

When you're syncing, you need to transform those into records. So these are probably a little hard to see from the back, but these are just straight up NSDictionary's and that's what you're going to be syncing back and forth to the engine. So let's look at one in more detail.

This is a media record. In white I have the actual name of the entity. And then I'm just highlighting the relationship to separate it from the attributes. Very straightforward. It's just a dictionary. We've got an array for the event, which is a relationship back to the enclosing event.

Hello. OK. This is what a event looks like. And the only difference here is that it has a list of media objects, since it's a one to many. Otherwise, very similar. So we have a very regular way of specifying all the records when you're syncing. You don't have to worry about pushing up different objects in different ways. Everything eventually grounds down to just being a dictionary.

So now let's talk about configuration. We know how to describe the schema for the data in our client. Now what we're going to do is we're going to set up a client description property list. This is also a plist file. It statically describes the characteristics of your client.

So what does it have? We've got a list of the entities and the properties in those entities. Now you might have a data schema that you're sharing with other applications and there could be a whole slew of entities in there and a lot of properties because you're trying to cover all the bases.

This particular sync client that you're writing may not use all those entities. It might not use all of those fields. That's fine. In your client description property list, you'll specify the subset that you use. That way the engine knows which entities and which fields to be giving to your client and it also knows what to expect from your client. So it won't erroneously delete things just because your client doesn't pass up certain attributes.

You can specify whether entities are push or pull only. Most of the time, you'll be both pushing and pulling entities. You'll be contributing to the pool of data. You'll be pulling in changes. But sometimes, for instance, in the case of an iPod, you'll only be pulling things down. The engine can make certain optimizations in its data store when you give it that information.

You also specify what type of clients you want to sync with. So if you're an application, typically when other applications sync the same data types that you do, you'll want to get notified so you can sync. When .Mac syncs, you'll want to start up. When devices sync, you'll want to start up. And we'll show that for our examples here, we specified that we wanted to sync when .Mac syncs and also when each of the other applications sync.

Here's a look at a property list. Very straightforward. We have the display name for the client and you can also specify, if I can get this to go here, an image path. When you have a user interface that presents a list of clients, which we provide for you, it's nice not only to have the name of the client, but to have some kind of an icon that represents it. Often it will be the same as your application icon, but in some cases you might choose something different, something that sort of illustrates that this is data that you're synchronizing.

So you can specify an icon relative to the path of this property list, and that way it will present something nicer to the user than just simply text. We also have a list of the entities as I described. Here we're specifying the event entity and the fields that we're going to synchronize.

I'm kind of going through these fast because this is pretty straightforward stuff. Okay, let's talk about syncing now. This is the interesting part. When you're going to synchronize, the first thing you need to do is register your data schema. Now, it's not that expensive to re-register the schema every time your application starts. You don't want to do it every time you sync if you can avoid it, but what you can do is start your application and just register the schema without worrying if it was already registered.

Typically, your data schema file isn't changing, so all this amounts to is a quick stat by the sync server. It checks to see if there's any differences, and if there's not, it actually doesn't do anything. So when your application starts up, make sure that your schemas are registered that you're using.

Then you need to register the client. Now, I'm sorry. I screwed up this slide. I've done this so many times. Let's talk a little bit more about registering the schema. I'm just going to show you the code. Now, can you -- I don't know if you can see this, and if you can, I'll talk through it a little bit more, but it's very straightforward. What you'll do is you'll keep your schema in a bundle in your application. You might as well keep the schema localized if it corresponds to your application. Now, we mentioned that sometimes data schemas are decoupled from applications.

Certainly from the engine's point of view, it doesn't make any assumption that a schema correlates to any one given application. If you're just providing a schema in your application for your own use, you can keep it inside of your resources. If you're not, you might want to put it inside of a framework somewhere so other applications can access it.

Once you have it, it's a simple path, and you make a call into the sync manager. Now, I'm introducing the sync manager and API here. There's just a few objects that you'll need to use in order to effect a sync. And the sync manager, as the name implies, does mostly management type of functions. You use it to register your schema.

You use it to register your client, as you'll see. Very straightforward call. You just pass in the path, and you're done. Now, the second thing I started to talk about is registering your client. Now, you only register your client if you haven't registered it before. So when we look at the code, if I can get to it, I'm convinced somebody's got a voodoo doll for this clicker somewhere.

There we go. OK. So when you're registering a client, you check to see if the client is already registered. So what you do is, by specifying the client's identifier, you ask the sync manager for your client object. If you get it back, great, you're done. You can just return. If not, then you'll need to register it. To register it is very similar to registering a schema. You just tell the sync manager that you want to register a client. Now the one difference is when you register a client, you provide an identifier.

provide both the client description file and the identifier so that you can refer to it again in the future, for instance, to start a sync operation. And the other difference is that you'll get a client object back, so you can then use that to proceed through the sync operation. Well, this is really hard.

Okay, the second thing you'll do when you register a client is specify an alert handler. This is pretty straightforward. We're just doing this so that we can sync when other applications or servers sync. In the example that we had, we specified programmatically to the engine that we wanted to synchronize when applications or when servers synced. We also specified an alert handler that gets called inside of our application. So you've got a running application.

While it's executing, if some other server or application goes off to sync, you want to get notified so you can join in and sync at the same time. That'll happen in the main run loop of your application, and it'll happen just with a simple callback that we're specifying here.

You should note that you could also specify what types of clients you want to sync with in your client description property list. It didn't really fit on the screen when I made an example of that before, and I also wanted to highlight that. You can do it programmatically. Okay, once you've got all that done, you're ready to sync data. So let's see what we have to do for that.

The very first time you sync, you need to do a slow sync. You do a slow sync because you don't really have any basis to compare to. So you're going to push up every record you have. After you've done that, you'll be able to fast sync the next time. And we've pointed out that you want to try to trickle sync as often as possible, and when you trickle sync, you only want to push up deltas to make it fast.

When your application is launched, you'll want to sync. Now, when we showed the demo before, we had a checkbox for trickle sync and a button for syncing. In a real application, you wouldn't have a sync button, nor would you have that checkbox. You would just have trickle sync behavior all the time.

When your application starts up, it would make sure that it had the most current set of changes, so it would sync immediately and pull changes down. Similarly, before you exit, you want to synchronize. If you synchronize before you terminate your application, that ensures that changes the user have made are not only flushed to a data file and saved, but they're also synchronized out to the rest of the world.

I've really got to figure this thing out. It's like if I pointed at somebody over there, it seems to go. OK, so we mentioned before that Sync Session is a finite state machine. So you have a certain set of steps you go through. I want to point out that pulling changes down is optional. If you want to, you can just push changes up.

So when would you want to do that? Possibly when your application's exiting. You'll just push your changes up and quit. When users quit an application, they don't want the application to sit there forever while they're waiting for it to sync. They want it to just get its business done and exit quickly.

So you want to make sure you don't spend a lot of time when an application's terminating going in a full sync operation. So you can opt out of this at any point. And in this case, you would just push your changes up and then exit. You do have to execute these steps in order, though. So let's look at them in more detail.

When you start a sync session, you can specify a blocking call or a non-blocking call. A blocking call would typically take a timeout. You don't want to call into a blocking method and then just wait forever and have your user locked out. If you call a blocking call from the main run loop, you typically want to specify a timeout around two seconds. After two seconds, you're going to get the little spinning beach ball. So you could specify an extra second and hope it doesn't happen, but typically you want the sync operation to start quickly or you're going to bail out of it.

With a non-blocking call, you give a call back into the engine, make a call that returns immediately, and then at some point in the future, the sync operation will start. Now, there's a couple of issues you have to be careful about. One is responsiveness. I just mentioned you don't want to go off forever waiting for a sync to start.

Secondly, if a user goes and makes modifications to data, they're going You want to make sure that you use the data at the point the session actually starts. So if your application decides to sync and makes that call to start a sync session, don't collect any data to use in the sync. Wait until the sync session actually starts and then you can use it. If you have a blocking call, don't loop.

If you call this blocking method and it returns without being able to sync, it actually returns yes or no whether or not you've got a session. Don't immediately call it again. First of all, you'll just be banging on the engine. Typically, the reason it returned no is because another client is syncing the same entity types as you. So if you have a phone that's synchronizing and then address book decides it wants to trickle sync but the phone's already synchronizing, it could take a while for that device to finish.

So if you ask the question to the engine to start a session and it returns no, you probably want to wait a sufficient amount of time before you try again. Or an even better approach is to just use the non-blocking call all the So here's some code to begin a session. And notice at the very top, the first thing that we do is we save our file. Now the code snippets I'm showing you are actually, they're taken from the demo. So here we save our data before we start a sync.

You don't want to synchronize data you haven't saved in a file because the next time your app starts, if it was unable to save, you're going to be out of sync, no pun intended, with the engine. So the first thing to do when you're going to synchronize, save all your data.

And then at the bottom here, you can see that we're starting a session. Now I actually committed an egregious crime here. I specified five seconds. I did that just for testing and I ended up leaving it in the slide by mistake. I'm using a blocking call and specifying a time long enough that the beach ball is going to come up if it takes more than two seconds. That's going to annoy a user.

So you typically want to keep that limited to two seconds or less when you use the blocking call. The next thing you need to do once you've actually established a sync session is negotiate. Do you want to do a slow sync or a fast sync? So I'm going to show a little code here from our app.

Even though we mentioned that you need to do negotiation up front and we sort of show it at the beginning of the sync operation, you can spread a little bit of it out if it's more natural for your app. And you'll see how we actually did spread that out.

Now in the previous talk, we discussed the syncing modes and then talked about how we don't actually have a call into the engine, I want a fast sync or I want a slow sync. There's no call back to your client asking you to do that. So we're not asking it what it wants to do. We actually have a set of methods that you can use because it's much more flexible. So in this case, we're checking to see if we want to do a refresh sync.

So we would do a refresh sync in our example if we lost our data file. So if we start our example code up and our data file is gone, we'll refresh sync. That way we'll restore everything from the engine. We'll also do a slow sync in some cases. We do a slow sync the very first time we ever do a slow sync.

Also, we catch errors with exceptions during the sync operation. If anything goes wrong during sync operation, on the next sync, we force ourselves to do a slow sync. So in this case, we're telling the session whether or not we've actually reset all the entity names so that we can refresh. That's the first line.

Or in the second if clause, we're actually checking and we're telling the engine that we want to push all of our records, which essentially amounts to a slow sync. But besides what you're doing, you're also asking the engine what it wants you to do. A user might have gone and said, "I want to reset every client from .mac." At that point, when you sync, the engine won't want you to push any records. You're going to be reset. Also, something might have gone wrong during the sync operation from the engine's perspective, so the next time you sync, it will want you to do a slow sync. And we'll see where we ask those questions in the API as we proceed.

Now we're going to go push changes. Very simple flowchart here just to show you what we're doing. If we want to push all of our records, then we're going to get every record that we have, convert it, and push it up to the engine. Otherwise, we'll just push up the deltas.

I want to point out one thing. When you're pushing records, you don't have to push an entire NSDictionary record. You can push something we call an iSync change. In this case, we've got one change to a record. We've changed the title of an event to "Sean's Birthday." So I've got a very small iSync change object. But if I wanted to push up the entire record, I'd have to push up all the relationships and all the other fields. And you can see that's a lot more information.

And that can add up if you're doing that for every record and you have a large amount, especially when you're doing an initial sync. So if you push up an iSync change, you'll save the engine a lot of time walking through an entire record trying to figure out what exactly has changed in it.

Here's our code. This is a little bit dense. I pointed out that negotiation is actually split up. I showed you a previous slide where we told the engine our intentions, but you also have to ask the engine what it wants you to do. So we're asking it if we should actually push changes for this entity type. We're walking through all of our entity types, so in this case, media, assets, objects, and event objects.

We're going to ask the engine, should we push them? There's a few reasons why it might tell you you shouldn't. It might be resetting you from state somewhere else, or you might have, in configuration, told the engine, I'm not going to actually sync this entity type. A user should be offered the choice. You might only want to sync events and not media assets. So if you turn one of them off, you don't have to remember it. You don't have to keep track anywhere. The engine knows that you've disabled it.

So when you ask it if you should push changes, it'll say no. If it wants you to push changes... Then you need to ask it if it wants you to push all changes. So I just mentioned before, sometimes the engine needs you to push everything and do a slow sync. This is where you ask it. Now the rest of this code is pretty straightforward.

We're just walking through all of the records, either every record if we're doing a slow sync and pushing them all, or just the changed records. And we're going through, and at the bottom you can see the call to session to push the changes from that record. So we're just simply pushing it up, specifying a unique identifier. That's it. So we're going to ask it if it wants you to push all changes. So I just mentioned before, sometimes the engine needs you to push everything and do a slow sync. This is where you ask it.

At the very bottom, I wasn't able to fit this on this slide, so I'm going to show you another slide that's the continuation here. If you're doing a fast sync, you need to explicitly delete your records. So what we do here is check to see if we've kept track of state and we have a list of any deleted records.

If we do, and the engine isn't forcing us to do a slow sync, then we push the records, the deletes up. Now why wouldn't we push the deletes up if we're doing a slow sync? Reasoning is, the engine knows what your previous state is. So if you push up your entire set of records, anything you don't push up, it treats as a delete. So that's the first step.

Now we come to the fun part: Mingling. Mingling is actually pretty simple. You're going to tell the engine that you're ready to pull changes and then the session is going to enter into the mingling state. If other clients are syncing at the same time, the engine is not going to return until they've finished pushing all their changes and it's been able to mingle from all sources. So this can take a while.

The engine is going to be doing field level differencing here, so it's going to be walking through all the changes that come in from every source that's syncing at this time and it's going to compare them on a field by field basis. You can call this blocking or non-blocking. If you call it blocking, again, you have the same set of issues.

You want to make sure that you're being responsive, so you don't want to go in and start mingling and waiting too long for the user and a blocking call. If you call it non-blocking, though, you have to be careful. If the user makes modifications to data while you're syncing, the engine is going to be pushing changes down to you that are predicated on the state you had before you started mingling.

If the user has made changes to any of the records that the engine also has changes for, you typically want to let the user win. So you need to have some mechanism in your application to keep track of changes that are being made during a sync operation and make sure they override any changes that come down from the engine.

So here's our call. We're going to make a blocking call in. First, we're going to build a filtered list of all the entities. This is similar to what we did when we pushed entities. We asked the engine, should I pull these entity types? If they're disabled, or if the engine wants to be reset from your client, it won't tell you to pull anything, so you may end up with an empty list. Then you simply ask the engine to prepare to pull changes. That's the call that goes into the mingling state.

Now, I committed an even worse sin here. Can anybody spot it? And if you can read this, you'll see I specified the date, distant future. So this application is going to wait forever. Now, we actually didn't do that in the example. I think I took this slide from an older version of it. This is an example of something to be very careful about. Here, your application is just going to go and wait for the engine forever. So if something takes an incredibly long time in another client, your client's going to be penalized.

Finally, we get to pulling. When you pull records down, you need to ask the engine if you're supposed to replace every record. If the engine's trying to replace everything because the user said, I want to pull everything down from .mac and clear everything and replace it from there on my machine, it's going to tell you to replace all of your records.

If that's true, don't go and whack your data store, because you don't know if the sync is going to be successful. Anything could go wrong. The user could pull the plug on the machine, something could happen that could cause your application to crash. Instead, just internally set all your data aside and take everything you get from the engine, and only when you're able to save that completely should you then throw out your other data.

This is pretty straightforward. You get an enumerator when you're pulling changes. The enumerator is pretty much like an array enumerator, except it contains iSync change objects. I talked a little bit about these before. This is an object that just contains a list of the changes for a record.

It's often more efficient, particularly when a record has a lot of attributes and there's only a few changes. However, sometimes with your client, the logic might be simpler if you could just get the entire record from the engine and just map it. You can do that by pulling the complete record out of an iSync change.

So you have your choice as to whether or not you want to use the fields that are indicated in the change or whether you want to pull out the entire record. We walk through the change enumerator. We pull out each iSync change with next object, and then we apply each one based on the change type. So let's look at that.

When we're applying the changes, we're going to use a two-phase commit. For each change that we get from the engine, we'll either accept, reject, or ignore it. If we accept it, the engine will assume that we've taken it and won't give it to us again, assuming we successfully complete the sync session.

If we reject it, the engine will assume, for some reason, we don't want it. Perhaps you have a device or an application that doesn't fit every field on it, so some of the fields that are handed to you, you just can't use. So you reject that chain so you don't keep getting hit with it.

If you don't do anything, the next time you sync, the engine is going to push that record at you again. Now, after you've gotten every record and successfully applied them or rejected them, if that's what you want to do, you tell the engine to commit. At that point, the engine says, "Great, whatever this client just told me, I'm going to believe is the truth, and I'm going to keep track of this." So the next time you sync, everything you accepted doesn't get pushed to you again.

Everything you rejected doesn't get pushed to you again. But if you don't make it to commit, if your application blows up before that or something goes wrong, the engine will then try to push all those changes back to you on the next sync. And that's good, because if something goes wrong, you don't want to lose all that data.

So let's look at the code for it. We look at the change type, we switch on it. If we have an add or a modify, we simply try to apply it. What we're using is a transformer here. If we're able to successfully transform it, then we tell the engine that we've accepted the record. Otherwise, we tell the engine that we're refusing the record.

So in our example, if there was something malformed in a record, something wrong with it, we would reject it. Otherwise, if the record looked good, we would accept it. You do the same thing for deletes. You take the delete down, check for consistency. If you've got that ID for that record it's referring to, you just delete it.

Once you're done going through all of that, make sure that you save all the data to your data store before you commit to the engine. You don't want to tell the engine, "Sure, I've taken all your changes because you've got them in memory," and then crash and not actually write them out to a file. So we're just showing that here. And then at the very end, we tell the session that we're finished and we're all done.

So I talked a little bit about state management before. I'll just reiterate, you need to keep track of the adds, the deletes, and the modifies that you do. That way, each time you sync, you can just push up deltas, which is much faster. You need to save this info if your application exits without synchronizing. If you don't save it, the next time you start up, if you try to fast sync, you'll have forgotten about the deltas from before. If you lose the delta information, make sure that you slow sync. Remember, the most important thing is correctness after that comes speed.

So I've just got a few best practices to describe to you before we close. Sync quickly and often. We keep talking about trickle syncing. I keep banging on this and saying it over and over again. It's really the best thing to do for the user. The more often you sync, the more frequently and the quicker you can sync, the more transparent it becomes, the less load on the system and the better the experience for the user.

Your data is getting moved around and it's where people want it at the time that it's changed. You want to be responsive. It's very important that you don't lose responsiveness in an application. Users hate it when they're banging in the text field or trying to do something or get a menu to come down and they don't know why the application is hung. You want to provide user feedback. By that I'm talking about something like a progress indicator or a progress bar, possibly a status line with some text telling the user what's going on.

Depending on how much data your application syncs and what type of application it is, if it's a pro app or if it's a user-friendly application, you want to be responsive. You want to be responsive. It's very important that you don't lose responsiveness in an application. Users hate it when they're banging in the text field or trying to do something or get a menu to come down and they don't know why the application is hung. It's very important that you don't lose responsiveness in an application. Depending on how much data or if it's just a simple application that users don't really pay much attention to, you might want to give them more or less information, but make sure that you show them something.

You need to offer choices to the user. If you go to quit an application and start syncing and you can't get a session within a reasonable amount of time, you might want to pop a panel up to the user saying, do you want me to synchronize your data before I exit? That way the user has more of a choice.

Similarly, when you start up, if you're not able to get a sync session right away, you might want to do a synch session. You might want to notify the user that you don't have all the data, but you certainly don't want to make a user wait 10, 15, 30 seconds for another client to finish syncing before your application starts.

Now, I put this here almost as a joke when I first wrote the slides, and it sounds kind of ridiculous. Don't corrupt the user's data. I might as well tell you, write an application that links and doesn't crash when it runs. But this is actually a critical point.

Keep in mind that when you start up, you don't want to have a user wait 10, 15, 30 seconds for another client to finish syncing before your application starts. So keep in mind that the data that you're synchronizing is not only being synchronized with your application, but it's being pushed out to other applications. So if you corrupt some data, it's going to fan out to other applications.

It might fan out to devices. It'll fan out across .Mac, and it'll have catastrophic effects. So it's very important to test that last 1%. Test all the boundary conditions. Make sure that you handle errors. It's really critical when you write a sync client that you get all of that right. It's very important. important to have correctness.

A couple people you can contact. We have Developer Relations. We also have Technology Evangelist. The names are here. Please also, we're looking forward to getting suggestions from people. File radars when you encounter problems and stay in touch with us. And don't hesitate to contact these two people if you have additional questions. So a couple more things you can look at for information. If I can get to it.

Okay. We've got a lot of reference documentation. We've got some concept documentation that's only available online. The reference is actually also available on the Tiger DVD that we gave you. Also on the Tiger DVD, the examples that we showed you, all the source code is there in a project. It's in Developer Example Sync Services. Have a look at it. You'll see everything I talked about in as much detail as you want.