[Silence]

Nick Colet: My name is Nick Colet [assumed spelling] [applause] I'm a [laughter] -- thank you. I'm a software engineer on the Core Data team here at Apple, and I'm excited to take you through what's new in Core Data in iOS 7 and Mac OS X 10.9. We've done a lot of great work in the Core Data framework over the last year.

We've made a significant number of improvements to key areas to improve performance, reliability and stability, but today, we want to take a moment to focus on just a few of them, beginning with iCloud. Now, we've made a number of improvements to our iCloud integration. We've been iterating on it for almost 12 months since our WWDC session last year, and we took into account a lot of your feedback that we've seen on developer forums, in radars and around the web.

And so, the improvements that we've made require us to revisit a few of the topics that we discussed last year, beginning with improvements we've made to the fallback store, as well as changes to how you set up the store asynchronously; and what happens with iCloud transition events like when the account changes.

We're also going to talk about some new API that we're introducing today. And finally, I'll introduce you to some changes that have happened across the organization and developer tools and frameworks to change how you live and work on iCloud. They'll make it easier to use and provide a lot more transparency than was previously available to you.

Then, Melissa Turner will come up and take you through some demos. And finally, Ben Trumbull will come up and give you an overview of several enhancements that we've made to the SQLite store. Now, before I dive into the meat of the discussion, I'd like to take a moment to frame it by talking about our goals and objectives related to iCloud. You see, last year after iOS 6 and Mac OS X 10.8 shipped, we took a look at our iCloud integration and tried to identify several key areas where we could focus on providing valuable improvements toward integration, the first of which is speed.

This year, we've made a number of improvements that will change not only how efficient our integration is in terms of its memory usage, its processing usage and battery consumption, but also how well it deals with iCloud under the hood. We've made a number of improvements to downloading files so that changes appear faster and peers can communicate their changes faster than ever.

We also strove to improve simplicity. You see, after we shipped last year, we felt the developers still had to write a number of complex pieces of code to integrate successfully with our iCloud integration and we found that this hampered your ability to build effective applications that worked on top of our integration.

And finally, we really want to focus on consistency. As you may have noticed, we didn't have a clear paradigm that you could reuse over and over again, when working with our integration, and this made it exceptionally difficult, or at least more difficult than we had originally planned to handle transition events like when the account changes. So with that in mind, let's take a look at the fallback store.

As you may remember, the fallback store is a concept that we introduced last year at WWDC as a way of providing a local persistent store that your application could use in the event that the iCloud store was unavailable for a certain period of time, and your application would simply add the store to the coordinator to make it available to your managed object context so that your user could write changes to disk while in the background we set up the iCloud store for you.

And this would produce a pairing, if you will, of a local persistent store file and an iCloud persistent store file that were tied to the same account. Now, this worked out rather well until, you ended up with multiple accounts, and this is when things became a bit cumbersome because developers would end up spending a lot of their time writing codes and manage these different persistent store files instead of focusing on building our applications. And so for iOS 7 and Mac OS X 10.9, we've completely eliminated the need for you to manage the fallback store. We've subsumed that responsibility into the framework and it will be entirely managed by Core Data.

This also means [applause] [laughter] -- thank you. This also means that your application will only ever use one persistent store file per account. Now, we say "per account" because we still have to provide a different actual persistent store file for each iCloud account that's in use on a system, and this requires that we change a bit of our guidance from previous years. You see, we had previously advised that you could keep the iCloud store inside of the iCloud container in a .nosync folder. This had a rather convenient advantage of automatically deleting the store file for you when the account went away.

However, because we now take responsibility for managing the fallback store and the fallback store must exist in local storage, we now require that you store the persistent store file somewhere in local storage, usually the application sandbox. However, if your application isn't sandboxed, you can keep it wherever you normally keep your persistent store files.

Additionally, and because this is now being taken over entirely by Core Data, we want to provide a bit more transparency than we used to about events that are happening related to the fallback store, and for this reason, we'll log some new events to console so that you can see them during your development.

You may be familiar with our Core Data logging. All of our messages start with CoreData: Ubiquity: and when we're using the fallback store, you'll see Using Local Storage 1. You'll see Using Local Storage 0 if we've been able to transition off the fallback store and your application is fully connected to iCloud.

Now, it's important that you understand the expectations associated with each of these log messages. Using Local Storage 1 means that changes that you make to the persistent store are being persisted vocally and they won't be available to other peers over iCloud until you see Using Local Storage 0.

This is when you can expect to start seeing import notifications. And when you should expect seeing your changes uploaded and made available to other peers. So that's the fallback store, and now I'd like to tell you about some of the changes we've made to how you set up the store asynchronously.

You may be familiar with this line of code. It simply adds persistent store to an instance of NSPersistentStoreCoordinator; and when you call it, you expect to get a store file back fairly quickly. In most situations, especially those without iCloud, we only have to do a small amount of network -- of local I/O to bring the persistent store file up, so it returns immediately. However, when it is connected to iCloud, we sometimes have to do a significant amount of network I/O to make sure the persistent store is in a state that's consistent and ready to be used by your application, and in iOS 6, we did this synchronously.

That meant that whenever you called add persistent store and you didn't have a persistent store file on disk, we would go out to the network and discover the state of the contents of the iCloud container. In iOS 7 and Mac OS X 10.9, we no longer do this work synchronously within the call to addPersistentStore, so addPersistentStore will always return immediately.

And this is a huge benefit to your application because it eliminates the need for you to call addPersistentStore on a background thread. Now, have we done this? Well, we've done it by using the fallback store. We will transparently pass the fallback store back to your application and so that you can begin working with it and your user can begin making changes immediately.

But, because we manage the fallback store now, we were able to take the integration one step further than we felt developers could on their own. We now store individual transaction records for each change that's made to the fallback store and this has implications for both performance and bandwidth.

You see, last year, if your application took responsibility for migrating contents from the fallback store to the iCloud store after the iCloud store was available, you would have to do a lot of extra work to scope that migration and ensure that it didn't consume too much memory or bandwidth over the wire.

However, now because we keep our record of each individual transaction, all that work is done for you. And this means that once the iCloud store is available, we can transport those changes into the container immediately and make them available to other peers. So there's no additional work for you to do the migrate data from the fallback store to the iCloud store.

But we also wanted to give you a better way to tie in with this transition and that requires that we introduce some new API. Today, we're introducing NSPersistentStoreCoordinator Storage Will Change notification as a way that we can tell you that an event has occurred where we need to change the contents of the persistent store file on disk. When you receive it, you can optionally call NSManagedObjectContext save if there are changes in your managed object context that you wish to persist, and then you can call NSManagedObjectContext reset and prepare your UI for new data.

When we get control back from your notification observer, we'll remove the store from the coordinator for you and then post NSPersistentStoreCoordinator Storage Did Change notification when the store is ready to come up. Now, it's important that you realize that this transition is very fast. It's almost as fast as if you called addPersistentStore on a store file without iCloud options, so you shouldn't expect to notice any interruption in what your user sees in the application.

You can then start working with the storage just as you normally would, and call NSManagedObjectContext save to make changes to the iCloud store. Now, once you've handle this transition, your application will be fully up and running on iCloud and you can expect that at that point changes will be available to other peers.

Now, I need to mention that this is a special case scenario. You will only see these notifications during our asynchronous setup process if there is no persistent store file on disk, or there are has been a change to the iCloud account or the contents of the container since we last launched. Normally, you'll just start seeing NSPersistentStore Did Import Ubiquitous Content Changes notification which you can merge into your managed object context by using merged changes from Context Did Save.

This is important to realize because the goal of our integration this year for asynchronous setup was to eliminate any work that you had to do by your application at all. However, we're providing this new transition paradigm as a way of enabling you to have a consistent implement once use case that you only have to pay attention to for our integration.

And so with that, let's take a look at changes for how you handle changes in the iCloud account. Normally in iOS 6 and Mac OS X 10.8, when the iCloud account changes, the only way to notice this is by subscribing to the NS UbiquityIdentityToken Did Change notification. When that happens, you have to remove the persistent store from the coordinator and then replace it with a new one to talk to the new account.

In code, this looks a little bit like this. You call NSManagedObjectContext reset and then remove persistent store, and then you call NSPersistentStoreCoordinator addPersistentStore to add the new file to the coordinator for use of the new account; and this would work just fine in iOS 7 and Mac OS X 10.9. However, because we control the fallback store and because we have a new integration for asynchronous setup, we thought that we might as well extend that to account changes, as well.

And so now, you'd no longer have to subscribe to any of the iCloud notifications at all. You simply implement your will change handlers and respond to NSPersistentStoreCoordinator Stores Will Change and will notify you automatically when we need to change the persistent store file because there's new account on the system.

Of course, you can then call NSManagedObjectContext save and NSManagedObjectContext reset. Now once you've done that, we'll remove the store from the coordinator just as with the asynchronous setup process and then we'll send you NSPersistentStoreCoordinator Storage Did Change notification, again, just like asynchronous setup and you can begin working with your application as you normally would. Now, let's talk about this in a little bit more detail.

When you receive NSPersistentStoreCoordinator Stores Will Change notification, the persistent store is still available to use, and so unlike what we advised you of last year where you had to immediately drop the persistent store and wipe out your managed object context, you can still write to the managed object context and those changes will be persistent locally to be imported to the account if it every comes back. This means that although your user's changes won't make it to iCloud immediately, if they ever sign in again, they'll be there and waiting.

Another thing you should know is that once you receive NSPersistentStoreCoordinator Storage Did Change notification, the store file may no longer be available to you to use on disk. This is because Core Data automatically manages all of the stores associated with the account. You provide us a single store URL inside the application's local sandbox and we then create an opaque container with an entry inside of it for each account on the system, including the local account, which is our term for what happens when there is no iCloud account on the system. This is a special store that's managed by Core Data so that you don't have to do anything special because your user doesn't have an iCloud account.

How does this work? Well, you give us a store URL somewhere inside the application samples, and we chop that up and put in a special root directory called Core Data Ubiquity Support, followed by a directory tree that allows us to uniquely identify each store and tie it into an account.

Now for simplicity's sake, we've kept the persistent store file name the same as the one you pass into us so that you can identify the store files inside of this container if you need to. Finally, all of these store files will be managed by Core Data and that means that we could remove them at any time.

Each store will be removed once its account has gone away because we can rebuild the file from the cloud. So we want to free up as much disk space as possible for your application to use and not have old store files lying around that could take up additional resources. And so that's our changes to the account. So now let's talk about some new API that we're introducing this year.

I'm sure you recognized this line of code. This calls NSFileManager's URLForUbiquitous ContainerIdentifier method, and it allows you to get the URL inside of the iCloud container where your application can store files and write data. However as you may know, when you change iCloud accounts, this method can take a little while to return, and so we couldn't provide you a truly seamless and fast integration without a eliminating the need for you to call it.

Normally, you take the value from this method and pass it to us and as an option called NSPersistentStore UbiquitousContentURLKey, and this is how we know where to keep all of your data in the iCloud account. So in iOS 7 and Mac OS X 10.9, you'll no longer need to pass a value for that at all, and we'll call URLForUbiquitous ContainerIdentifier automatically under the hood for you.

Now as you may remember, our advice in past years has always been that you should keep our transaction log content inside of a special subdirectory in the container, and you may be wondering how it is that we'll know where to find this if you're not passing a URL into us. Well, NSPersistentStoreUbiquitous ContentURLKey now takes a string subpath, as well.

This is optional, but if your application already exists in iCloud and you have a special subdirectory where you've been keeping all of our content, you need to pass this value in so we can find it. Otherwise, we'll create our own value by default called Core Data Support. This is a special directory we create in your iCloud account and store all of our transaction log content in.

One of our main goals this year is to provide a very simple integration; and that means that we really need to consider all of the cases of applications that are going to be in use with it. So this year, we're introducing NSPersistentStore UbiquitousContainerIdentifier key as a way for applications that have multiple iCloud container identifiers in their entitlements.plist to tell us which one they want us to use when we call URLForUbiquitous ContainerIdentifier.

So how does this work? Well, if you're application has multiple container identifiers in its entitlements.plist, by default, NSFileManagerURLFor UbiquitousContainerIdentifier will select the first one if pass No. However, if you want to use something that isn't the first one, you need to pass us this option so that we know which one to use, and this value will be passed directly to URLForUbiquitous ContainerIdentifier under the hood.

We're also trying to help you manage iCloud content and for that reason we're introducing NSPersistentStore Rebuild From Ubiquitous Content option. This is an option that allows you to remove the persistent store file on disk and examine what happens when we rebuild it from the iCloud content. It's important that you know addPersistentStore will always return an empty store when you pass this option, that's because we need to switch over to the fallback store while we -- excuse me -- while we rebuild the iCloud store from the transaction log content in the cloud. So how does this work? Well, you just pass it -- pass us a [inaudible] that evaluates TS inside the options dictionary, and we go off and replace the store file with one that's freshly built from the cloud.

We're also introducing a new option to help you create backups or local copies of the iCloud persistent store called NSPersistentStore Remove Ubiquitous Metadata Option. This removes all associated metadata from the iCloud store; that means, anything that we write into the metadata dictionary as well as the store file itself, and it's critical if you want to use the migration API to create backups or local copies at a persistent store you wish to open without the iCloud options.

Finally, we're adding a class method to NSPersistentStoreCoordinator which removes all of the iCloud content and the persistent store files on disk. This is our way of providing you with a clean slate. If you call this method, we will go into the iCloud account and delete all of the content associated with a given persistent store. You pass us the store URL and the options dictionary that you normally pass so that we can correctly identify the store and its content in the iCloud account.

Now, I can't over state this enough. This is our clean slate. If you call this method, none of the data will remain on disk or in the iCloud account that's associated with a given store, but it is the easiest way for you to start over. And because of that, we've made this method synchronous.

There is a significant amount of network I/O that it has to do under the hood when it talks to iCloud, and because of this, it may take a little while to run. However, you can't work with a persistent store until it's finished. And so, it will be synchronous and once it returns successfully, you can work with a store again.

How does this work? Well, you'll have some content and a persistent store file on disk and we'll erase the persistent store file first and then nuke the iCloud content, as well. And we have some special integration under the hood to make sure that this is robust and reliable as possible.

It actually only requires one I/O operation with iCloud to propagate the delete to all of your devices and that makes it very fast and very robust, but we still recommend that you only do it when you have a good network connection such as an Ethernet connection from a Mac. It works just fine on iOS devices, however, if the connection isn't great or iCloud isn't available on that device, you won't see the change propagate for quite some time.

Now as you've already seen, we're introducing a new notification this year called NSPersistentStoreCoordinator Storage Will Change notification, and this only applies to iCloud persistent stores. You won't see it with any other type of persistent store on the system as it's our way of creating a consistent and reusable paradigm for handling transition events related to iCloud.

The user info dictionary will contain instances of NSPersistentStore that identify the iCloud store that's about to change, and you can subscribe to it just like any other notification. We recommend that you scope this using the persistent store coordinator that your application uses to talk to iCloud as the object when you subscribe to the notification.

As well, you should subscribe to NSPersistentStoreCoordinator Storage Did Change method -- or, sorry -- Storage Did Change notification and scope that by the persistent store coordinator, as well. And this is because the implementation of your Did Change handler will probably be slightly different for the iCloud store than it will for the other persistent stores on your system.

So let's take a look at those notification handlers. Now, I mentioned earlier that our goal for this year was to provide a very simple integration that significantly reduced the amount of complex code that developers have to write. And so, with the Will Change notification, we've tried to make it as easy possible to get up and running and all you have to do is optionally call NSManagedObjectContext Save and then NSManagedObjectContext reset.

This will completely prepare your application for use with the new iCloud store. Of course, you may have some custom code you need to run to prepare your user interface for this. Now as I mentioned, this transition will happen fairly quickly and so you should not need to block your UI while you wait for the store file to be swapped out. However, you may wish to prevent the user from trying to write new data to disk such as blocking a Save button which you can then enable inside of the Did Change handler.

And as you can see, this is only a handful of code. In fact, most of the sample applications that we use internally have completely changed by reducing a ton of code to just a few lines to integrate with our iCloud integration. And so that's all for the new API.

Let's talk a little bit about how things have changed for living on iCloud and iOS 7 and Mac OS X 10.9. There have been a number of substantial improvements to the developer tools and frameworks that are available to you as well as the underlying infrastructure related to iCloud.

Perhaps one of the most significant is Xcode's new iCloud Debugging pane. This is a pane that automatically runs inside of the debugger when your application uses iCloud, and you can see the amount of storage space you consumed as well as the status of a given iCloud account whether it's idle, uploading, downloading, or actively working to set up the account.

But I think the real gem here is the file transfer graph. This shows all of the activity on the iCloud account, both uploads and downloads, and so you can see any activity that's generated on your behalf by the Core Data integration under the hood. You can also see a list of every single file inside the container, including our hidden directories and special files that's in use by the Core Data integration; and as you can see, the status of each file is printed in the column to the right.

This will let you know whether or not a file is what we call current on disk, meaning that it has been downloaded and is available to the Core Data integration to use. And so if you're looking through our transaction log directories, you can actually see whether or not files from other peers that were changed recently have been downloaded and are available to us to import. So this can be a big help when you're wondering why you're not seeing NSPersistentStore Ubiquitous Content Changes notification without having to enable a ton of logging.

The iOS simulator also now fully supports iCloud. You can use the same account with your application that you're using on your Mac or a completely different one, and this allows you to test iCloud integration without having to change settings on your Mac. As well, you can also test sync between a Mac and an iOS device without actually connecting to any devices, and to me, this is a huge advantage because it allows you to evaluate your cross-platform integration without having to worry about carrying devices with you. It fully supports iCloud document storage and the iCloud key value store, as well, you'll also receive support for pushing notifications now in the simulator [applause]. [Laughter] Thank you.

But it wouldn't be a Core Data session if I didn't tell you about logging. Now, all of this has existed in our iCode integration for the last couple of years. But, in iOS 7 and Mac OS X 10.9, we've made it possible for you to enable the logging by setting a user default inside your application.

Of course, you can still pass it as a launch argument using com.apple.coredata. ubiquity.logLevel. Number 3 is the highest, so if you turn that up, expect to get a lot of logs. Of course, we also have com.apple.CoreData.SQLDebug which allows us to see what activity and what SQL we're generating when you use the iCloud store.

And I have to mention that if you're having problems with iCloud related to your local managed object context that is the one that's in use by your persistent store, this is a critical tool for us to help you debug that because we need to see what activity your store is at -- or, sorry, your managed object context is actually attempting to do in relation to the iCloud store. Of course, you can pass both of these arguments as a launch argument through application inside of Xcode scheme editor.

There have also been improvements to the underlying logging system related to iCloud itself. On OS X, we're introducing a new tool this year called ubcontrol, and ubcontrol interface is directly with the underlying daemons that talk to iCloud. It allows you to do things like enable debug logging and you do this by calling ubcontrol-k7. Now, 7 is the highest logging level that they allow and it's what we need to debug any iCloud issues related to the actual file transfer on the system.

So if you see a bug, enable this, reproduce the issue and then follow radar. Of course, on iOS, we still have the iCloud Debug Provisioning profile and this is available from developer.apple.com as a download. You can download and unzip this file on your Mac and then email it yourself. Finally, on iOS, open the email. Follow the onscreen instructions and then reboot the device once the profile is installed.

It's critical that you reboot your device because this is what restarts all of the daemons on your iOS device that talk to iCloud. Without it, they won't pick up the new logging level. Once you've reproduced the issue, you should sync your device directly with iTunes and then gather the logs from Library/Logs/CrashReporter /MobileDevice and then the device name and diagnostic logs, and you can file a bug with those in it and we'll take a look.

Now, I want to talk for a minute just about filing bugs. We received a lot of bug reports from you over the last year and this has been critical in not only helping us scope our integration but also decide what areas of improvement we can provide to you that will have the most value.

So when you file bugs, please, please, please include all of the logs that you can, and if possible, your local persistent store files because this helps us get all the information that we need to identify the issue that you're running into. So with that, I'd like to bring Melissa Turner up and she'll take you through some demos of a few other changes that we've made this year.

I'm Melissa Turner. I'm one of the Core Data engineers, and I totally thought I was going to go out talking on stage at WWDC this year until a couple of weeks ago when my manager came to me and said, "Hey, Melissa, you totally are the one who's going to see all the weird esoteric bugs that nobody else is going to hit. I want you to do a demo, take one for the team." So here we are.

We'll do it this way. If my demo fails gloriously on stage, I'll have found bugs that we will then not be shipping to you, guys. OK. So here we have a little application; it's a note-taking application. I'm going to add a note. I need to add another note.

It's pretty simple. Not much to see here, but as important as what you're seeing is, what's more important is what you're not seeing. And what you're not seeing here is, well, a hang as I wait for addPersistentStore to return with my ubiquitized store. You can see that as I was sitting there waiting and as I was talking to you, as I was working with that responsive UI, our iCloud integration off in the background was going off and fetching information about our lab times today -- for the next week. So, you can see as Nick said, this is the asynchronous store setup. We've made things a lot better, a lot smoother, a lot easier for you, guys -- one line of code.

So, because I'm me and this is a demo on stage, there was a good potential something was going to go wrong and I was going to have to come in here and try debugging whatever had gone wrong on stage. So this is, as Nick said, the iCloud Debugging pane and it comes up whenever you are trying to debug an application that uses iCloud.

The top left-hand side, you can see some information about your application and about your iCloud status. If you bring up this pane when you're running an application and all you see is the little cloud on the left and a thing saying, "iCloud not enabled," well, that would be why you're not seeing any of the data that you're expecting to see from the cloud because you haven't enabled iCloud on your computer.

There's a storage meter, it tells you how much of your quota is there. If you're not seeing data and that meter is pegged red, well, probably you're not seeing it because the client couldn't write it. Below that, we have the transfer activity gauge. You can see some green stripes there.

This coincides with those notes I created that were then pushed up into the cloud. Green stripes are data being pushed; blue stripes are data being pulled. Because this is a Mac and it's [inaudible] up here and it's been sitting on stage connected for a while, we don't see any blue stripes. All of the data that would've been downloaded for this application was downloaded well before we launched.

And below that, we have the documents in data view. This is showing you a listing of all the files that are in your ubiquity container, and you really don't need to pay attention to the stuff that starts with a dot, that's our private stuff; you're not going to be able to figure out very much from there.

But under that, you'll see some directories that have an accountname.uuid and these are your peer containers. And digging notes inside those can be interesting because that's where you'll see the Core Data receipts. And if you've got data coming, you're debugging an application on two pairs and you're expecting data from one to show up in the other, this is where you'll find out whether or not that data has actually made it through the cloud. If you've got five receipts on one file and only three -- on one machine and only three on the other, well, that's some transactions that have not yet made it across the network.

Transaction logs appear in these lines and you can see over on the right-hand side the status whether data is current, whether we know it exists but it's still in the cloud because it's a large file and we haven't managed to download it yet, whether it's current but waiting to be pushed to the cloud.

So, and here's that using local storage that Nick was talking about -- logs that Nick was talking about. We're not using it in this case; we're using the cloud. So, on top of that, Nick also talked about account transitions, about how Core Data has taken over managing the iCloud state transitions for you. And, well, how to demo that? Well, that's pretty easy. You need to go off. I'm going to go into system preferences. I'm going to hit the Sign Out button.

And voila! Automatically, we have noticed that the account was signed out, sent you a notification saying -- oh, by the way, that file is going away, and my application received that, and basically refreshed its UI and you can see all of the data that was in the cloud is now gone. That's because once you remove an account, well, you no longer have that iCloud data. It's off in the cloud, not on your local system. So our local store, because we had not created any data before we logged into cloud is empty.

So, well let's signing out and you're probably saying, "Well, sign out? Yeah, that's easy. How about sign in? Can you do that?" Well, we can do that, too. And to show you that, I'm going to actually switch over to my other computer here, and actually [inaudible] that demo with the First Launch Experience demo.

So, for those of you who are extremely perceptive, you might notice that up here in the corner I'm not connected to Wi-Fi, and those of you who are in the front rows can probably see that I'm waving Ethernet cable at you. This laptop is not connected to the network and has not been connected to the network since I airdropped my project on to it. So this is your canonical first launch situation.

The application has never seen the network, but there is data in the cloud out there. So how does this work? Well, we need to run my application and it comes up and as we expect, it's empty. We're not attached to the cloud, no reason there should be data there. Create an offline note. OK? It's a note. And then I'm going to come into system preferences, but first I'm going to find my Ethernet cable and plug in my Ethernet cable and log in.

I don't care. I don't care [laughter]. Do show me documents and data. Let's see. And it looks like something has gone wrong, of course, because something would have to go wrong. Oh well. You could see that we actually -- oh, I see what's happened. We're blocking waiting for the initial sync to happen. This is the log you'll see when Core Data tries to go out and connect to the iCloud account, will basically have to block as we sync data in.

And eventually because, you know, you guys are probably hammering the network and that took a little while, we come back and we say, "OK. Now we're not using the local storage anymore. But you can see that offline note I created is still there. So, yes, this is what we've been working on. This is how we've been trying to make your developer experience with iCloud and your customer's experience while using iCloud a lot easier.

You've seen asynchronous setup, it worked. You've seen account transitions to and fro and you've seen, you know, how you might go about debugging issues that you're seeing with a Core Data application. And at this point I'm going to bring my manager, Ben, on stage to talk to you about the changes that we've been making to the Core Data SQLite store.

Great. Thank you very much, Melissa. Good morning, everyone. So I'd like to talk to you a little bit today about the changes we've made to some interest structure items underneath it covers here with the Core Data and the SQLite store, and there's one in particularly, which is we've changed the default journaling mode that Core Data is using with the SQLite store.

And this change is going to be active for every app that rebuilds against iOS 7 and Mac OS X 10.9. Now just to sort of recap, we've been using the original SQLite journaling mode, the rollback journaling for years now -- since about 2004, and we're switching over to the write-ahead logging journal mode. So this is going to be something that apps need to accommodate.

So the difference is that in rollback journaling, SQLite copies the original pages out of the main database file into a file on the side just in case something goes wrong with the transaction, and then it updates the main database file directly in place. And when it's done, in order to do the commit, it just deletes the journal file.

For write-ahead logging, it actually leaves the main database file alone and it just keeps appending transactions to this WAL file, and it uses the shared memory file to keep track of which page it -- the most recent copies of each individual pages. And then once you've aggregated enough [inaudible] together, a series of transactions, several megabytes of data, it'll perform a checkpoint operation automatically to merge the WAL file back into the main database file.

So, to give you a little diagram of what exactly is going on here, generally, as you're going along with the SQLite in the rollback mode, you'll only see the main database file. But, when you start a transaction, you'll end up with this intermediary-journal file floating alongside of it. And then when you commit the transaction, it'll go through and basically delete the journal file.

In WAL mode, you're going to end up having all three of these files around almost all of the time, and this is one of the behavioral changes that you'll need to sort of accommodate yourself. So what ends up happening is when you begin the transaction, it'll start appending these transactions to the WAL file.

And unlike delete mode, when you come through and you commit the transaction, it actually does almost no additional work, so the transaction stays appended to the WAL file. And then, once you've aggregated several megabytes of data in the WAL file, it'll come through and then perform a checkpoint operation. And the checkpoint operation will merge the WAL file back into the main database file.

And then you may end up sporadically in a place where there is no WAL file, the WAL file is being truncated, but that's going to be a fairly transient state of affairs, and generally you're going to see in WAL mode that you have all three of these files.

So, the reasons why we're doing this is that on the consumer grade hardware we've experienced, there have been some difficulties in getting the level of reliability that we really want to have across kernel panics, and power loss, and some very severe failure modes, and updating the main database file in place has been somewhat difficult to manage. So under WAL mode -- because that's not happening, we're able to do a more expensive sync operation that's basically being managed by the WAL and the checkpoint aggregating several megabytes of work together.

We also see some very substantial performance improvements because we're doing fewer syncs. There isn't any actual syncing going on in WAL mode with the individual commits, right? So it's just appending these things to the WAL file. And this has actually been a pretty big difference on [inaudible] devices and iOS.

And finally, then we see some improved concurrency. Under WAL mode, it's possible to have multiple readers going concurrently with the writer. So, I'm going to talk about this in a second, but it's actually much better than the standard reader/writer lock model that rollback journaling uses. And we've adopted this internally at Apple across the system and it's seen some great benefits, so that's part of the motivation of rolling this out.

So WAL databases support -- or write going concurrently with the readers as opposed to blocking out readers the way it does in the traditional journaling. Now for Core Data itself, NSPersistentStoreCoordinator still serializes each individual fetch and save. So to maximize concurrency using WAL journaling within Core Data, you're going to need to have two Core Data stacks.

So for leveraging concurrency, you have this SQLite file that you're sharing between multiple coordinators; and you'll set up an additional coordinator and managed object context for each of these types of operations and they can both be pointed at the same SQLite store. And with this kind of setup you have, you can basically have one of these coordinators doing reads while another one is doing writes.

Now when you're using multiple coordinators, you can't really pass any objects between them, so there is some friction there, and this includes object IDs. So if you want to pass references between these two stacks, you'll need to use the URI representation. Nevertheless, this is actually excellent for doing background imports where you have a very segregated stack that's sort of siloed away from the main UI and everything else you're doing.

You can do importing, all kinds of background batch changes, and stuff like that, and it allows the UI thread to remain unblocked and to do a lot of aggressive reading or faulting and fulfill the UI and the main thread while you're doing a fairly substantial write operation in the background. So this is particularly useful for responsiveness around launch time where you might want to be importing changes from some other data source around the time the user starts up the app again.

Now, there are some sharp edges using the WAL database file, but we don't recommend this for any kind of file that you're going to be moving around yourself. So if you're using file system operations now on the database file, then WAL mode is probably not going to be very compatible with the code you have because there are going to be two additional files that your code probably isn't prepared for. And we don't recommend this for read-only files, so there's been some issues there and this is primarily useful for library files that you're going to be writing.

And finally, we don't recommend this obviously for a document format that you already have if you have a standalone SQLite file as part of your document. Now, you can put a wall of database inside a document package, and that's fine, but a lot of people we've seen have sort of free standing SQLite files as their documents and some of them is running to some challenges where the journal file itself is also outside of any kind of wrapper package, so that can be a little bit of a friction.

We definitely recommend that if you're going to use SQLite for a document file, you put it inside a package wrapper. And something else to keep in mind is the WAL journaling really only goes back as far as MAC OS X 10.7. So, if you need to communicate these files persistently with older systems, you'll need to use the rollback journaling.

So to get back to rollback journaling, the SQL command is fragmentjournalmode=delete, and to set this in Core Data, you just set the options dictionary that you passed addPersistentStore with type, and you just pass in this pragma on the journal mode to set it to the delete mode. And this will get you back to the same behavior you had in previous releases.

Now, some general caveats is you really shouldn't be using any file system routines directly against any open SQLite database file and we've seen a lot of people try to try to like delete files or maybe they're trying to make backup copies or something like that. And this really messes up file locks and a bunch of other things that are going under the hood and at its core using NSFileManager or at the POSIX file routines, it's going to be bypassing the SQLite API. So if the files open, we really strongly discourage this and you have gotten away with this in the old classic delete mode for journaling but it's not going to work out very well under WAL journaling.

Also for network file systems, we've seen a bunch of people try and they keep trying. But at the end of the day -- at the end of the day, the caching that's going on in the kernel is not going to be coherent between multiple different physical machines. Right? So it's OK to have an SQLite file in a network home directory that's being used by a single machine, but sharing these files across multiple different machines simultaneously is not going to end very well.

And finally, a lot of people try to change for performance reasons the synchronous pragma, and especially under WAL mode, we've customized this behavior for Apple's build of SQLite, so please just don't change it at all. And if you want to continue doing whatever it is you're doing with the rollback journaling, that's fine.

And as part of some of this other work, there are general infrastructure changes going on with the SQLite store, in particular to integrate with the power and I/O changes, the throttling changes that have been talked about earlier in the week to improve battery life on Mac OS X. So, we've done some extra work in Core Data to integrate with I/O throttling and improve the amount of battery time that SQLite is using if you have multiple contending connections.

And finally, there's also a guarded file descriptor concept that's being used. We're not going to go into this very much, but you should be aware that it exists and SQLite is going to be making efforts to prevent accidentally corrupting databases by smashing the file descriptor by closing a file descriptor that doesn't belong to you and things like that. So that's available on Mac OS X 10.9 and iOS.

And so that's most of the infrastructure we've done this release, and for more information, you can contact Dave DeLong, or send your email to the Cocoa feedback group, and we have documentation in all the usual places and we have some related sessions. Tim Isted is going to be doing a Core Data performance talk this afternoon and there will also be a -- hidden gems in Cocoa and Cocoa Touch on Friday. Thank you very much. [Applause] [Silence]