Welcome everybody.

My name is Eric Tamura, and I am a manager on the file systems team primarily for iOS and our embedded devices, and I'll be talking to you about what's new in Apple file system. I'll also be accompanied by Pavel Sokolov, who's a manager of the file systems team largely for the Mac.

So let's get started. So this presentation will roughly be divided into two sections. I'll tell you a little bit more about what's new for iOS as well as tvOS, watchOS, and then Pavel will come up and tell you all about what's new on macOS. So let's jump right in.

So last year I got up on the stage at this conference, and I told all of you that Apple [inaudible] Apple file system, as the default for all devices starting in 2017, and we've certainly done that. So as of iOS 10.3 as well as our tvOS and watchOS analogs, all devices have been migrated to APFS. We'll tell you a little bit more about that.

We'll talk a little bit more about the conversion process, how did devices actually move from HFS+ to APFS, and then a little bit on normalization and case sensitivity, particularly as it relates to iOS devices. So let's talk about that conversion. So at this point, millions, literally hundreds of millions of devices have now been converted successfully to APFS.

So starting on iOS 10.3 as well as tvOS and watchOS, Apple file system became the default. We had millions of devices converted. And one of the things you might have realized that this update actually does free up some storage for end users. In particular, there was space previously occupied by the prior volume manager, lightweight volume manager, which is no longer needed, and we were able to free up that space and give it back to users.

One of the other things is that starting in iOS 11, we are able to start taking advantage of the APFS capabilities in iOS and tvOS and watchOS. Particularly iOS where iCloud backup can now use Snapshots in order to take a stable copy of the file system and upload everything to the Cloud without worrying about whether or not a process might be modifying a file and getting rights intermingled. And finally we've been listening to developer feedback in order to improve and refine the file system and all the software that we've been working on. So let's talk a little bit about that conversion process.

So the APFS conversion occurs in line during the update to iOS 10.3 and beyond. So whether you're moving to iOS 10.3, 10.3.2 or even iOS 11 from some prior update, it will automatically get converted. And one of the reasons that we were able to do this so successfully is that we actually employed a dry run conversion during iOS 10.0, 10.1, and 10.2 and be able to track how the health of our conversion process was actually going. And we've also been monitoring the health of the conversion during iOS 10.3 and beyond updates, and we've been watching that as well.

Finally, we've been also looking at the numbers and statistics and everything that we're watching and be able to improve and refine the conversion on the Mac as well. So sometimes an illustration can help make this easier to understand, so I've got one here. So let's assume that this rectangle represents the entirety of a block device and on the far left you'll notice that there's and SB that really refers to the superblock or the HFS+ volume header.

Let's say for simplification we have three metadata regions that report all the entirety of what the file system might be tracking, things like file names, time stamps, where the blocks actually live on disk, and that we also have two regions labeled file data, and if you recall during the conversion process the goal is to only replace the metadata and not touch the file data. We want that to stay exactly where it is as if nothing had happened to it.

So the first thing that we're going to do is identify exactly where the metadata is, and as we're walking through it we'll start writing it into the free space of the HFS+ volume. And what this gives us is crash protection and the ability to recover in the event that conversion doesn't actually succeed. Now the metadata is identified. We'll then start to write it out to disk, and at this point, if we were doing a dry-run conversion, we'd end here.

If we're completing the process, we will write the new superblock on top of the old one, and now we have an APFS volume. The final step is to simply remove the metadata that was previously occupied by the HFS+ content. We'll do that, and at this point it's a completely converted APFS volume.

So now a quick word on Unicode. So if you know anything about Unicode, you know there's actually multiple variants, but for the purpose of discussion here, I will only be referring two really, which are NFC and NFD, and these roughly correspond to precomposed characters versus decomposed characters. And what this means is that you can actually store the same character visually, the same visual representation can actually be stored multiple ways depending on which form you're using. Now prior to iOS 11, APFS stored file names as nonnormalized UTF-8, and what this means is that if you have a file with one representation versus the other, they will actually be treated as different file names for the purposes of lookup and creation and comparison.

So let's give another example to make this easier to understand. Let's say we have the character n with tilde or [foreign language] in Spanish. You can represent this as the Unicode hex f1 character. Alternatively, you can also store it as a decomposed character in which the n is stored separately from the tilde. Now in this case, on the right the n and the tilde will be stored as two separate code points; however, they will be treated as having to be combined for the purposes of visually representing that character on your screen.

So the problem is that even if the file system is taking care to treat these as separate, that's not necessarily what the rest of the system expects and interacts with. This might be well-defined behavior for NFS; however, you want to make sure that the entire software stack from file system all the way up to foundation and above is able to treat everything properly. So in order to rectify this, we're introducing two new normalization schemes for Apple file system. We have a native scheme as well as a runtime normalization scheme.

Now the native normalization scheme is available in your high Sierra beta that you got earlier this week for case-sensitive volumes only. For case insensitive content, Pavel will get up on stage in a little bit and tell you all about that. The native normalization is coming soon for iOS 11 for erase restores only. This means if you plugged in the phone or other device to iTunes and did a restore to factory settings starting with iOS 11 will automatically pick this up.

However, we don't want anyone to have to erase install in order to get this behavior. So we're also introducing a new runtime normalization mechanism, and the runtime normalization will automatically convert between the NFC content versus the NFD for the purposes of file comparison, being able to do lookups. If it doesn't find one, it will automatically look up with the other to make sure that your app doesn't receive an ENOENT [phonetic] error back from the file system.

And a future update will automatically convert devices from whatever form they were in, if they were already migrated starting in 10.3 or 10.3.2 to the new form in a future update. So that's the update for iOS, and with that, I'm going to hand it off to Pavel, who is going to tell you all about the Mac.

Good morning everyone. My name is Pavel Sokolov, and I manage file system team in Apple. Today I'm going to talk to you about what's new in macOS with regards to APFS. Let's begin. So first, I would like to talk to you about what features are here just in case you missed the presentation from the previous year.

Important features that we have-- first of all, I would like to remind you that APFS is available to every Apple-supported platform. It works on iOS, tvOS, even on your watch, and macOS, of course. One of the awesome features is cloning. It allows you to do fast copies of the files. It employs our copy-on-write technology that also powers our Snapshots.

Snapshots allow you to take a [inaudible] Snapshot of your file system, entire file system, remember it's date, and later possibly return to it. We will also natively support encryption. We have space sharing. I'm going to talk to you about that a little bit later, and we introduce defragmentation.

How will get to APFS you might wonder? So let's talk a little bit about that. Actually, all you have to do is simply call a new macOS High Sierra and your system volume could automatically be converted by installer. In case you wonder what happens with other volumes, they are not going to be automatically converted, but you can still manually do so.

If you convert multiple volumes, the important point to remember after conversion each volume becomes APFS container. I'm going to talk about that a little bit later. And multiple containers do not share space. Space sharing is one of the most important and awesome features that's going to save you space. And we would like you to take advantage of that, and I can show you how. You might wonder what volumes are supported, what we can convert APFS. We can convert HFS+ volumes, CoreStorage, Fusion, and even FileVault encrypted. They all become APFS automatically.

If you'd like to do a manual conversion of your existing volume, all you have to do is go to disk utility [inaudible], right click on the volume, select convert to APFS, and here you go. As I mentioned, if you do so, the volume becomes a container and continues do not share space.

We would recommend if you have multiple volumes and would like to take advantage of that feature that you actually go and add volumes to the system container and move your data over. Why space sharing is so important and what's so good about it? Let's talk a little bit about that. Here's the typical volume layout that you might have as a developer. You probably have multiple volumes, right?

What's the deal with multiple volumes is as you start running out of space in each of them, you might have a little bit of free space here, a little bit of free space there, but you do not have enough in one single location so when you try to copy your files, nothing you can do.

In APFS world, the container is a partition on your volume, and volumes share the free space of that partition. So let's look at that picture. In a typical case, you would have a GPT header, [inaudible] service partition, and [inaudible] each volume is a partition. In APFS world, as I said, we have eight containers of this partition and volume [inaudible] side of it. Let's say you want to fill up the space. As you can see in HFS you run out. In APFS, you still can use your free space and have your data here.

Let's explore the cloning. Cloning is [inaudible]. It allows you to make fast copies of the files. In this particular case, let's take a local [inaudible] file. It has a couple of blocks, and traditionally if you copy a file you have to copy the data over, and that takes time. Not only it takes time, it takes space. In APFS, all we have to do is to record the references to that file, remember where the file is, and only your metadata is [inaudible].

It's also because it's fast and you don't waste space. You might wonder what happens if I have to modify the file. It's a single copy after all. What happens is that copy and write technology kicks in, and the new file data is located in the free space, and we just create yet additional reference. Quite simple, saves you space. Let's explore another interesting aspect of APFS is sparse files. Let me give you an example. Let's say you have a picture of a cat on your desktop, something like that, and you want to edit it.

Well let's say in APFS world and both in HFS world, you would have the same initial view. You have a file that's 1 megabyte in size, is [inaudible], and you want to edit that. What would happen if your application for example decides to seek past the end of the file and do additional write. Here's where the difference is. In HFS, the file system would fill the space that you seek after with zeros, and your file would be 3 megabytes of size, right?

So mega portion of the file would be zeroed out by the file system, and it would take your space. In APFS, what you would have is a sparse file. So middle portion of the file, still logically here, if you read from that, you would read zeros, but physically it takes only two blocks. Again, saves you space, makes your life faster, but you have to be aware of that feature.

So as you can see, there are two sizes. Logical size and the physical size. You need to be aware of that, but it actually becomes a little bit more complicated if you put Snapshots or clones into the picture. So let's explore that a little bit. How big is your file is [inaudible]. Let's say the same situation applies and you have a file that sparse, only takes two blocks and just going to take a Snapshot, record the location of those blocks, and now we want to modify the file.

As you see, copy and write technique kicks in, will allocate new space for the new block that you just wrote. So how big is your file? It's still three megabytes, right? Physically you still only have two blocks for it allocated, but [inaudible] delete the file, you only get the one block back, important thing that you need to be aware about. So let's talk a little bit about next important aspect in your file system is Unicodes and normalization support.

As you guys all know, that's almost invisible until we have problems. So let's explore how the world differs in APFS versus HFS. By default, APFS is case insensitive and normalization insensitive, unlike HFS, which is case insensitive and normalizing file system. What makes the difference? Actually pretty simple. In APFS, we keep the file names in exactly the same way you pass it to us. We compute case insensitive and normalization insensitive [inaudible] together with the file name and store it together. So when you want to do a lookup, we perform the same hashing approach. We do hash comparison, and that's how we can find the name fast and efficient.

Important caveat here, if you do a [inaudible] of directory structure, the names would come out in the order of the hash, not in the order of natural lexicographical comparison like you normally would expect. We're also supporting Unicode 9.0, unlike HFS that supports 3.2. We do not allow unassigned code points unlike HFS, and this is done for the sake of future compatibility.

So whenever new Unicode standard emerges, you can easily take advantage of that by replacing folds and tables. In APFS, we support canonical folds unlike in HFS, which supports simplified folds, and canonical folding is better because it allows you natural sorts and quarter in the same way that users expect.

Let's talk a little bit about Unicode 9.0 and why that's a big deal. You might wonder, who cares about Unicode version. Well, you should. Unicode 9.0 makes your applications truly global. For example, in Unicode 8, the support for Native American Cherokee was added, and now we have lower case symbols and capital symbols to properly support it. We support pretty much everything language that's written and understood by man, and if that's not global, I don't know what is.

We support latest emojis. You might wonder why? Of course, because it [inaudible]. And you might think, okay, HFS supported that too. So what's the big deal? The big deal is actually correctness. HFS supported that sacrificed some aspects of correctness. Now the folds in APFS is completely Unicode 9.0 compatible and for some of them, we just were certain rules succeed, let's say like capital I was taught in Turkish language or sharp S in German, they would be capitalized properly.

Let's talk a little bit about boot support. Of course APFS is bootable. You would like to boot your Mac, right? And will have embedded EFT driver. What embedded means in that particular case is the driver actually baked in a file system for [inaudible] and it's very easy, can be loaded by third parties that support virtual machines.

You don't really have to complete the full driver and the standard full format. All you have to do is follow a couple of pointers from the superblock and load the driver that's baked into the volume itself. That makes it future approved so in case volume format changes all you have to do is just load new driver from the new volume. We can boot from encrypted drives. We can boot from fusion drives. An encryption boot actually makes it more secure.

Let's talk a little bit about security and encryption. We support the same FileVault model that we used to. We have converted for existing FileVault volumes. We support the same recovery keys and not only will we support the same mechanisms, we will also make it compatible so whenever you convert, you preserve your existing recovery keys, you preserve your existing passwords.

You do not have to move your data all over the place, and conversion doesn't take that much time. Also important point, your Snapshots are encrypted as well, and that means if you took a Snapshot at some point and you didn't have a password, and then you enabled encryption, your old data is protected as well.

We support Fusion drives. The he same features that you used to have before are still there, is write back cache and read cache. To take advantage of faster SSD devices, we made small improvement, and now all your metadata always going to be on SSD, that makes this a little bit faster. Another important feature is defragmentation.

We support intelligent defragmenter. What makes it intelligent? It's smart enough to understand [inaudible] performance first to most were which files were most fragmented and it defragments them first. It defragments only hard drives, and defragmentation happens when your machine is idle and it doesn't interfere with your normal workflow.

Another interesting topic is Snapshots. Let's explore this a little bit. So what happens is imagine you have a file system, a couple of files here, and you want to remember the state of that file system for future reference. What happens here, will remember all the references to all the files, create a Snapshot of the metadata and record it.

If you later make [inaudible] to your file system, let's say you [inaudible] file, the same copy on the write technology kicks in, will allocate space in a new location, write your files. Your existing blocks are intact. Now let's say you delete a file. The blocks are not actually gone, only the reference to that block is gone.

So later if you want to revert to a Snapshot, it will restore the state the way it used to be. For example, your [inaudible] will be gone, and the file that you deleted would be restored. A Snapshot [inaudible] feature. They're fast to take. You need to be mindful that they occupy space. Because of that, creating Snapshots requires entitlements, and if you want to take advantage of a [inaudible], please talk to us. With that, I'm going to show you a demo. I'm going to invite Pavel Cisler and going to show you a demo of the time machine.

Thank you, Pavel. Hello everybody. My name is Pavel Cisler, and in case you wonder what's up with this guy's first name and the last guy's first name being the same, let me just say, we did some testing on clone file, and [laughter]-- anyway, so I manage the Finder and Time Machine teams.

Who is a Time Machine user here? Wow, cool. A lot of you guys. Anybody here know about the mobile Time Machine feature? Some of you. Not as many. Okay. So I'm going to show you mobile Time Machine a little bit. Let me switch over to my demo machine here.

So mobile Time Machine is this convenient part of Time Machine that's a feature that lets you back up your changes when you are on the go and you're away from your backup device, your time capsule, or your hard drive, right. Let's say you take your laptop on the road. Your time capsule is at home.

Your machine still continues to make hourly backups through mobile Time Machine, and that's a great feature, right? It doesn't necessarily help you with things like you lose your laptop or you lose your entire hard drive, but if you accidentally erase that important spreadsheet or whatnot, mobile Time Machine can save the day. So we have this feature since MacQuest and Lion, and let me show you real quick how it works. So here I am.

Here's the Time Machine restore UI, and my laptop here, I brought it from my office, I'm not connected to any time capsule, and yet I still have a backup history here, and I can browse and see my files. And here's that secret budget document that I was working on earlier that I accidentally deleted, and just like that I restored it, and I can continue working on it. So simple feature. You can really save the day when you need it.

Now you guys are developers, and you would like to know how it works under the hood, right? So on MacQuest and Lion when we first developed this feature, it was quite involved. It takes no less than two demons to do. One of them is a virtual file system overlay. If any of you had a chance to work on something like that, you know it's pretty tricky. So it's literally tens of thousands of lines of code. So for macOS 10 High Sierra, the Time Machine team was super busy deleting all this code.

So what we did, we reimplemented mobile Time Machine on top of APFS Snapshots, and that's actually what I just demoed to you here. So never mind this UI. Let me open up Terminal and show you some of it under the hood. When we talk about Snapshots, what do they look like? What do they feel like? Well, they feel very much like any other volume, and as such, you can inspect them using a variety of volume management tools.

My favorite is Mount, so let me go ahead and type Mount here. Without any arguments, if you do that, it lists all your disks, and you can see here that at the top there's my slash, there's a swap space here. A bunch of other common volumes, but then there's these funny volumes that are mounted at a mount point that continues to [inaudible] that Time Machine in the path, and there's also this strange time stamp.

Well, these are all APFS Snapshots that represent the individual hourly mobile Time Machine backups, right. Each of them is a Snapshot. Each of them mounted as a volume. So if I go into the Time Machine restore UI on the right here, that's what they are. So let me go back.

So these Snapshots aren't actually mounted all the time. They get mounted [inaudible], so they don't have to stick around. Mobile Time Machine only kicks them in when it needs them and [inaudible] them after. So let me show you how that, what happens when you do that. So I'm going to simulate what mobile Time Machine would do when it's done with Snapshots and unmount them all.

I'll use this command line too called tmutil with a verb called unmount local Snapshots. And it's going ahead and unmounting all the Snapshots. So now if I entered the restore UI, it actually has to remount them, right. So that was super quick, right. So it just mounted what, like some 20 volumes just like that. So that's fantastic.

Now, I'm going to go ahead and do a backup. Mobile Time Machine backs up once an hour. It wakes up and creates a Snapshot. We're not going to sit around for an hour here and wait for that. So I'll use a tool again. Tmutil Snapshot and it just backed up, right. So you saw it took a little less than a second to back up the entire machine. So that's fantastic.

So I'm going to go ahead and show you that it actually worked. I'm going to accidentally delete some data here. Oh no, what did I do, my important movie. And I'm going to go back into the restore UI and, ah, whew, there it is, and restore it. So again mobile Time Machine saves the day.

So how come this was so fast? Well, as the other Pavel explained to you, snapshotting is a copy and write technology, so you basically just grab references to your data, and the actual copying happens later if there's an actual change. It's deferred. It's lazy, right. So much more faster, much more efficient. So the other thing I'm going to point out here is that this movie that I just restored is actually pretty big. It's a movie of our last year's talk.

It's 1.1 gigs, and the old mobile Time Machine had certain limits, like it couldn't back up changes to files over 20 megabytes. Well, all these limits are gone now. Mobile Time Machine on top of APFS Snapshots will back up anything in a very speedy way and efficiently. So that's mobile Time Machine on top of APFS Snapshots. All right. S otherwise that, I'm going to switch back to the slides and Pavel will tell you more about some of the APIs we've used to make this happen. Thank you everyone.

Thank you, Pavel.

So you might wonder what about APIs, how to take advantage of all that. So good news is if you program [inaudible] like you probably should, it's all taken care of for you. The copyItem API automatically takes advantage of API files clone feature. How that happens behind the scenes we see if volume is compatible with cloning, and we take clones automatically as you copy the files. There's also replaceItem API, which takes care of [inaudible] save save, again automatically you don't have to do anything for that.

There are a couple of guidelines that I want to remind, good practices, and important points. In particular, [inaudible] data is duplicated. We announced that a year ago, and not only is it duplicated, it's also not going to work with API files. Instead of that, if you want to take advantage of [inaudible] file swap, [inaudible] save save, use new API which is called rename underscore np. Also, please do not use carbon APIs. Again that's [inaudible]. It works in compatibility mode with API files. It's going to be much slower. It's [inaudible] the file system. Instead, use foundation APIs.

If you want a little bit lower layer, you can use [inaudible] copyfile with copyfile function that supports copyfile clone flag. There are a couple of caveats for that [inaudible] preview. In particular, we're not going to automatically upgrade HDD-based file systems. We're still working on performance, and if you'd like to get bootable APFS, it's not enough just to run the converter. You have to go through the installer.

So APFS is coming to macOS. It's here. In fact, it's supported file system. It supports Fusion drives. It will support native encryption. Space sharing is an awesome feature. There are clone files and snapshots that are going to make your storage more efficient and faster. Please test your apps as you just noticed there are a couple of important differences between HFS and APFS. We would like your apps to be successful and awesome. Please give it a try, and report your bugs. There's more information available for you at that URL. Thank you.