Morning. Good morning. My name's Eric Hanson. I'm the Technology Evangelist for the Photos platform. And, I'm joined by three of my colleagues from the Photos engineering team today to talk to you about integrating your application with Photos on macOS. We have two key things that we're going to talk about.

The first is a full update on the Photos Project Extension API that we introduced last year in macOS High Sierra. And then, we're going to get into interacting with your application via drag and drop. But first, let's talk about Photo Project Extensions. When iPhoto was first launched in 2002, Apple became one of the first companies to allow people to create beautiful books, and later cards and calendars. To date over 70 million photo books, cards, and calendars have been created using iPhoto and Photos.

But, in that same amount of time, 16 years, we've seen this blossoming market. There is a tremendous breadth now of choice for users to create all sorts of things with their photos, both physical and digital. And, it's the observance of this amazing ecosystem that led us to the introduction of the Photo Project Extension API last year, to build on that ecosystem.

We've seen some great extensions already. For example, Mimeo Photos. It lets you create these books, and cards, and calendars of exceptional quality. There's Whitewall, that lets you create gallery-quality framed prints from your photos. And, we have digital services like Wix.com, that make it very easy for you to create web photo albums to share with your friends and family. And, we're now seeing some new extensions on the horizon. Like this one, called Motif. It is a brand-new fully native experience, integrated directly into Photos, coming to you this summer.

All of this choice, this whole ecosystem is a tremendous opportunity to you, as a developer. You can take the very rich metadata and the context and the image curation that we pass to your extension, and you can build on that, using the full stack of native frameworks on macOS, to create something really extraordinary, and new, that delights users around the world.

And, it's for this reason that Apple is announcing we will be transitioning our entire print product business to this ecosystem in macOS Mojave. Project Extensions will be the way users will create books, cards, calendars, all sorts of things. Whatever you surprise us with with your extensions. And, by doing this, we're building a better Photos experience for everyone. So, let's talk about what's new in the Extensions API.

First, with the UI. We now take the UI of your extension, and integrate it directly into the Photos app. So, the sidebar that you're familiar with is always available. Right? A user can be creating a project in one of your extensions, and can grab content from the sidebar, and just drag and drop it right onto your project. Or, they could pop up to the photo search, search for a photo they want to use, copy it, and paste it into the project that they're working on.

We're also allowing you to integrate directly with the powerful editing tools of Photos. For example, you may want to allow a user to simply double-click on a photo to edit it. And, you can do that now, with a single line of API. You can invoke the actual full editor of Photos. We pushed the edit session to the top of the view stack, with that photo loaded. Let the user make any adjustments they want to make.

And, we also give them access to all of the other assets that are used in that project. When they're done with their editing session, they simply hit the Done button, and they're returned to the project that they're working on. You get a notification that the library changed. You can respond to that, and update your UI accordingly.

But, none of this would matter if it was hard to find the apps and the extensions that you create. And so, we're stepping up the game in the discoverability of your applications. The Photos app links to the Mac App Store in the Create menu. And now, with the brand-new Mac App Store, on macOS Mojave, we actually can link directly to a fully curated story that's targeted specifically at the extension experience. And, this is an evergreen story that will live on. It's something that we can update, and feature new experiences, and educate users on new ways to do things with their photos.

And, when a user downloads an app from this story, they're downloading an app that contains an extension. And, historically, that may have led you, as a developer, to put some time into, kind of, educating users in your standalone app on how to use the extension in Photos. But, we think we can do something better. Wouldn't it be great if the user could download an app and immediately launch into the project creation experience in Photos? We're making that possible.

We've added a custom URL scheme to Photos, where you can simply pass in your extension identifier, and optionally pass in a category. And, Photos will be brought to the back-- brought to the foreground, with the options of your extension displayed for the user. So, they're immediately taken into the creation experience as a first-launch experience. And, we think this is great.

And, finally, when they create a project, we want those projects to live where they belong, right inside the user's Photo library. And so, that's exactly what we do. We have this Project Gallery in Photos that lets a user see all the things that they've created. And we now let the extension create a custom preview for each project. So, that preview can represent what the project really is. Take this photo mug for example.

Not only can a user double-click on one of those projects to get back into your extension, to continue to work on it, they can create projects from other projects. They can simply select a project, go to the Create menu, create something new from it. And, we pass over everything we can to help your extension, kind of, start with some great context. But, in the case of the Apple projects, we can do a lot more. And so, we do.

We send you photo-by-photo, page-by-page exactly how that was laid out. So that your extension can create a complete conversion experience from the Apple projects. And, we highly encourage you to do this. So, that's just a very high-level, from a UI perspective, of some of what's new. But, we'd like to take you a little deeper. And, to do that, I'd like to invite to the stage, my colleague from Photos engineering, Tobias Conradi. Tobias.

Good morning. Thank you, Eric. Hello, my name is Tobias Conradi. I'm an engineer on the Photos team, and I will go into detail for some of the changes we did around Photos Project Extensions. First up, the Create menu. When we first introduced Photos Project Extensions in macOS High Sierra, we put all extensions as a flat list into the Create menu. But, as it turns out, in some cases, it's kind of hard to guess, just from the name of the extension what kind of projects you can create with that.

We want to improve the user experience, so we are introducing project categories. The new Create menu in macOS Mojave looks something like this. We have categories, submenus in the Create menu. And, the categories we're introducing this year are Books, Calendars, Cards, Wall Decor, Prints, Slideshows, and for any extension that doesn't fit into these categories, Other.

Now the user can go into the menu with a very specific intent. For example, I want to create wall decorations, and these [inaudible] all extensions that support projects in that category. How does the extension show up in the category? We have a new key in the extensions attribute dictionary in the extensions info.plist. It's called PHProjectCategory. And, the value for the key, is a list of the supported project categories. In this case, it's Wall Decor and Other.

The next thing the user sees from the extension, is the [inaudible] sheet. You provide us with the data, and we display the project type descriptions you provide. The problem with the API we introduced last year is that it's kind of hard to provide up-to-date price information or current offers.

That's why we're introducing a new, improved API to allow for dynamic updates, which would look like this. In addition to dynamic updates, we now also support display of a custom footer text at the bottom of the sheet, which you can use to display legal text if you have the requirement to do so.

On an API level, the new dynamic API looks like this. We have the new method on the projectExtensionsController protocol, and we ask for a data source, instead of a list of product type descriptions. And, pass in the category from the menu in which the selection-- the extension was selected. And also, an invalidator object.

Once you return the data source, we ask the data source for the project type descriptions, and for the optional footer text. Whenever your extension has the need to invalidate the information returned, you can use invalidator to invalidate the project type descriptions, or the footer text, and when necessary, photos will refetch the data from the data source, and display the up-to-date data in the UI.

The next topic is the projectInfo. The projectInfo is structured, additional information about the contents of the project. And, it's structured into sections, and sectionContents which reflect creation level. And, the elements have basic layout hints, and also weights and scores for the assets. And, assets can contain important regions in the asset's regions of interest. This is just a quick reminder what the projectInfo looks like. If you want to know more about the projectInfo, I highly recommend last year's session, What's New in Photos API.

The projectInfo is handed to your extension during begin project. And, whenever the user adds new assets to the project, the projectInfo will be outdated, because it's a static object. And, we want to solve the problem by introducing new API on the projectExtensionsContext to get updated project info, with the current state of the project.

Let's take this as an example. We handed you this projectInfo during project creation, and then the user added some more assets to the project. You can then call the updatedProjectInfo method, with this projectInfo, and we will update any existing sections, and append a new section for any added assets. If the user adds some more assets, you can do the same again. Pass in this project info, we will update the existing sections, and append a new section for any added assets.

Let's now go into a more detailed element of the projectInfo, the regions of interest. The regions of interest highlight important regions in the asset. For example, the faces of people. And, if the same person appears in multiple images, the region of interest identifier will be the same for the same person. Let's focus on Person B for a moment.

In addition to the region of interest identifier, we have weight on the regions of interest, which represents the importance of the region of interest in the project. But, if you want to decide which image to pick as a representative image for a person, or for a region of interest, it's kind of hard to do with this API.

That's why we're adding a new quality score, which represents the quality of the region of interest in the asset. On the left-hand side, the Person B is kind of out of focus and not really central to the image, while on the right-hand side, he's really in focus. That's why the quality score on the right-hand side is higher than on the left-hand side.

On an API level, that looks like this. We have this alt weight, which is the importance of the region of interest in the context of the project. And then, the quality in the asset. I would now like to show you a quick demo, how you can use regions of interest in your extension to improve the experience, and also listen to a library notification to get notified whenever the assets or the project changes. And, use a new updateProjectInfo API.

OK. I created a slideshow extension, and to show you how it looks like, I selected some assets in an album, and will create a new project with the extension. The extension has two views, one is the overview, which contains all assets of the project. And, the other is the playback modus, which plays the slideshow. As you might notice, the slideshow always zooms into the center of the image, which is kind of boring. And, I want to use regions of interest to always zoom into interesting regions in the photo. So, let's switch to Xcode to fix that.

Here in my asset model, I have this property, preferredZoomRect, and it always returns the same rect, which is kind of boring. So, let's replace it with the code. What I want to do is to get all regions of interest from the asset element, and then sort them by their weight and quality. And, from the sorted list of regions of interest, we want to get the last element, and return its rect. If we now rerun the extension-- We always zoom into the most important region of interest in the asset. I think that looks way better.

So, next, I want to add some more assets to the project. I grab an album in the sidebar and drag it over the extension. There's a plus next to the cursor, so it looks like the extension accepts the drag. I release the mouse and nothing happens. The assets were added to the project, but my extension doesn't list the change notification. And, doesn't know that assets were added to the project. So, let's switch back to Xcode and fix that. In my projectViewController, I have the begin and resumeProject method, which are a part of the projectExtensionController protocol. And, here I want to register for change observation.

And, we add the same code in both methods. First, we get the PhotoKit object we're interested in. We're interested in additions of assets to the project, so we are getting a fetch result for all assets on the project, and then we are registering asset change observer for the library. In finished project, we [inaudible] registering.

And, since Xcode complains that we don't implement the protocol, we add that. Change observer. And, down here-- We implement photoLibraryDidChange. PhotoLibraryDidChange is called whenever anything changes in the photo library, and we get a change instance as a method argument. We can ask the change engine instance for changeDetails of an object we're interested in. We're interested in the changes of the fetchAllAssets on the project, so we pass that fetchResult in. And, if change results are returned, we update our locally cached fetch result with the fetchResultsAfterChanges, and get the projectExtensionContext, and call updatedProjectInfo. And pass in the project info we have.

Once the updatedProjectInfo is returned to us, we call the same setup code, but you could do something more sophisticated in here. Let's rerun the project again. OK. I can now drag the album from the sidebar, release it over the extension, and the assets are added to the project, and the extension listens to the photo library change observation, gets notified of the change, and we update our project info.

So, as you just saw, with only a few steps, I was able to improve my extension by using the projectInfo, and the regions of interest in the projectInfo, and also registering asset change observer to get notified whenever anything changes in the photo lobby, and also update my project info.

There are two things I want to highlight with integration. By default, Photos handles copy and paste of assets or albums from Photos into the extension. But, if your extension also wants to implement the paste section, which you should probably do for text and things like that, we need your help to know when your extension should handle the paste section, and when Photos should try to handle the paste section.

So, please implement validateMenuItem, and if the menu action is a paste action, check if your extension can handle the current pasteboard contents, and if no, return false, and we will try to handle it as Photos. And, otherwise, if you can handle it, return true, and your photos will get the paste action.

Something similar applies to drag and drop. Photos handles any drags of PhotoKit objects to the extension by default, but your extension can potentially interfere with that if you register for the wrong types. So, please be careful what types your extension is registering for, and only register for extension internal drags, or other drags you really want to handle. Be especially careful if you use WKWebView, because it registers for a bunch of drag types by default. That was everything about photo project extensions. Let me now hand over to my colleague Sanaa to talk about integrating with Photos-- interacting with Photos as a third-party application. Thank you.

Thank you, Tobias. Hello, everyone. My name is Sanaa. I am a Photos engineer, and today with my colleague Joachim, we are going to talk about some best practices for receiving and providing images or videos between apps on macOS, using drag and drop. Drag and drop is one of the most intuitive and easiest way to move items from one place to another. But, sometimes, you might have encountered this situation instead. If this is something happening with your app, then you are sitting in the right session. So--

Let's take a step back to understand what's happening here. Drag and drop on macOS is using NSPasteboard, and when it's wrapped in with the pasteboard, reading and writing data happens on the main trade, for both the receiving and providing apps. In the past, since all local was stored on local disk, you could put the file URL's into the pasteboard. But, things have changed today. Images or videos might not be on disk if the user is using iCloud, so Photos, we need to download the full resolution item first, before putting the file URL into the pasteboard.

Also, Photos respect the privacy settings of drag and drop, so if the user choose to save the location information, then Photos will export a new file which does not contain this metadata. Both downloading and exporting file takes time, and you don't want to do that on the main trade, since it's going to block your app UI.

So, in order to fix this, we need an asynchronous API. In fact, we do have one, it's called file promises. File promise is a promise of-- that's a file of a certain type that does not exist yet on disk, will be written in a provided location. It also allows the sender to write files in the background. There are two ways to interact with file promises: receiving files by using NSFilePromiseReceiver and providing files by using NSFilePromiseProvider. Both of these modern API's have been introduced two years ago with macOS Sierra. So, let's start first with receiving file promises.

As a general rule, apps supporting drag and drop should always accept both file URL's and file promises. And, I'll explain why. There are multiple apps providing file promises. Photos is using file promises when dragging an image, a video, or entire album, and since macOS Mojave, you can-- we added the ability to drag the people or memories. So, we are not the only app using file promises. Mail is using file promises when dragging a message to Finder, to save the entire email, including attachments.

Safari is using file promises when dragging an image. And, Keynote is using file promises when dragging a selection of slides to create a new document containing those slides. So, if you want to receive files from those apps, or any app providing file promises, then your app needs to read file promises, and need to accept those files. So, let's see how we can do that by looking at some code.

First, during setup, a view must register what types it'll accept by calling registerForDraggedTypes. And, in order to accept file promises, you can use the class property readableDraggedTypes on NSFilePromiseReceiver. Then, when performing the drag operation, and when enumerating or draggingItems, you should add support to NSFilePromiseReceiver, and make sure to handle it first. Because it's more likely to contain the highest quality representation.

For each filePromiseReceiver, you call in the promise, and when the file is ready, then the reader block is called on the provided operationQueue, where you can handle the file. It's very important to provide a background operationQueue to not block the main trades while waiting for the file to be downloaded over-- to be downloaded, to be written by the source file. Because this process can take a long time, and you don't want-- especially if the file needs to be downloaded over a slow network.

So, for a better user experience, you need to display a loading activity, and when the file is ready, then you can replace the UI with the real content. Here you will see an example of Mail, showing a placeholder UI while waiting for the image to be downloaded via file promises.

So, that was receiving file promises. Now, let's see how we can provide file promises. And, you should consider implementing this in your app, if the data you want to send over drag and drop does not exist yet on disk. So, let's have a look on how we can do that by looking at the API.

First, you will need to create an instance of NSFilePromiseProvider. You should create one instance for each promised file. And, before writing the filePromiseProvider object to the pasteboard, it must contain a file type, and a delegate. These delegate is doing the heavy lifting of writing files to disk. There are only three methods for NSFilePromiseProvider delegates. The first one is called by the drag destination, and returns the file name, not the full path.

The second one return an operationQueue, and when-- an operationQueue where the file, it will be written. And, we highly recommend that you implement this optional method, and provide a background operationQueue, since otherwise, the main queue will be used otherwise. And, finally, writePromise to file is called, asking you to write file to disk. And, when you are done, remember to call the completion handler.

So, that was receiving file promises-- providing file promises, and we have covered the API to-- that you can use to adapt your app to receive and provide file promises. And now, I'd like to invite Joachim on stage to show you all of that in a sample app. Thank you. Joachim?

Thank you. Good morning. We're going to look at a simple beam generator app, which has a few issues with drag and drop. And we're going to dive into Xcode, and try to fix those issues together. So, here is my simple app. And, I can just grab a file I have here on desktop, and drop it onto its window. Just like that. On the upper right, I have a little button, where I can add a text box, and I can add some text to my image.

I can select it, move it around, and when I'm ready, I can send it over iMessage, for example. So, while this image is great, I really want to use an image I found on the web. So, let's open up Safari, and try to drag this image onto our app. Unfortunately, this doesn't work.

So, let's jump to Xcode, and see if we can fix that. Here I am in the main viewController of our app. And, we going to jump straight to view [inaudible]. And, have a look at what's going on. Right here, we're only registering for file URL's. So, let's fix that, and register for FilePromiseReceiver as well.

Next, when handling the drag operation, we need to take care of FilePromiseReceiver as well. So. Let's jump to the next method right here. So, here is the list of supported classes, and we only have NSURL. So, we're going to add at the first index, NSFilePromiseReceiver. In the enumeration method here, we're going to add a new case.

Where we're going to receive the promised files, and if we get a file URL, we're going to call the exact same method we will have called if we had just gotten the file URL over drag and drop. So, let's see if it works. So, here's the little app. Let's go back to Safari, take the image, and drop it onto the window.

So, that was pretty easy. I only had to modify two existing methods in order to accept file promises. Now. Let's add some text. Like this. And, wouldn't it be great if I'm not quite ready to send out my meme to somebody, if I could just drag it onto the desktop to save it as an image file? As you can see, this doesn't work. So, let's jump back to Xcode and fix this as well.

At the bottom of the class, we have this method which returns an object conforming to NSPasteboardWriting, and as you can see, we're just returning a simple NSImage here. So, we're going to replace it with a filePromiseProvider. So, we're creating a file provider, and we're going to provide a JPEG image. And, we're going to use this userInfo property to store the snapshotItem, which we're going to use later to write out the file to disk. As a next step, let's conform to NSFilePromiseProviderDelegate.

Just like this. Jump back down. And, implement the three delegate methods you heard about just before. The first one returns the file name. For simplicity, we're just returning a static file name here. The second one returns an operationQueue. And, we happen to have one right here. And, the third method is actually going to write the file to disk.

So, what's happening here, is that we're getting the snapshotObject out of the filePromiseProvider object, and we're going to use its JPEG representation to write the file to disk. So, let's have a look. This time, we're going to jump to Photos. Going to take this image, drag it into our app, add some text. And, drop it onto the Finder desktop. And, here we are.

So, if you just follow the simple steps I just showed you, you can add support for file promises to your app, and improve the user experience. And, with that, I'd like to hand it back to Eric. Thank you.

Great stuff. Great stuff. I hope you really enjoyed that. I just have a few words, just in summary. To, kind of, tie this all together. And, I'll start by saying, you know, photos really, really matter to all of us. Right? Millions and millions of people around the world take millions of photos every day. And, all of those photos find a home inside the Photos app.

But, they're nothing if they can't be shared. If they can't be shown. If they can't be preserved into beautiful keepsakes. And so, all of these people capturing all of these images, they rely on you, the developer, to create amazing apps that let them do creative things with those images. So, if you only remember two things from this session today, please remember this. Support file promises. You saw it's really easy to have some very immediate interaction with the Photos app.

Please, please do that. And, secondly, we strongly encourage you to explore all of the API that's available to you in the project extension experience within Photos. Going back to what I said at the beginning, this is a blossoming ecosystem. And, there's a tremendous opportunity to embrace it. And, collectively for us to create a better Photos experience for everybody.

So, with that, please join us in the lab this afternoon. We have tons of engineers on staff to help you with any of your PhotoKit questions, Photos extensions, be it iOS, macOS, we really look forward to talking to you there. And, enjoy the rest of your WWDC. Thank you.