Hello and.

Welcome to what's new in Core Spotlight. I'm John Hornkvist, Senior Manager for Core Spotlight and joining me today will be my colleague Lyn Fong. Today, we'll cover some new APIs, Core Spotlight on macOS, Drag-and-Drop, Quick Look previews, and then we'll give an update on ranking before finishing with a review of indexing and search.

So, let's jump right into macOS. Core Spotlight on macOS is the exact same API as in iOS so it's great for cross-platform apps. It's already used by Notes and Safari and because it's cross-platform CoreData Spotlight support has been reimplemented using Core Spotlight and is now available for both iOS and macOS.

Core Spotlight is great for databases and shoeboxes where your app has full control over the contents. It's not for items that the user monitors in the finder, for that the classic Spotlight API still exists and still works great. Finally, the Core Spotlight API is per user, so there's no sharing. For those of you who are new to Core Spotlight we'll review the basic concepts later in the session.

An important new feature in iOS 11 is Drag-and-Drop and of course, Drag-and-Drop is very important to the Mac as well. Drag-and-Drop with Core Spotlight is built on the concept of promises. You make the promise of indexing time and then you fulfill it when the user drops a Core Spotlight item later.

So, let's see this in action. We start the drag in Spotlight and then we hit the Home button to go to SpringBoard where we can enter notes and we can drop the image. This is an awesome accelerator letting users get to the content incredibly quickly. Now you may wonder what's actually going on behind the scenes. First your app indexes items and each item carries a promise.

Your app quits and sometime later the user runs a query in Spotlight. The user drags the item in Spotlight and Spotlight creates a promise that gets sent to the drag receiver. The app picks the type it wants and the request goes back to Spotlight. Spotlight will then call your application or your app extension with the type and item information.

So, then you provide the data and it gets passed to Spotlight. Of course, the receiving app it can take the content. This might look complicated, but there's actually not that much that you have to do. Your first task is deciding what drag types to support. Drag types use uniform type identifiers or UTI types which are the [inaudible] of the system.

These provide a uniform way of describing types in our hierarchy going from the most to the least specific. You can find great information about uniform type identifiers on developer.apple.com. You can define new types for your application, but for Drag-and-Drop we want you to declare types that are well-known so that other applications can receive the data that you have to offer.

As an example, a note taking app might have its own UTI type as a type of its contents, but it might advertise that it can provide RTF, HTML and plaintext when an item is dropped. So that's types. Now how do we let Spotlight know what can be provided? For that there are three new attributes in CSSearchableAttributeSet. You can promise a data representation, a file representation which will get copied or an in-place file representation if your data is saved [inaudible] copying.

As for the designing provider, you should specify the highest fidelity representation first. So how is it used? You create the CSSearchableItemAttributeSet as usual. In this case, we already have an image file so we'll provide that for the file type identifier. And we can also convert the image [inaudible] so declared it can provide plaintext as data. So that's our promise. Next, let's look at how to produce that.

When the user drops items, your extension will get called and in the rare case that your app happens to be running this may be your app's index delegate instead. Depending on what the receiving app requested one out of two methods will get called. If they ask for data or they ask for a type or you could provide data, the data method gets called and takes a searchable index, an item identifier, and a type identifier and you're expected to return the data object.

The file URL method takes the same arguments and additionally Booleans specifying whether an in-place file should be provided. The implementation of the data method might look as follows. We look at the model object for the item identifier, we check the type that's being requested, and then we produce data accordingly.

The implementation of the file URL method is very similar. Once again, we look at the model object for the item identifier, check the type that's being requested, but here we produce a file. And that's actually all you need to do for Drag-and-Drop on both iOS and macOS. So, to summarize the new API.

You declare and promise drag types [inaudible] indexing time. The Core Spotlight extension is critical, it will get launched to fulfill your promises. Make producing the data as fast as possible, the user will be waiting for the drag to finish. And of course, the new API is for both macOS and iOS. Next, Lyn is going to come onstage to tell you all about Quick Look previews for Core Spotlight.

Hi [inaudible], let's talk about Quick Look previews for your Core Spotlight items. On iOS Spotlight shows previews of your content when you 3-D touch to Peek and Pop. By default, Spotlight shows a text only preview based on the text in your Core Spotlight item, but now you can customize your preview by adopting a Quick Look preview extension.

So, this is an example of a default Core Spotlight preview. If you've tried Peek and Popping on a Core Spotlight item in iOS 10 this may look familiar. This is an example of what your preview could look like by adopting a Quick Look preview extension. This area here is where your preview will be displayed and it comes from a view controller in your extension. So, let's talk about how we go about implementing one of these extensions.

The Core Spotlight previews use a Quick Look preview SDK which is new to iOS this year and comes with a shiny new Xcode template. When you create your target from this template you will get an extension Info.plist and if we take a look at that plist under NSExtension attributes you'll see the QL supports searchable items attribute. For Core Spotlight previews you want that set to yes. This lets us know that your Quick Look preview extension supports Core Spotlight items. So, let's take a look at the API.

When you create your target, you will get a view controller with this method, preparePreviewOf SearchableItem. This is what will get called when a preview is required. In this method, you will get an identifier, this is the Core Spotlight identifier unique to your result. You'll get a query string which is the string the user searched for to get to your result. This is helpful if you want to highlight content relevant to the search in your preview. And finally, you will get a completion handler that you have to call once you're done.

So, debugging a Core Spotlight preview extension is different from debugging a typical extension, but don't worry it's still easy. Instead of picking a host app and launching your extension from that host app you'll be launching your extension from Spotlight directly. So, when Xcode asks you for an app on launch pick any app, you won't be using it, instead you'll go to Spotlight, look for your Core Spotlight item and then Peek and Pop. Xcode will automatically attach to your extension for you. So, let's take a look at this workflow with a demo.

So, before we begin the app that you will see in this demo is available as sample code so feel free to take a look after the session. So, let's start by taking a look at our main app. So here we have a simple app, it's just a list of pictures. If you select one of the pictures you get a more detailed view with a title, a rating and some description. So, let's see if we try to find a picture in Spotlight and Peek and Pop on it.

So, there's item. Oops, it popped right in. So, there's a bit of text, it's not the greatest. Let's see if we can do better. So back in Xcode here I'm going to add a new target from the Quick Look preview extension template and we'll call that pictures preview extension for iOS. We'll go ahead and activate that.

One thing I should mention is that the view controller that we saw in the app is in a framework so that it can share across multiple targets. If you have code or resources to share across targets we suggest you use the same approach. So, I'm going to go ahead and import that framework now.

And then we can jump into the meat of this file. PreparePreview OfSearchableItem, this is the same method you saw in the slides. And here we have the identifier and what we're going to do is use that identifier to find a picture with a matching identifier. And once we have that picture we can just go ahead and set up our view controller with that picture.

Again, this is the same view controller you saw in the main app. If you have a lightweight view controller in your main app you can certainly use the same approach. If your view controller is more memory or speed intensive you might want to consider making a lighter weight version for this purpose. So, then we're going to go ahead and present it. Here I have a little printout so I can see when Xcode has attached and finally, we call that completion handler. So, let's go and give this a spin.

So, as I mentioned, it doesn't matter what you pick here you're not going to use it, we're going to pick pictures because that's our app. So, there's pictures and we're going to head right back into Spotlight and we're going to try Peek and Popping again. And there's our preview.

And you can see from the printout in Xcode that we have successfully attached and now we can go ahead and debug. So, as you saw implementing a Quick Look preview extension can be very simple, especially if you already have a view controller to display your content. Maybe the view controller in your main app is lightweight already or maybe you've got a lightweight version for 3-D touch in your app. Either way, you can simply reuse that view controller here. Some final tips for your extension.

A loading spinner will show until you call that completion handler so call it as soon as you can. You can expect to see that spinner when Xcode is attaching to your extension for the first time like in the demo. But once it has already attached or if you're not running in Xcode you want to see your preview immediately.

This is an extension so memory is limited, be efficient. And finally, once you call that completion handler your job is done, don't do any more background work after the fact. So that's Core Spotlight previews on iOS. The Quick Look preview SDK also supports file-based previews and for that see the building great documents based apps in iOS 11 session.

So, as John mentioned, Core Spotlight is also coming to macOS and just like on iOS you can customize your preview. On macOS a preview is shown when you select a search result in the Spotlight window. Here you really do want to implement a Quick Look preview extension for your Core Spotlight item because Spotlight on macOS does not have a default preview.

So, this is what it's going to look like without a Quick Look preview extension. And this is what it could look like with one. This area here is where your preview will be displayed and just like on iOS it comes from a view controller in your extension. So, you can do just about anything a regular view can do.

Debugging a Core Spotlight Quick Look preview extension on macOS is again different from debugging a typical extension and also different from the iOS workflow. Because Spotlight's window vanishes when another app gets focused it can be difficult to work with breakpoints in Xcode. So instead we've provided the Quick Look simulator to launch your extension for you and it will stick around while you debug. So, let's take a look at how that works.

All right, so let's take a look at the Mac version of the app. So here we have the same app that we saw on iOS. It's got a list of pictures, if you select a picture you get a more detailed view with a title and a larger picture. Let's see what happens if we search for it in Spotlight.

We get a big blank space. Let's see if we can fix that. So, I'm going to create another target this time with the macOS Quick Look preview extension template and we'll call that pictures preview extension for macOS. Go ahead and activate it. And again, I'm going to import the framework.

That one and jump to prepare preview of searchable item which is the same method that you saw on iOS. So, we're going to do the same thing here and use that identifier to get a picture with the matching identifier and then we're going to go ahead and set up our view.

This is the same view that you saw in the app, I'm going to add it to our view hierarchy, do a little printout so we know when we've attached and finally, we call that completion handler. So, let's give that a go. When you run your Quick Look preview extension Xcode will offer the Quick Look simulator by default. So, go ahead and select that.

And there's a Quick Look simulator. On the left you will see your Core Spotlight items. If you have a lot of index search results you can use the search field above to narrow it down. When you select one of these results your preview will appear on the right and you can see that Xcode has successfully attached and you can go ahead and debug to your heart's content. So, let's see what it looks like in Spotlight now. So, as you can see Spotlight has successfully replaced the blank spot with your extension.

So, as you saw, the API for Core Spotlight previews on macOS is identical to the one on iOS. Some final tips here. One caveat to remember is that you should not make any views in your extension first responder. Your preview is not meant to be interactive, Spotlight is the interactive element here. And finally, the Quick Look preview extension on macOS only supports Core Spotlight items. For file-based previews the classic Quick Look generator API is still the solution. So that's it for Core Spotlight previews. Back to John.

Thanks Lyn, that was great. Ranking is very important for Spotlight. In iOS 11 and macOS [inaudible] we've added a new machine learning based ranker for Core Spotlight. This is personalized and adaptive, it runs on device using Core ML and we've worked very hard to keep your data private.

All the personalization and adaptation to the user happens on device. The ML model is trained in the cloud using features known locally from your devices. Features are private, they do not include actual results and they do not include actual queries. Data for training is only submitted if you're opted in to device analytics. This is a privacy friendly way of doing machine learning.

We've also added some new properties to let you help us rank your content. We've added a rankingHint which is a number from 1 to 100 with 100 being the best. And when the ranker can't tell the difference between items this can be used to elevate the more important content.

A new Boolean attribute was created. This lets us know whether the user created the item. UserOwned lets us know whether the user has purchased this item. And userCurated lets us know whether this is an item that the user selected, for example a bookmarked news article. Now keep in mind that this is just input to the ranker. If you try to fool the ranking system by setting the rankingHint at everything to 100 it won't really affect anything. This is only for ranking within your own items.

Match quality and usage information is still critical for ranking. So, to get the best ranking behavior use NSUserActivity so that we know what the user interacts with in your app. Provide a rich metadata for the ranker to work on, so set a great title, set an informative description, specify dates, and judiciously use keywords to make items easier to find, but don't misuse keywords because straight keyword matches means that your application's results will rank lower. Now let's get back to basics.

You need to get content into the index and the primary way to add content is directly via the CSSearchable index API. Secondarily, you can also index NSUserActivity which we recommend doing because it provides an important ranking signal. Sometimes you'll need to delete items reacting to what the user does or to external events. And of course, we have APIs for that as well.

Adding CSSearchableItems to the index is quite easy and you're in complete control of what you add. You first create a CSSearchable item attribute set that will hold the metadata for the item. You initialize the attribute set with the universal type identifier. Here we're using kUTTypeImage with a generic type frame rich content.

You also want to use something more specific and there are many built-in types of the system to inherit from. It's important in iOS and critical in macOS that you use the right type because it affects where and how your content is displayed. Then you set some attributes and the attribute set display name is a bare minimum.

You create a searchable item with a unique identifier, a domain identifier and the attribute set. And keep in mind that unique identifier is what you'll get back when Spotlight wants to launch into your application. And you index it and the completion handler will get called and the data has been safely committed to storage. Just like for Spotlight NSUserActivity can be used to index content and navigation points in your app. NSUserActivity reflects what the user has done in the application, whereas CSSearchable reflects what your app has to offer.

So, the difference is that the Core Spotlight API lets you index items that the user hasn't visited and is generally preferable as it gives you complete control over what is indexed. But on the other hand, because NSUserActivity is only for items that the user has visited it provides that important signal for ranking. To use NSUserActivity to inform ranking you need to relate them to the CSSearchableItems that you index.

To do this when you create your NSUserActivity you also create an attribute set. You set properties on the attribute set and then you set the related unique identifier or if you don't want the lifetime of your NSUserActivity tied to your Core Spotlight items use the [inaudible] related unique identifier instead. Then you mark your user activity as eligible for search and you set the attribute set on the user activity.

The many reasons to delete items from reacting to user's actions to getting rid of stale content. Using the Core Spotlight API you can delete specific items by their identifiers. For example, if the user deletes a document. You can delete groups of items by their domain identifier which can be useful if the user signs out of account or ends a subscription and you want to remove all content for it.

You can also use this to delete NSUserActivities that you've indexed if you set the domain identifier on them. Finally, you can delete all searchable items for your applications, which is useful if you have a version change and you need to restart indexing. This is also called on your behalf when the user deletes an application.

Now let's get into the details of indexing Core Spotlight. Let's start with how to register as an index delete and then talk about how to build a Core Spotlight extension which does the job of a delegate when your app isn't running. We'll talk about how client state works and how you can use it to make indexing robust and efficient. And we'll discuss some performance considerations.

Registering as an index delegate lets Spotlight reach out to your app when we need you to take action. It lets us request that you index all your content or index particular items [inaudible] accurate and up-to-date. It's also responsible for responding to index throttling and for providing Drag-and-Drop data. As usual, setting up the delegate is just a single line of code, but to be a delegate you need to implement the CSSearchableIndex delegate protocol.

This is a complete protocol. The first two methods are required, we look at those in a bit more detail in a moment. The second two are optional and let you know that indexing has been slowed down to favor foreground activity giving you the option of stopping any noncritical indexing and focusing on the most important items. And finally, there are the two methods for Drag-and-Drop that we discussed earlier.

When the index all method is called you add everything to the index. And when you get the call back for the last item you call the acknowledgment handler. If your app quits and is relaunched before the handler is called the Spotlight will call that again with the same callback. When reindex items with identifiers is called you look up the items the Spotlight is requesting and re-add or delete them as appropriate. And again, you call acknowledgment handler only when you've received the last callback for any outstanding work.

The Core Spotlight extension implements a CSSearchableIndex delegate protocol and allows callbacks to happen when your app is not running. This gets your content back into Spotlight as quickly as possible after the user is restored from backup or when disaster recovery is needed. The Core Spotlight extension will be called before your items expire allowing you to update them if necessary, even if the user happens to not be using your application.

Since the interface extension is the same as for the index delegate it's best to factor your code so that you can share the implementation. And ideally, the shared implementation will live in the framework. Remember also that the Core Spotlight extension is critical to support Drag-and-Drop. If you don't have an extension there will be nothing to call when the user drops an item for your application in another app. Well you can get the indexing right without using client state. We found that it makes the task far easier. Client state allows you to keep Spotlight and your own database in sync without redundant work.

Client state is an opaque Spotlight. What it is is your choice. It's often a simple integer denoting a sequence number which could be in a marker in a database journal, but we've seen more complex cases as well. Let's look at how this works. Your app sends batches to Core Spotlight, each batch is journaled with the client state.

When the batch has been committed to the journal your callback log is called letting you know that the batch has been received. So here the app is just indexing a new batch, but disaster strikes and the app crashes. Now what happened to the data that was in flight, did it make it to the index? With client state, you can find out.

When your app starts again you request the client state. Here you get state two back since this was last state that actually made it into the journal and you can restart indexing at just the right point. If the data had already made it to the Core Spotlight process when the crash happened you could continue it from state three instead, so you do the minimal amount of work.

To store client state, you need to create a named index. You can't use the default instance. The name lets us know which states to fetch which is required because some apps need more than one token. For example, because they're multiple databases. So, in your code first you create a named index instance, you'll begin an index batch, you add searchable items as usual, and then you compute the state that you want to save.

Finally, you end the batch with your opaque state and pay attention to any errors returned with the completion handler. So, at some later point when you're app or extension starts use client state to resume indexing. You fetch the client state and you compare it to the current state doing whatever work is needed to bring them in sync.

So, in your code you create an index instance for the same name, you fetch the client state, you examine the state after dealing with any errors, and then you just pick up where you left off if required. And so, you can replay exactly the operations that are needed to bring your index and Core Spotlight in sync.

Let's talk some more about indexing performance. Indexing is a background task and you don't want to slow down your app or the device with indexing work. So, minimize overhead, optimize any access to the file system or databases that you have to do in order to create items. And remember that each call to Core Spotlight carries overhead. So, has batches of items instead of single items whenever it's possible. That said, consider that memory is limited, so keep your batches reasonably small. Even batching just 10 items decreases the overhead by an order of magnitude.

And it's often more efficient to give multiple batches in flight in the pipeline manner than to use a single large batch. This allows indexing to happen in parallel with your work. Sine your app will be indexing while it's in use don't block the main thread. And finally, index on a background queue this will help with power and responsiveness.

To get a great presentation in Spotlight you want to set a good thumbnail. By default, Spotlight will use your app icon, which makes it hard to distinguish results at a glance. Just as important the thumbnail is a title, the title is not just created visually, it's also what users most frequently search on.

After the thumbnail and the title, you're going to set other fields that are suitable for your contact. Your description creates a much richer result and providing the content creation date can be very helpful as well. Where appropriate, rating and rating description can make a big difference. And if you know the location for something, setting the location name can be a very nice touch.

So, set a title, provide a great thumbnail, and set the right content type for your content. Then fill out the UI with additional metadata to write a great visual representation of your content. And remember that setting the right metadata isn't just about looks it also affects behavior, so let's take a look at that.

For starters, enabling quick actions like directions and calling makes the user interface richer and provides great value to the user. To support navigation, you set the latitude and longitude attributes and set supports navigation to true. Similarly, to support phone calls, you need to set the phone numbers attribute and set supports navigation to true sorry, and set phone call to true.

Setting attributes that the user can understand makes it easy to search. And setting attributes that aren't naturally associated with the item makes it hard to search and leads to poor ranking of your results. By setting contact identifiers, you can enable focus contact search which is a great way to get the user to your content. Supporting features like Drag-and-Drop and Quick Actions makes for a first-class experience. Another part of experiences is being able to engage with [inaudible] in your own application, so let's look at that.

Make sure launching from Spotlight is fast and that it takes the user directly to the found items. Use NSUserActivity to restore the state, your app delegate will be called a continuous activity and your [inaudible] activity type and the userInfo dictionary is necessary. If you're being launched because the user selected a CSSearchableItem in Spotlight the activity type will be the CSSearchableItem action type and the unique identifier can be retrieved from the userInfo dictionary by using CSSearchableItem activity identifier. Another reason for being launched is that the user wants to continue the search in your application. In that case, you get the CSQueryContinuation action type and you can retrieve the search query using the CSSearchQueryString from the userInfo dictionary.

Of course, the search system wouldn't be complete without a search API. Core Spotlight provides the ability to search the data that you have provided. It's the same search engine that's used in many places on the system. By using it you get consistent behavior with Spotlight and the system applications. It's great for all your content on the device and of course, it works in both iOS and macOS.

You can query for equality, for greater than or less than. So, if you want to find items with more than a certain number of pages the query is very short and simple. If you want to find all items with page count in a certain range you can use the InRange operator.

You can use Boolean operators for example, to select only items for the given width and height. You can use string matching with flags that makes Spotlight's matching more or less strict. From use in case insensitive word matching [inaudible] insensitive matching to other combinations of flags or to strict matching of full fields.

Or you can make your matching very laxed and treat each word as an individual query. This is what Spotlight itself does so if you want to be consistent it's a good place to start. And of course, if this doesn't suit your own application you can combine and mix-and-match as you like.

Core Spotlight supports a full range of operators for comparison and Boolean logic and of course, you can nest expressions using parentheses. The field wildcard will match any default search metadata and the double wildcard will match that, as well as text content. We have a number of options for string matching. Our index is heavily optimized for exact and prefix search and it's incredibly fast if you use these.

As a general rule, the longer the prefix the faster the query. Partial matching is very similar to prefix matching, depending on the string whichever has your results is faster. Phrase matching which means matching only on consecutive words is significantly more expensive. And finally, suffix and infix matching are all slower and using phrase matching combined with these multiplies the cost.

The query syntax also offers a set of flags that you can apply to make the matching less strict. The C is for case insensitive, D is for diacritics insensitive, which means that characters like an over [inaudible] can still be matched with an old character if the user is using the English locale. Word matching means that we match words internal to fields instead of just being anchored at the beginning of a field. And T for tokenized breaks out the individual words of the query. So, let's look at an example.

We're implementing a search function that takes a user query as input. First, we make sure to cancel any currently running query so that we don't have multiple concurrent queries as this will slow down the new query. Since it's user input we make sure to state the query string. We then use the double star syntax with cdwmt operators to make a very forgiving query. We create the query object specifying that we want to fetch the display names. We set the found items handler and then we set the completion handler.

The completion handler will get called only when there are no more results to receive from the query while the found items handler can be called with multiple batches of results. After that all they need to do is start the query, that's how easy it is to use the Core Spotlight search API.

So, in summary Core Spotlight is now available for the Mac, as well as iOS and it's great for all your managed content. Please adopt the new APIs or previews for Drag-and-Drop on both iOS and macOS. Provide a rich metadata for search, display and ranking. And use NSUserActivity to provide usage information. And as always, keep the index accurate and up-to-date by implementing indexing extension and taking advantage of client state.

For more information, visit developer.apple.com and you can also watch some sessions from earlier in the week if you're interested in Drag-and-Drop, introducing Drag-and-Drop, as well as mastering Drag-and-Drop would be recommended. And if you want to know about how CoreData and Core Spotlight interact the session on what's new in CoreData is great to watch. Thank you very much.