Good morning. Welcome. I'm Michael, and this session is about what's new in Core ML. So introduced a year ago, Core ML is all about making it unbelievably simple for you to integrate machine learning models into your app. It's been wonderful to see the adoption over the past year. We hope it has all of you thinking about what great new experiences you can enable if your app had the ability to do things like understand the content of images, or perhaps, analyze some text.

What could you do if your app could reason about audio or music, or interpret your users' actions based on their motion activity, or even transform or generate new content for them? All of this, and much, much more, is easily within reach. And that's because this type of functionality can be encoded in a Core ML model.

Now if we take a peek inside one of these, we may find a neural network, tree ensemble, or some other model architecture. They may have millions of parameters, the values of which have been learned from large amounts of data. But for you, you could focus on a single file. You can focus on the functionality it provides and the experience it enables rather than those implementation details.

Adding a Core ML model to your app is simple as adding that file to your Xcode project. Xcode will give you a simple view, describe what it does in terms of the inputs it requires and the outputs it provides. Xcode will take this one step further and generate an interface for you, so that interacting with this model is just a few lines of code, one to load the model, one to make a prediction, and sometimes one to pull out the specific output you are interested in.

Note that in some cases you don't even have to write back code because Core ML integrates with some of our higher-level APIs and allows you to customize their behavior if you give them a Core ML model. So with Vision, this is done through the VNCoreML Request object. And in the new Natural Language framework, you can instantiate an MLModel from a CoreML model.

So that's Core ML in a nutshell. But we are here to talk about what's new. We took all the great feedback we received from you over the past year and focused on some key enhancements to CoreML 2. And we are going to talk about these in two sessions.

In the first session, the one you are all sitting in right now, we are going to talk about what's new from the perspective of your app. In the second session, which starts immediately after this at 10 a.m. after a short break, we are going to talk about tools and how you can update and convert models to take advantage of the new features in Core ML 2.

When it comes to your app, we are going to focus on three key areas. The first is how you can reduce the size and number of models using your app while still getting the same functionality. Then we'll look about how you can get more performance out of a single model. And then we'll conclude about how using Core ML will allow you to keep pace with the state-of-the-art and rapidly moving field of machine learning. So to kick it off, let's talk about model size. I'm going to hand it off to Francesco.

Thank you Michael. Hello. Every ways to reduce the size of your Core ML app is very important. My name is Francesco, and I am going to introduce quantization and flexible shapes, two new features in Core ML 2 that can help reduce your app size. So Core ML, [inaudible] why should you learn in models and device.

This gives your app four key advantages compared to running them in the cloud. First of all, user privacy is fully respected. There are many machine-learning models on device. We guarantee that the data never leaves the device of the user. Second, it can help you achieve real-time performance. And for silicon [phonetic] and devices are super- efficient for machine-learning workloads. Furthermore, you don't have to maintain and pay for Internet servers.

And Core ML inference is available anywhere at any time despite natural connectivity issues. All these great benefits come with the fact that you now need to store your machine-learning models on device. And if the machine-learning models are big, then you might be concerned about the size of your app.

For example, you are -- you have your [inaudible] map, and it's full of cool features. And your users are very happy about it. And now you want to take advantage of the new opportunities offered by machine learning on device, and you want to add new amazing capabilities to your app. So what you do, you train some Core ML models and you add them to your app.

What this means is that your app has become more awesome and your users are even happier. But some of them might notice that your app has increased in size a little bit. It's not uncommon to see that apps grow up, either to tens or hundreds of megabytes after adding machine-learning capabilities to them.

And as you keep adding more and more features to your app, your app size might simply become out of control. So that's the first thing that you can do about it. And if these machine-learning models are supporting other features to your app, you can keep them outside your initial bundle. And then as the user uses the other features, you can download them on demand, compile them on device.

So this -- in this case, this user is happy in the beginning because the installation size is unchanged. But since the user downloads and uses all the current Core ML functionality in your app, at the end of the day the size of the -- of your app is still large.

So wouldn't it be better if, instead, we could tackle this problem by reducing the size of the models itself? This would give us a smaller bundle in case we ship the models inside the app, faster and smaller downloads if instead of shipping the models in the app we download them.

And in any case, your app will enjoy a lower memory footprint. Using less memory is better for the performance of your app and great for the system in general. So let's see how we can decompose the size of a Core ML app into factors to better tackle this problem.

First, there is the number of models. This depends on how many machine-learning functionalities your app has. Then there is the number of weights. The number of weights depends on the architecture that you have chosen to solve your machine-learning problem. As Michael was mentioning, the number of weight -- the weights are the place in which the machine-learning model stores the information that it has been learning during training. So it is -- if it has been trained to do a complex task, it's not uncommon to see a model requiring tens of millions of weights.

Finally, there is the size of the weight. How are we storing these parameters that we are learning during training? Let's focus on this factor first. For neural networks, we have several options to represent and store the weights. And the first, really, is of Core ML in iOS 11. Neural networks were stored using floating-point 32-bit weights.

In iOS 11.2, we heard your feedback and we introduced half precision floating-point 16 weight. This gives your app half the storage required for the same accuracy. But this year we wanted to take several steps further, and we are introducing quantized weights. With quantized weights we are no longer restricted to use either Float 32 or Float 16 values. But neural networks can be encoded using 8 bits, 4 bits, any bits all the way down to 1 bit.

So let's now see what quantization here is. Here we are representing a subset of the weights of our neural networks. As we can see, these weights can take any value in a continuous range. This means that in theory, a single weight can take an infinite number of possible values. So in practice, in neural networks we store weights using floater 32 point -- Float 32 numbers. This means that this weight can take billions of values to better represent the -- their continuous nature. But it turns out the neural networks also work with lower precision weights.

Quantization is the process it takes to discontinue strings of values and constrains them to take a very small and discrete subset of possible values. For example, here quantization has turned this continuous spectrum of weights into only 256 possible values. So before quantization, the weights would take any possible values. After quantization, they only have 256 options.

Now since its weight can be taken from this small set, Core ML now needs only 8 bits of stored information of a weight. But nothing can stop us here. We can go further. And for example, we can constrain the network to take, instead of one of 56 different values, for example, just 8.

And since now we now have only 8 options, Core ML will need 3-bit values per weight to store your model. There are now some details about how we are going to choose these values to represent the weights. They can be uniformly distributed in this range, and in this case we have linear quantization instead in lookup table quantization, we can have these values scattered in this range in an arbitrary manner.

So let's see practically how quantization can help us reduce the size of our model. In this example, you are focusing on Resnet50, which is a common architecture used by many applications for many different tasks. It includes 25 million trained parameters and this means that you have to use 32-bit floats to represent it. Then the total model size is more than 100 megabytes.

If we quantize it to 8-bits, then the architecture hasn't changed; we still have 25 million parameters. But we are now using only 1 byte to store a single weight, and this means that the model size is reduced by a factor of 4x. It's only -- it now only takes 26 megabytes to store this model. And we can go further. We can use that quantized representation that only uses 4 bits per weight in this model and end up with a model that is even smaller.

And again, Core ML supports all the quantization modes all the way down to 8 bits. Now quantization is a powerful technique to take an existing architecture and of a smaller version of it. But how can you obtain quantized model? If you have any neural networking in Core ML format, you can use Core ML Tools to obtain a quantized representation of it. So Core ML 2 should quantize for you automatically. Or you can train quantized models.

You can either train quantized -- with a quantization constraint from scratch, or retrain existing models with quantization constraints. After you have obtained your quantized model with your training tools, you can then convert it to Core ML as usual. And nothing will change in the app in the way you use the model. Inside the model, the numbers are going to be stored in different precision, but the interface for using the model will not change at all.

However, we always have to consider that quantized models have lower-precision approximations of the original reference floating-point models. And this means that quantized models come with an accuracy versus size-of-the-model tradeoff. This tradeoff is model dependent and use case dependent. And it's also a very active area of research. So it's always recommended to check the accuracy of the quantized model and compare it with the referenced floating-point version for relevant this data and form metrics that are valid for your app and use case. Now let's see a demo of how we can use -- adopt quantized models to reduce the size of an app.

I would like to show you a style transfer app. In style transfer, a neural network has been trained to render user images using styles that have been learned by watching paintings or other images. So let me load my app. As we can see, I am shipping this app with four styles; City, Glass, Oils and Waves. And then I can pick images from the photo library of the users and then process them blending them in different styles right on device. So this is the original image, and I am going to render the City style, Glass, Oils, and Waves.

Let's see how this app has been built in Xcode. This app uses Core ML and Vision API to perform this stylization. And as we can see, we have four Core ML models here bundled in Xcode; City, Glass, Oils, and Waves, the same ones we are seeing in the app.

And we can see -- we can inspect this model. These are seen as quantized model, so each one of these models is 6.7 megabytes of this space on disk. We see that the models take an input image of a certain resolution and produce an image called Stylized of the same resolution.

Now we want to investigate how much storage space and memory space we can use by -- we can save by switching to quantized, models. So I have been playing with Core ML Tools and obtained quantizer presentation for these models. And for a tutorial about how to obtain these models, stay for Part 2 that is going to cover quantization with Core ML Tools in detail. So I want to focus first on the Glass style and see how the different quantization versions work for these styles.

So all I have to do is drag these new models inside the Xcode project, and rerun the app. And then we are going to see how these models behave. First we can see that the size has been greatly reduced. For example, the 8-bit version already from 6 or 7 megabytes went down to just 1.7.

In 4-bit, we can save even more, and now the model is less than 1 megabyte. In 3-bit, it is -- that's even smaller, at 49 kilobytes. And so on. Now let's go back to the app. Let's make this same image for reference and apply the Glass style in the original version.

Still looks as before. Now we can compare it with the 8-bit version. And you can see nothing has changed. This is because 8-bit quantization methods are very solid. We can also venture further and try the 4-bit version of this model. Wow. The results are still great. And now let's try the 3-bit version.

We see that there are -- we see the first color shift. So it probably is good if we go and check with the designers if this effect is still acceptable. And now, as we see the 2-bit version, this is not really what we were looking for. Maybe we will save it for a horror app, but I am not going to show this to the designer.

Let's go back to the 4-bit version and hide this one. This was just a reminder that quantized models are approximation of the original models. So it's always recommended to check them and compare with the original versions. Now for every model and quantization technique, there is always a point in which things start to mismatch.

Now we -- after some discussion with the designer, extensive evaluation of many images, we decided to ship the 4-bit version of this model, which is the smallest size for the best quality. So let's remove all the floating-point version of the models that were taking a lot of space in our app and replace them with the 4-bit version. And now let's run the app one last time.

OK. Let's pick the same image again and show all the styles. This was the City, Glass, Oils, and big Wave. So in this demo we saw how we started with four models and they were huge, in 32-bit -- or total app size was 27 megabytes. Then we evaluated the quality and switched to 4-bit models, and the total size of our app went down to just 3.4 megabytes. Now --

This doesn't cost us anything in terms of quality because all these versions -- these quantized versions look the same, and the quality is still amazing. We showed how quantization can help us reduce the size of an app by reducing the size of the weight at the very microscopic level. Now let's see how we can reduce the number of models that your app needs.

In the most straightforward case, if your app has three machine-learning functionalities then you need three different machine-learning models. But in some cases, it is possible to have the same model to support two different functions. For example, you can train a multi-task model. And multi-task models has been trained to perform multiple things at once. There is an example about style transferring, the Turi Create session about multi-task models. Or in some cases, you can use a yet new feature in Core ML called Flexible Shapes and Sizes.

Let's go back to our Style Transfer demo. In Xcode we saw that the size of the input image and the output image was encoded in part of the definition of the model. But what if we want to run the same style on different image resolution? What if we want to run the same network on different image sizes? For example, the user might want to see a high-definition style transfer.

So they use -- they give us a high-definition image. Now if I were Core ML model all it takes is a lower resolution as an input, all we can do as developers is size -- or resize the image down, process it, and then scale it back up. This is not really going to amaze the user. Even in the past, we could reship this model with Corel ML Tools and make it accept any resolution, in particular, a higher-resolution image.

So even in the past we could do this feature and feed directly the high-resolution image to a Corel ML model, producing a high-definition result. This is because we wanted to introduce a finer detail in the stylization and the way finer strokes that are amazing when you zoom in, because they have -- they add a lot of work into the final image. So in the past we could do it, but we could do it by duplicating the model and creating two different versions: one for the standard definition and one for the high definitions.

And this, of course, means that our app is twice as much the size -- besides the fact that the network has been trained to support any resolution. Not anymore. We are introducing flexible shapes. And with flexible shapes, you have -- if you have -- you can have the single model to process more resolutions and many more resolutions. So now in Xcode --

-- in Xcode you are going to see that this -- the input is still an image, but the size of the full resolution, the model also accepts flexible resolutions. In this simple example, SD and HD. This means that now you have to ship a single model. You don't have to have any redundant code. And if you need to switch between standard definition and high definition, you can do it much faster because we don't need to reload the model from scratch; we just need to resize it.

You have two options to specify the flexibility of the model. You can define a range for its dimension, so you can define a minimal width and height and the maximum width and height. And then at inference pick any value in between. But there is also another way. You can enumerate all the shapes that you are going to use. For example, all different aspect ratios, all different resolutions, and this is better for performance. Core ML knows more about your use case earlier, so it can -- it has the opportunities of performing more optimizations. And it also gives your app a smaller tested surface.

Now which models are flexible? Which models can be trained to support multiple resolutions? Fully convolutional neural networks, commonly used for MS processing tasks such as style transfer, image enhancement, super resolution, and so on -- and some of the architecture. Core ML Tools can check if a model has this capability for you.

So we still have the number of models Core ML uses in flexible sizes, and the size of the weights can be reduced by quantization. But what about the number of weights? Core ML, given the fact that it supports many, many different architecture at any framework, has always helped you choose the right -- the model of the right size for your machine-learning problem. So Core ML can help you tackle the size of your app using this -- all these three factors. In any case, the inference is going to be super performant. And to introduce new features in performance and customization, let's welcome Bill March. Thank you.

Thank you. One of the fundamental design principles of Core ML from the very beginning has been that it should give your app the best possible performance. And in keeping with that goal, I'd like to highlight a new feature of Core ML to help ensure that your app will shine on any Apple device.

Let's take a look at the style transfer example that Francesco showed us. From the perspective of your app, it takes an image of an input and simply returns the stylized image. And there are two key components that go into making this happen: first, the MLModel file, which stores the particular parameters needed to apply this style; and second, the inference engine, which takes in the MLModel and the image and performs the calculations necessary to produce the result. So let's peek under the hood of this inference engine and see how we leverage Apple's technology to perform this style transfer efficiently.

This model is an example of a neural network, which consists of a series of mathematical operations called layers. Each layer applies some transformation to the image, finally resulting in the stylized output. The model stores weights for each layer which determine the particular transformation and the style that we are going to apply.

The Core ML neural network inference engine has highly optimized implementations for each of these layers. On the GPU, we use MTL shaders. On the CPU we can use Accelerate, the proficient calculation. And we can dispatch different parts of the computation to different pieces of hardware dynamically depending on the model, the device state, and other factors.

We can also find opportunities to fuse layers in the network, resulting in fewer overall computations being needed. We are able to optimize here because we know what's going on. We know the details of the model; they are contained in the MLModel file that you provided to us. And we know the details of the inference engine and the device because we designed them. We can take care of all of these optimizations for you, and you can focus on delivering the best user experience in your app.

But what about your workload? What about, in particular, if you need to make multiple predictions? If Core ML doesn't know about it, then Core ML can't optimize for it. So in the past, if you had a workload like this, you needed to do something like this: a simple for loop wrapped around a call to the existing Core ML prediction API. So you'd loop over some array of inputs and produce an array of outputs.

Let's take a closer look at what happens under the hood when this -- when we are doing this. For each image, we will need to do some kind of preprocessing work. If nothing else, we need to send the data down to the GPU. Once we have done that, we can do the calculation and produce the output image. But then there is a postprocessing step in which we need to retrieve the data from the GPU and return it to your app. The key to improving this picture is to eliminate the bubbles in the GPU pipeline.

This results in greater performance for two major reasons. First, since there is no time when the GPU is idle the overall compute time is reduced. And second, because the GPU is kept working continuously, it's able to operate in a higher performance state and reduce the time necessary to compute each particular output.

But so much of the appeal in Core ML is that you don't have to worry about any details like this at all. In fact, for your app all you are really concerned with for your users is going from a long time to get results to a short time.

So this year we are introducing a new batch API that will allow you to do exactly this. Where before you needed to loop over your inputs and call separate predictions, the new API is very simple. One-line predictions, it consumes an input -- an array of inputs and produces an array of outputs. Core ML will take care of the rest.

So let's see it in action. So in keeping with our style transfer example, let's look at the case where we wanted to apply a style to our entire photo library. So here I have a simple app that's going to do just that. I am going to apply a style to 200 images.

On the left, as in your left, there is an implementation using last year's API in a for loop. And on the right we have the new batch API. So let's get started. We are off. And we can see the new is already done. We'll wait a moment for last year's technology, and there we go.

In this example we see a noticeable improvement with the new batch API. And in general, the improvement you'll see in your app depends on the model and the device and the workload. But if you have a large number of predictions to call, use the new API and give Core ML every opportunity to accelerate your computation.

Of course, the most high-performance app in the world isn't terribly exciting if it's not delivering an experience that you want for your users. We want to ensure that no matter what that experience is, or what it could be in the future, Core ML will be just as performant and simple to use as ever. But the field of machine learning is growing rapidly. How will we keep up? And just how rapidly? Well let me tell you a little bit of a personal story about that.

Let's take a look at a deceptively simple question that we can answer with machine learning. Given an image, what I want to know: Are there any horses in it? So I think I heard a chuckle or two. Maybe this seems like kind of a silly challenge problem. Small children love this, by the way.

But -- so way, way back in the past, when I was first starting graduate school and I was thinking about this problem and first learning about machine learning, my insights on the top came down to something like this: I don't know -- seems hard. I don't really have any good idea for you.

So a few years pass. I get older, hopefully a little bit wiser. But certainly the field is moving very, very quickly, because there started to be a lot of exciting new results using deep neural networks. And so then my view on this problem changed. And suddenly, wow, this cutting-edge research can really answer these kind of questions, and computers can catch up with small children and horse recognition technology. What an exciting development.

So a few more years pass. Now I work at Apple, and my perspective on this problem has changed again. Now, just grab Create ML. The UI is lovely. You'll have a horse classifier in just a few minutes. So, you know, if you are a machine learning expert, maybe you are looking at this and you are thinking, "Oh, this guy doesn't know what he is talking about. You know, in 2007 I knew how to solve that problem. In 2012 I'd solved it a hundred times." Not my point.

If you are someone who cares about long-lasting, high-quality software, this should make you nervous, because in 11 years we have seen the entire picture of this problem turn over. So let's take a look at a few more features in Core ML that can help set your mind at ease.

To do that, let's open the hood once again and peek at one of these new horse finder models, which is, once again, a neural network. As we have illustrated before, the neural network consists of a series of highly optimized layers. It is a series of layers, and we have highly optimized implementations for each of them in our inference engine. Our list of supported operations is large and always growing, trying to keep up with new developments in the field.

But what if there is a layer that just isn't supported in Core ML? In the past, you either needed to wait or you needed a different model. But what if this layer is the key horse-finding layer? This is the breakthrough that your horse app was waiting for. Can you afford to wait? Given the speed of machine learning, this could be a serious obstacle.

So we introduced custom layers for neural network models. Now if a neural network layer is missing, you can provide an implementation with -- will mesh seamlessly with the rest of the Core ML model. Inside the model, the custom layer stores the name of an implementing class -- the AAPLCustomHorseLayer in this case.

The implementation class fills the role of the missing implementation in the inference engine. Just like the layer is built into Core ML, the implementation provided here should be general and applicable to any instance of the new layer. It simply needs to be included in your app at runtime. Then the parameters for this particular layer are encapsulated in the ML model with the rest of the information about the model.

Implementing a custom layer is simple. We expose an MLCustomLayer protocol. You simply provide methods to initialize the layer based on the data stored in the ML model. You'll need to provide a method that tells us how much space to allocate for the outputs of the layer, and then a method that does the computation.

Plus, you can add this flexibility without sacrificing the performance of your model as a whole. The protocol includes an optional method, which allows you to provide us with a MTL shader implementation of your model -- of the layer, excuse me. If you give us this, then it can be encoded in the same command buffer as the rest of the Core ML computation. So there is no extra overhead from additional encodings or multiple trips to and from the GPU. If you don't provide this, then we'll simply evaluate the layer on the CPU with no other work on your part.

So no matter how quickly advancements in neural network models may happen, you have a way to keep up with Core ML. But there are limitations. Custom layers only work for neural network models, and they only take inputs and outputs which are ML MultiArrays. This is a natural way to interact with neural networks.

But the machine learning field is hardly restricted to only advancing in this area. In fact, when I was first learning about image recognition, almost no one was talking about neural networks as a solution to that problem. And you can see today it's the absolute state of the art.

And it's not hard to imagine machine-learning-enabled app experiences where custom layers simply wouldn't fit. For instance, a machine-learning app might use a neural network to embed an image in some similarity space, then look up similar images using a nearest-neighbor method or locality-sensitive hashing -- or even some other approach. A model might combine audio and motion data to provide a bit of needed encouragement to someone who doesn't always close his rings.

Or even a completely new model type we haven't even imagined yet that enables novel experiences for your users. In all these cases, it would be great if we could have the simplicity and portability of Core ML without having to sacrifice the flexibility to keep up with the field.

So we are introducing custom models. A Core ML custom model allows you to encapsulate the implementation of a part of a computation that's missing inside Core ML. Just like for custom layers, the model stores the name of an implementation class. The class fills the role of the general inference engine for this type of model. Then the parameters are stored in the ML Model just like before. This allows the model to be updated as an asset in your app without having to touch code.

And implementing a custom model is simple as well. We expose a protocol, MLCustomModel. You provide methods to initialize based on the data stored in the ML Model. And you provide a method to compute the prediction on an input. There is an optional method to provide a batch implementation if there are opportunities in this particular model type to have optimizations there. And if not, we'll call the single prediction in a for loop.

And using a customized model in your app is largely the same workflow as any other Core ML model. In Xcode, a model with customized components will have a dependency section listing the names of the implementations needed along with a short description. Just include these in your app, and you are ready to go. The prediction API is unchanged, whether for single predictions or batch.

So custom layers and custom models allow you to use the power and simplicity of Core ML without sacrificing the flexibility needed to keep up with the fast-paced area of machine learning. For new neural network layers, custom layers allow you to make use of the many optimizations already present in the neural network inference engine in Core ML. Custom models are more flexible for types and functionality, but they do require more implementation work on your part.

Both forms of customization allow you to encapsulate model parameters in an ML model, making the model portable and your code simpler. And we've only been able to touch on a few of the great new features in Core ML 2. Please download the beta, try them out for yourself.

Core ML has many great new features to reduce your app size, improve performance, and ensure flexibility and compatibility with the latest developments in machine learning. We showed you how quantization can reduce model size, how the new batch API can enable more efficient processing, and how custom layers and custom models can help you bring cutting-edge machine learning to your app.

Combined with our great new tool for training models in Create ML, there are more ways than ever to add ML-enabled features to your app and support great new experiences for your users. After just a short break, we'll be back right here to take a deeper look at some of these features. In particular, we'll show you how to use our Core ML Tools software to start reducing model sizes and customizing your user experiences with Core ML today. Thank you.