Wow. It's very exciting to be here. I'm Gaurav. And today we are going to discuss machine learning. Machine learning is a very powerful technology. And together with the power of our devices, you can create some really amazing experiences. For example, you can do things like real-time image recognition and content creation. In the next 40 minutes, we are going to tell you all about those experiences and how Apple is making it easy for you to integrate machine learning in your app. In particular, we will be focusing on Core ML, our machine learning framework.

At Apple, we are using machine learning extensively. In our photos app, we use it for people recognition, scene recognition. In our keyboard app, we use it for the next word prediction, smart responses. We use it even in our watch to do smart responses and handwriting recognition. And I'm sure you guys also want to create similar experiences in your app. For example, some of you might like to do real-time image recognition. In this way, you can tell your users, provide them more contextual information about they are seeing, for example whether they are eating a hotdog or not.

We want you to enable all of these great facility in your app. And, hence, we are introducing machine learning frameworks just for you guys to enable all the awesome things. You can clap.

But before we clap, we should understand why do we even need machine learning in the first place? And when do you need machine learning? So let's just say I'm a florist. And all I want to show is all the images of roses in user's photo library. This seems like a simple task, so I can do some programming. Perhaps, I'll start with the color. I say if the dominant color in the picture is red maybe it's a rose. The problem is there are many rose that are white in nature, white in color, or yellow in color.

So I will actually go forward and start describing the shape. And soon I will realize that it's very difficult to write even such a simple program programmatically. Hence, [inaudible] to machine learning for our help. Rather than describing how a rose looks like programmatically, we will describe a rose empirically. We will learn that representation of rose empirically. Machine learning actually has two steps. One, training.

So what you will do in this case, off-line you will collect images of roses, lilies, sunflowers and label them. You will run it past through a learning algorithm. And you will get what we call as a model. This model is an empirical representation of how a rose look like.

Almost all of this stuff happens off-line. And there is a big [inaudible] on the Internet around training. Training is a complex subject. Now once you have the model, you want to use this model. So what you will do, you will take a picture of a rose, embed this model in your app, you run it past through your model, and you will get the label and the confidence. This step is known as inference. Training is very complex, but inference is also getting very, very challenging nowadays.

For example, if you want to describe a new, deep neural network programmatically in your code, you have to write thousands of lines of code just to describe the network. It can be [inaudible] tedious and perhaps you can face your biggest challenge in proving the correctness of such a program. Once you make sure that the program is correct, you have to make sure that you're getting the best performance possible.

Finally, you also have to make sure that there is energy efficiency involved here. These tasks can be very, very challenging and can bog you down completely. At Apple, we are using on-device machine learning for several years now. And we have solved these challenges. We have faced these challenges and solved them.

And we really don't want you to solve them. We will solve them for you. We want you to focus only on the experience. We will give you the implementation, the best implementation possible. And, hence, we are introducing machine learning frameworks for you in which case we take care of all the challenges, and you take care of all the experience in your app. By doing so, we are going to follow a layered approach. At the top is your app.

We are going to introduce new domain specific framework, in particular, Vision. The Vision framework will actually allow you to do all the tasks related to computer vision. The best part about computer vision of Vision framework is that you don't have to be a configuration expert for using Vision framework.

We are also enhancing our NLP APIs to include more languages and more functionality. We are introducing a brand-new framework called Core ML. This framework has an exhaustive support of machine learning algorithms, more deep learning as well as a standard. And finally, all of these frameworks are built on top of Accelerate and Metal Performance Shaders.

To give you a sneak peak, we have Vision framework. Vision is our one-stop shop to do all things related to computer vision and images. You can use it to do things like object tracking or more modern deep learning-based face detection. There is a lot more to Vision framework. We are going to discuss Vision in detail tomorrow afternoon.

NLP, this is our one-stop shop to do the text processing. You can use it to do things like language identification. And we are also releasing Named Entity Recognition APIs. These APIs can actually tell you in your text if it is a name of the location, people, and organization. We believe this will be very powerful for many app developers so please check this out tomorrow more in the morning session on NLP.

Core ML, until now, we were discussing domain specific framework. Core ML is our machine learning framework and is domain agnostic. It supports multiple kind of inputs such as images, text, dictionaries, raw numbers. So you can do things like music tagging. There's another interesting part of our Core ML that it has the ability to deal with mixed input/output type. So you can take an image and you can get a text out.

The best part about all these frameworks is that they can work together. So you take a text, you pass it through your NLP, you get the output, and you pass it through Core ML and do things like sentiment analysis. We are going to discuss it in detail in our in-depth Core ML session, how you do that exactly on Thursday.

All of these frameworks are being powered by Accelerate and MPS. These are our engine. Now you can use Accelerate and MPS whenever you need some kind of math functionality. So it doesn't have to be necessarily related to machine learning. You can also use these Accelerate and MPS when you are doing custom ML models. So if you are using a machine learning model, you might like to use Accelerate and MPS.

All these APIs run on user's device. And they're very, very highly performing. And we work very hard to make sure you get the best performance possible. Now because they learn on user's device, there are advantages. First, user privacy. Your users will be happy to know that you're not sending their personal messages, texts, and images to the server.

Your users don't have to pay extra data cost just to receive these -- just to send this data to the server to get a prediction. You also don't have to actually give huge amount of money to server companies just to set up the servers and get a prediction. Our devices are very, very powerful. You can do a lot here.

And finally, your app is always available. Let's just say your user goes to Yosemite in the wilderness where there is no network connectivity. They can still use your app. And this is very powerful. However, perhaps one of the best things you can do is real-time machine learning. Our devices are very powerful. They can run modern deep neural networks such as ResNet 15 [inaudible]. You can use our devices to do real-time image recognition. In these cases, you really don't have [inaudible] to actually get the application.

So for any latency sensitive things, you might like to do real-time image recognition. So just to recap here, we are releasing a series of machine learning frameworks. If your app is a vision-based app, please use Vision framework. If your app is a text-based app, please use NLP. Now if you think that NLP and the Vision framework are not providing the APIs you need, go down on Core ML.

Core ML provides exhaustive support of standard machine learning and deep learning algorithms. And let's just say you are [inaudible] your APIs machine learning algorithms then you should use Accelerate and MPS. We are going to discuss all these frameworks in detail in the next couple of days so you are [inaudible], but today let's just focus on Core ML. And to do that, I'll invite my friend and colleague, Michael.

Thanks Gaurav. Hi everyone. I'm so excited to talk about Core ML and its role in helping you make a great app. We're going to start out with a high level overview of Core MLs main goals. We'll then talk a little bit about models and how they're represented. And then we'll walk you through the typical development process using Core ML.

So the Core ML framework is available on macOS, iOS, watchOS, and tvOS. But Core ML is more than just a framework and a set of APIs. It's really a set of development tools all designed around making it as easy as possible to take a machine learned model and integrate it into your app. This will let you focus on the user experience you're trying to enable rather than the implementation details.

Core ML is simple to use, will give you the performance you need while being compatible with a wide variety of machine learning tools out there. Its simplicity comes from having a Unified Inference API across all model types. Xcode integration will let you interact with machine learned models using the same software development practices you're all already experts in.

It uses the inference engines Apple is already shipping to millions of customers in its apps and systems services now being made available to you via Core ML. And as mentioned, these are built on top of Metal and Accelerate to make the best use of hardware that your apps are being deployed on.

Core ML is also designed to work in this rapidly evolving machine learning ecosystem. It defines a new public format for describing models as well as a set of tools that will allow you to convert the output of popular training libraries into this format. So that's Core ML in a nutshell. It's all about making it super easy to get a learned model integrated into your app. So let's talk a little bit more about these models.

Since Core ML, a model is simply a function. Now the logic for this function happens to be learned from data but just like any other function, it takes in a set of inputs and produces a set of outputs. In this case, shown on the slide, we have a single input of an image and an output, perhaps telling you what type of flower is present in it.

Now many of the use cases in which you may want to apply a machine learning model have some key function at its core. A common type of function is performing some sort of classification. It's taking a set of inputs and assigning some categorical label. Take, for example, sentiment analysis. Here, you may take in an English sentence and output whether the sentiment was positive or negative, here represented by an emoji. Now, in order to encode functions of this type, Core ML supports a wide variety of models.

It has extensive support for neural networks both feedforward and recurrent neural networks with over 30 different layer types. It also supports tree ensembles, support vector machines, and generalized linear models. Now each of these model types are actually a large family of models. And we could spend the rest of this session and the rest of this conference just talking about any one of them, but don't let that intimidate you.

You do not need to be a machine learning expert to make use of these models. You continue to focus on the use case you're trying to enable and let Core ML handle those low-level details. Core ML represents a model in a single document. That is, sharing a model is just like sharing a file.

This document has the high level information you, as a developer, need to program against. That is the functional description, its inputs, types, and outputs as well as those details that Core ML needs to actually execute the function. For a neural network, this may be the structure of the neural network along with its trained parameter.

We encourage you to come to our Thursday session to learn more about this format and the set of models it can encode. But maybe now in the back of your head you're starting to think well, where do I get these models? Where do models come from? Well, a great place to start is to visit our machine learning landing page on developer.apple.com.

There you'll find a small collection of task specific, ready to use models already in the Core ML format. We'll be expanding the set overtime. And we encourage you to explore. Give it a try. This is a great way to get introduced to machine learning in general and Core ML and how it can be used in your app.

But we may not have all the models you need. And for your application, you may need to make some customization to an existing model or make a whole new model from scratch. And for that, we encourage you and your colleagues to tap into the machine learning community. There is a thriving community out there.

There are tons of libraries and models already existing for you to start playing with. In addition, there are wonderful online courses and new tutorials are popping up every week. You can do it. There's great resources to learn. And I'm confident that all of you can get started. It's a wonderful time.

But you'll see that most of these libraries are focused around training. That is, it's a very important step but they'll spend a lot of time making sure that you can train a great model. What we want you to do is let Core ML take you that last mile from taking that machine learned model and actually deploying it in your app.

So we're doing this by introducing the Core ML Tools Python package. This is a Python package that is centered completely around taking the output of these machine learning libraries and getting them into the Core ML format. We'll be expanding these tools over time, but they're also being made Open Source.

So what this means, if you don't find the converter you need there, we're super confident you can already write your own. All of the primitives are there ready for you to use. Now I encourage you, again, to visit us on Thursday to learn more about this Python package and how you can easily convert models. OK. Now we have this model. Let's talk about how we use it in our apps. So you're going to start with the machine learned model in this Core ML format.

You're going to simply add it to your Xcode project. And Xcode will automatically recognize this model as a machine learned model, and it will generate an interface for you. That will give you a programmatic interface to this model using data types and structures you're already familiar programming against. You'll program using this interface and compile it into your app.

But in addition, the model itself will also get compiled and bundled into your app. That is, we'll take this format that's been optimized for openness and compatibility and turn it into one that is optimized for run time on the device. To show you this in action, I'm going to invite my friend and colleague, Lizi, to the stage.

And I'm an engineer on the Core ML team. Today, let's take a look at how you can use Core ML to integrate machine learning into your applications. Now first some context. As much as I love machine learning, I also love to garden. And if you're anything like me, you may get really excited whenever you come across a new kind of flower and instantly want to identify it.

So we sought out to build an app that would take images of different types of flowers and classify them by type. But first, in order to do that, we needed a model. So we went ahead and we collected many different images of different types of flowers, and we labeled each one.

I then trained a neural network classifier using an Open Source machine learning tool and converted it into our Core ML format. So with this trained model and with this app shell of this flower classifier, let's take a look at how we can put the two together. So over in Xcode, you can see I'm currently running the flower classifier app and nearing the display on my device. And what I can do is I can go into the photos library and choose an image, but right now it doesn't currently predict anything for the output. It just displays this string.

So if I move over to the ViewController, what we see is it has this caption method that currently just returns that blank string. It doesn't actually do anything. Over here, I have this flower classifier in the ML model format. And all I need to do is take it and drag it over into the project.

And we can see, as soon as I add it, Xcode automatically recognizes this file format. And it populates the name of the model as well as what type it is underneath. We can see that the size is 41 megabytes and also other information like the author, license, or description associated with it.

In the bottom, we can see the inputs and outputs that this model takes which, first is flower image, a type of image that's ordered red, green, blue, the following width and height. We can also see that it outputs a flower type as a string. So if give it an image of a rose, I would expect flower type to be rose. There's also a dictionary full of probabilities for different types of flowers.

Now the first thing I want to do is I want to make sure that I add this to my app. And you'll see that afterwards this middle pane will show that the Swift generated source has arrived in the application. What that means is we can actually use this generated code to load the model and predict against it. To show you, let's go to the ViewController.

The first thing I want to do is define this flower model. And to instantiate it, all I need to do is use the name of a model class itself. We'll see that it's highlighted because it exists in the project. Now next in the caption method, let's define a prediction.

And we'll actually use the flower model and look inside. We'll see using auto complete that this prediction method is available that takes a CVPixelBuffer as input. We'll use that and pass the image directly to it. Now, instead of returning this preset string, we can use the prediction object and instead return the flower type.

Now if I save, let's build and run the app. And what's happening while I do this is the model itself is actually getting compiled and bundled with the application. So if we go back to the device, if I go to my photos library, I can choose a rose on the second row. And we can see it actually is able to display rose.

We give it another one, maybe the sunflower on the third. It's also able to get this one. Or even something a little bit more difficult like the passion flower on the bottom, it can still do. So to recap, we were just able to drag a trained ML model into Xcode and easily add it into our application with three lines of code.

And now we have a full neural network classifier running on the device. But I have some other models that we might want to try so let's go through this again. You may have noticed there's also this FlowerSqueeze. And the first thing I'll do is try to drag it in again.

And then I'll also add this one to the flower classifier, the hello flowers target so that it starts generating the Swift source in the background. And we'll see, again if we zoom in, that the name of this model is different. It's called FlowerSqueeze but in the bottom the inputs and outputs are exactly the same.

What this means is that if I go back into the ViewController, instead of flower -- actually, let's delete flower classifier. We don't want that model anymore. And in the ViewController, instead of instantiating this one, we can use FlowerSqueeze. And the really neat thing about this model is its size. So if we go back, this one is only 5.4 megabytes, which is a huge difference.

Back in the device, we can go to the photos library and choose another one like love in the mist and see that this app is also able to classify this as well. Try another passion flower or even a more difficult photo of a rose on the side and it's still able to predict correctly. So to recap, we just saw that we're able to run this neural network classifier on device using Core ML, all with only a couple lines of code.

Now let's go back and recap what we just saw. We saw that in Xcode, as soon as you drag an ML model that's been trained into your application, you get this view that's populated that gives you information such as the name of the model, the type that it is underneath, and also other information like the size or any other information the author put in.

You can also see in the bottom that you get information such as the input and output of the model which is very helpful if you're actually trying to use it. And once you add it to your app target, you get generated code that you can use to load and use the model to predict.

We also saw in our application how simple it was to actually use it. Again, even though the model was a neural network classifier, all we needed to do to instantiate it was call the name of the file. This means that the model type was completely abstracted so whether it's a support vector machine, a linear model, or a neural network, you all load them the exact same way. We also noticed that the prediction method took a CVPixelBuffer. It was strongly typed so you know you'll be able to check input errors at compile time rather than at runtime.

But now let's take a deeper look at the generated source that's being created by Xcode. Here, we see it defines three classes, first of which is the input that defines the flower image as a CVPixelBuffer. We then see the output with the same two types that were listed in the generated interface. And next, in the model class itself, we see the convenience initializer that we use to actually create the model and the predict method as well.

But also inside this generated source, you have access to the underlying model. So for more advanced use cases, you can use it as well to programmatically interact with it. We take a look inside the MLModel class, we can see that there's a model description which gives you access to anything you saw in the metadata in Xcode.

There is also an additional prediction method that takes protocol conforming input and returns protocol conforming output giving you more flexibility in how the input is provided. So we walked through the top half of this development flow and saw what happens when you drag a trained model into Xcode. It's able to generate Swift source that you can then use in your -- to use the model in your application. The model itself, as we saw, is also compiled and bundled in your app.

To elaborate on this step a bit more, what we do, we first take the model from its JSON form, which is optimized for portability and ease of conversion, and compile it so that it loads quickly on your device when you initialize it. We also make sure that the model is matched to the underlying inference engines that are ultimately evaluating it on your device giving you optimized prediction when you need it.

And the last thing we saw in the demo was we swapped out models. Mostly just to reduce the size but there are other reasons why you might want to play around with different ones. For example, accuracy or just to test out different types to see how they perform.

In summary, today we saw an overview of some of the machine learning frameworks that we have available. We also introduced Core ML, a new framework for on-device inference. And also we showed you the development flow in action. For more information, make sure you check us out at developer.apple.com with our session number 703.

And also, come visit some of the related sessions such as Vision and NLP and Core ML in depth on Thursday if you want to see some more use cases of how you can use Core ML or how to actually convert models into the Core ML format. Thank you and enjoy the rest of the conference.