Hello. Welcome everyone. My name is Gaurav. And today we are going to talk about machine learning. Last year, we launched Core ML. And the response from developers, from you guys have been tremendous. We are just amazed by the apps you have made, the [inaudible] phenomenal. So let me first begin by saying thank you. Thank you for embracing Core ML. And we are -- we love seeing so many of you using it and giving intelligent features to our users. We are in this together. Thank you.

It's an applause for the devlopers. Okay. So if you recall, Core ML gives you an easy way to integrate an ML model in the app. The idea is very simple. You get an ML model, you drag and drop in Xcode, and with just three lines of code, you can run state-of-the-art ML model with millions of parameters and billions of calculations in real time. It's just amazing.

And you give -- your users get real time machine learning as well as privacy-friendly machine learning. All you have to do is to drag and drop an ML model in Xcode and Core ML takes care of the rest. I think the big question remains is where do I get these models from? So last year, we provided you two options. The first one was you could download some of these models, popular models from our website but, more importantly, we also released Core ML tools.

Core ML tools allow you to tap the work which is done by amazing ML community. So the idea is, again, simple. You choose your favorite learning library, train your model in that training library, convert it into Core ML from that and then just integrate it into your app.

When we released Core ML, we released with only five or six training libraries support for five or six training library but within a year, we have support for all the famous training libraries out there. We are enhancing our tools to even allow you more customization. And we are going to talk about more about Core ML tools in tomorrow's session.

Another thing we did towards the end of the year, we released Turi Create, our open source machine learning library. We are going to talk about Turi Create in tomorrow's session. But this year, we want to give you something even more. We want to continue our journey. We want to give you something native, something Swifty, something that harnesses the power of our Xcode, something that puts the focus on you, our developers, something that just demystify machine learning for you. Hence, we are introducing Create ML --

Our machine learning framework in Swift. So Create ML completes the left-hand side of the equation. The idea is you make a model in Create ML and you run it in Core ML. You do complete end-to-end machine learning in Swift, our favorite language. So you are not dealing with language oddities where you are training in one language and then running in, for instance, another language.

Create ML is simple and very powerful. It is tailored to your app. It leverages core Apple technologies, and you do everything on your Mac. So for this year we are going to focus on three very important use cases. The first one is images, second is text, and the third one is tabular data. These are the top use cases that we believe will benefit you.

So you can do things like custom image classifier. Idea is that you make your own image classifier that can recognize product from your product catalog. You can do things like text classifier so you can make your own sentiment analysis, topic analysis, domain analysis. And you can also do classical regression and classification on tabular data. For example, let's just say you want to predict the wine quality using its chemical composition. The possibilities are endless, and we are going to discuss them in detail in the next 30 minutes.

However, before we do, let's take a look at common workflow. First, let's just say you are trying to enable an experience in your app, make sure that machine learning is the right thing to do there. So don't just blindly apply machine learning. Make sure machine learning is the right thing to do there and define a machine learning problem.

Second, collect data. Make sure this data reflects the real usage of your app. So, for example, if you're making a custom image classifier that is going to be used by users on their iPhone, so collect pictures from your iPhone. Do not collect -- collect less screenshots but have more iPhone pictures.

Then you train your model. Finally, an important step here is to evaluate this model. The model evaluation is done on a separate handout set. If you're happy, you write out the ML model. But let's just say the results are not good. You should either retrain your model with different parameters or you collect more data.

Create ML actually helps you across all four stages of this workflow. We have powerful in-built data [inaudible] utilities, data source and data table that we will talk in the remainder of the presentation. You can actually train your model using only one line of code. And the training is done hardware optimized.

There are built-in evaluation metrics, so you don't have to write your own precision and recall and confusion metrics calculation. Use them. And finally, when you're happy, just write out the model. Now we will take a deeper look in all three use cases: images, text, and tabular data. So let's start with images. And to do that, I will invite Lizi Ottens, Senior Engineer in Machine Learning team. Thank you.

Thank you, Gaurav. Since enabling image-based experiences are some of the most powerful and interactive ones that you can add to your apps, today we'll take a look at how to train custom image classification models. Image classification is the problem of identifying what label out of a set of categories you'd like to apply to an image. Depending on the type of training data, you can target domain specific use cases to enable in your apps. The first step is to collect training data.

In doing so, we'll take a look at a fruit classifier and see how you would do so. First, you'd want to gather many varied types of images that reflect the true data that you'll end up seeing and then label them. First, you can do this as a dictionary with the string label corresponding to arrays of images.

Or what we've noticed is many popular data sets are organized in hierarchical directory structures such that the label is the name of the folder that contains all images within it. There are also other data sources such as single folders that contain labeled filenames. And in the Create ML API, we've provided conveniences to extract these structures.

Now training is the more complex part of the equation. So once you have your data, this is what you will get next. And what you can do is you can start training a very complex model from scratch on your input images. And for this you need lots and lots of label data. You need big compute and you need a lot of patience. But another well-established technique in the industry is transfer learning.

And since Apple has lots of experience in training complex machine learning models, we already have one in the operating system that you can take advantage of. So what we do is we apply transfer learning on top of this model that already exists in the OS, and we augment it, retraining the last few layers to your specific data so you no longer need millions of images. You can train a good classifier using the amount of data that you have. This results in faster training times. And for developers that we've worked with, we've seen them go from hours of training down to minutes for thousands of images or for small data sets, even seconds.

This also results in much smaller models going from hundreds of megabytes down to just a few megabytes for thousands of images or even kilobytes. The goal of Create ML is to abstract much of this and make it simple and easy to use. But to prove it, let's take a look at a demo.

First, to set up the problem, I started by running an app that's using a state-of-the-art image classification model that's already in the industry. This one, though, is quite large. It's 100 megabytes in our app. And if we run it, we have some fruits but it's not quite what I was looking for.

I'd really like it if, instead, we could classify these particular ones. So what we can do is we can switch to a new playground and import CreateMLUI and walk through how to do this using the UI for it. We can define a builder. Initialize it. And to enable drag-and-drop training, we can show the builder in the live view.

This brings up a prompt in the live view to drag in images to begin training. And here I set aside some photos of fruits. Here's some blueberries and other types. And you can drag them in and automatically an image classifier model begins training on the Mac. All of this is accelerated by the GPU on however many categories you end up training on.

It automatically tells you what the accuracy is on the training data set, but what's more helpful is to try this on new images that the model hasn't seen before to predict how it will do on real use cases. So I can drag in this other folder containing unseen images.

And now the model is evaluating all these new types of fruits. And if you scroll, you can see what the true label is of each type as well as with the predicted one was by the model. And if you're happy with this accuracy, what you can do is you can take the model and drag it into your app. I'll add it here. And if we take a look, this model is 83 kilobytes. It's a huge savings down from hundreds.

So we can delete the old model that we were using before. And in the view controller, we can initialize this new one, ImageClassifier. We can then re-run the app, bring up the simulator, and see how it does on some of those fruits. On the raspberry, it can now correctly predict it since we trained the model to recognize raspberries. We can even see if it can distinguish from strawberries and it can now.

But there are other workflows you can use. Perhaps you want to do this programmatically or perhaps you want to automate it. We can also walk through how to use Create ML to do so. So now we can switch to another playground and import Create ML. Since we'll be using URLs, we also can import foundation.

And since, on our desktop, we still have these folders of fruits, we can say where they are and also say where the testing fruits are. And then the next step is to actually train the model. So we can define a model, and we can initialize an image classifier.

And now if we take a look at what auto complete shows to us, we can see we can provide training data in the form of a dictionary of labels to arrays of images or we can use a data source or even specify model training parameters if we want to. Let's use a data source. And we'll use label directories since that's how our data is organized and specify the training directory.

And since we're running in the new [inaudible] mode of Xcode playground, I just need to hit shift enter and the model begins training right away. You can even pull up the console and see output of one, its extracting features and how many iterations it's running through. Afterwards, you can also open quick looks and see the name of the model and how many instances it's trained on.

Now we might want to evaluate on the testing data that we've set aside. So what we can do is we can call evaluation on another data source since that folder is organized the same way, specifying the URL of the testing data. You can hit shift enter and now the model is evaluating testing images. Once it's complete, we can also look at the quick look and see how many examples it evaluated on as well as how many classes were in that folder altogether and the accuracy. If we're happy with that, we can write it out.

And say that I want to write it to the desktop with the name fruit classifier ML model. Once I do, you can see this new model appears on the desktop. We can double-click it and take a look and see it's exactly the same. This is also 83 kilobytes. Furthermore, we can integrate it back into our app the same way. Let's recap.

We saw two ways of training image classifier models in Create ML. One was with the UI which makes it super simple to drag-and-drop your training data and evaluation data to produce an ML model. The other way was with the Create ML API. If we walk through some of this code, we can see the first thing we had to do was import Create ML. The next was to specify where our training and testing data was and then actually begin training the model by specifying how our training data was laid out. We can then evaluate on the testing data and finally save ML model.

If you want to automate this, you can also turn these into scripts, which is a very popular way of saving what you've done and re-running it whenever. You can then change permissions on the file and run them like so. Or for other workflows, you can always use Swift command line [inaudible]. So we've seen today how to train image classification models using a few different workflows. But next, I'd like to pass it off to Tao to talk about natural language. Thank you.

Thank you, Lizi. Hello everyone. My name is Tao. I'm an engineer here at Apple working on the Core ML team. You just saw how easy and intuitive to train an image classifier with just a few lines of code. Now I'm going to show you the same can be done for natural language.

In this year's release, we're going to support two natural language tasks: text classification and word tagging. Today, I'm going to focus on text classification. For details on word tagging, please join the natural language session that happens tomorrow. Text classification can be used in a few machine learning applications. For example, sentiment analysis. The energy of developers is amazing. That's a positive note. You want your app to know it.

Spam analysis. If you saw this message in your mailbox, you know it's very likely it's spam. So you want your app to know that as well. Topic analysis. The Warriors just had an amazing comeback win. That's a sport post. You want your app to be able to classify that.

So to train such a classifier, the first thing you do is to collect some training data. With Create ML, we support a few different ways for you to organize your training data. For example, label directories. Here you have two folders. One named positive, the other one named negative. Within each folder, you have a number of articles with just raw text whose truth label is simply the name of the folder.

Alternatively, you can prepare your training data using simple CSV where you prepare your raw text and the truth label separated by comma. We also support JSON formatting the training data and know that we just talk about the training data organization and you can actually organize your test data in the exact same way.

Now with your training data and test data ready, what other steps involve to train such a text classifier? A typical workflow would look something like this. You start with your raw text. You do a language identification to figure out which language it is in. You convert that into tokens.

And then you convert that into some feature values and then you can apply a machine learning model that gives you some predictive value that you have to map to some desired label. And then you can compare that label to your truth label and start iterating on it. With Create ML, though, we took away all these complexities so that all you need to do is to prepare raw text with their truth label and start training immediately.

Now let me give you a concrete example like how you can train such a classifier and use it. For example, we have this simple app called Stay Positive whose purpose is to encourage positive post. If a user entered I hate traffic, the background turns red and it will disable the post button. I love driving my car at five mile per hour just chilling in traffic. That's a positive post. We encourage you to post it. Just imagine what our Internet would look like with this app running on everybody's phone?

Now, in order to do that, let me give you a live demo. So to train such a classifier, the first thing I do is collect some training data. On my desktop, I have a train folder and also a test folder. In train folder, we have two folders. One is named positive, the other one negative, and there are a number of articles in each folder.

And test folder is organized in a very similar way. So the first thing I do is to import Create ML. Now I need to tell the [inaudible] where to find my training data. For that, I'm simply using a URL capability and then I can start training my model using the label directories that Lizi just showed you.

Look. The training has started. As you can see on the bottom there, there is some progress report for you to check. Looks like training has finished. Now you can check some basic performance numbers on this model. For example, model.trainingMetrics that shows you this model has been trained on over 2000 examples and accuracy is 100%.

But how does it perform on some unseen data? So I'm going to do the same to define test data and then evaluate that model on the test data. As you can see, we have 77 test examples, and we are achieving over 94% accuracy, which is very good. I'm sure you want to iterate on that if you want to see like even a higher number, but this number is pretty good enough for my app so let me just test it out.

So to save out the model, what I need to do is define a URL where it's saving to and then write out a model to my desktop. Looks, that model has been saved. So now I need to switch back to my app. Just drag and drop it. There you go.

Now I can start use it. I will do let model equal to textClassifier which should auto complete. And then I'm going to insert some basic inference code. In this inference code, as you see, the first line I do is using model.prediction to get prediction. And then in order to hook up with this simple app UI, I just convert that into some double value.

Let's give it a try. Yeah. Let's try some example we have showed you. I hate traffic. Negative. I love driving my car at five mile per hour just chilling in traffic. Positive. Let's try something different that'll be fun. Machine learning is hard. Create ML makes it so easy. Positive.

So that's how you train your customized text classifier and drag it into your app to use it. Here's a recap. So to train such a classifier, the first thing you do is to specify your data. You specify your training data as well as your test data and then you can create your model on the training data. To evaluate its performance, you evaluate a model on the test data.

Finally, to use your model in your app, you simply save it out using this write API. To summarize, with just a few lines of code, you can train your customized text classifier simple intuitive. With that, I'd like to hand back to Gaurav who is going to talk about tabular data. Thank you.

Thank you, Tao. Besides images and text, another common source of data that occurs very frequently when you're solving a machine learning problem is tabular data. What I mean by tabular data, I mean the data is in special format or in a table format. This kind of data occurs fairly frequently.

For example, let's just say you're trying to predict house prices using number of beds, number of baths, or square footage. Generally the data is arranged in a tabular format. You want to predict the quality of wine using its chemical compositions. Chances are data will be arranged in table format. Or something even simple like where to hop, which bar to hop tonight using happy hour or its price, the data will be in tabular format.

To handle the data which is in tabular format, we actually introduce a new data structure which we call as MLDataTable. MLDataTable is based on [inaudible] technology that we will discuss in detail tomorrow. There's something interesting about these data tables. The rows contains the observations or examples. So here, house number two has four bed, three bath, and 500K price.

The columns contains what we call as features. So the beds are features, baths are features, square feet, etcetera are features. There is one special column that we want to predict, in this case price, and this column is known as target or response variable. The whole idea behind tabular data is that we want to predict target variable as a function of one or many of these features.

So what are the common sources that we support? Well, CSV, JSON as well as you can actually have code. So let's talk a little bit more about MLDataTable. First, you can read data simply by using CSV. What is more important that you can access the column using a subscript notation. So all you do is house or the price and you get an entire column of price.

You can add two columns, subtract two column, multiply two column, divide two columns. And the way you do it is in very natural looking syntax. So you just simply say house or the price divided by house or the square foot to get price per square foot. Behind the scenes, this calculation is done using [inaudible] evaluation and through vector operations.

It can also do some of the other interesting things. For example, you can split data table in training as well as you can even do filtering. So for example, if you're only interested in large houses, you can create an indicator variable and filter it out. There a lot of operations that data table support. I urge you to try it out in Xcode playground. They're fun. Now once you have data in data table, you would like to do the training on it.

Create ML supports a large number of algorithms such as Boosted Tree Regression, Random Forest, etcetera. And all of these algorithms are represented by their class. In order to train your model, you only have to write one line of code. Basically, you tell what is the target and where you're getting the data and which is the algorithm you are instantiating.

So in this case, let's just say you are running Linear Regression or Regularized Linear Regression, you just actually tell it that the data is house data and the column is price. If you do Boosted Tree Regression, just replace Linear Regression with Boosted Tree and you're all set. Now Random Forest like that. Plus we also provide a high level abstraction MLRegressor that automatically runs all these algorithms and choose the best one for you.

This is in line with our philosophy that you should focus on task. So the task is to predict the price. You should not focus about nitty-gritty details of the algorithm. Having said that, in case you're an expert, you can actually use Boosted Tree and change its parameters also.

So a complete end-to-end would look like this. It follows exactly the same pattern as image and text. First, you specify the data. Second, you just create your model. Third, you evaluate the model. And once you're happy, you save it out. So tabular data, image data, or text data, they all follow the same pattern.

So let's just take a quick summary of what we saw in this session. So Create ML is our ML framework insert. It's very simple to use and it is very powerful and it leverages core Apple technologies. You do end-to-end machine learning in Swift on your Mac. We also discussed about our workflow.

Once again, you start from an experience. What is the experience you're trying to enable? Then define the problem. Then collect the data. Make sure this data is reflective of the real-world usage of your scenario. Then you train the model. And finally evaluate it. And once you are happy, you just save it out.

Create ML is in Swift. And it's available on macOS Mojave. You can use it in Xcode Playground, Swift Scripts and [inaudible]. So please try it out. We would love to hear from you. We are here to receive your feedback, and we hope that you will love it as much as we do.

We will be in the machine learning get together as well as the labs. So there is -- tomorrow there is a get together. We will be in labs also, so please give us your feedback. There are also related sessions in the WWDC App. We have Core ML session tomorrow morning and ML session tomorrow afternoon, Vision sessions on Thursday. And we have labs on Wednesday and Friday. Thank you.