Hi there, I’m Jaimin Upadhyay, an engineering manager on the On-Device Machine Learning team at Apple. Today, I would like to talk about how you can make use of Apple Intelligence and machine learning in your apps and personal projects. Whether you are an app developer ready to tap into Apple Intelligence through UI components or directly in code, an ML engineer converting and optimizing models for on-device deployment, or an AI enthusiast exploring the frontier of what is possible on your Mac, we have the tools for you. I’ll walk you through a high level overview of these tools, highlight the latest additions, and point you to resources to learn more along the way.

We will start with an overview of the intelligence built into the operating system and its relationship with your app. Next, we will explore how you can programmatically tap into this intelligence through our system frameworks. We will then talk about how Apple’s tools and APIs can help you optimize and deploy any machine learning model for on-device execution. And we will finish up by discussing how you can stay on top of the latest innovations in ML and AI on Apple hardware. We’ve got a long and exciting tour to cover, so let’s get started. We start with platform intelligence.

Machine Learning and Artificial Intelligence are at the core of a lot of built-in apps and features in our operating system. Whether it’s Optic ID to authenticate you on Apple Vision Pro, or understanding your handwriting to help you with math on iPad, or removing background noise to improve your voice quality on FaceTime, machine learning is at the core.

ML Models powering these features have been trained and optimized for efficiency on device and last year marked the start of a new chapter, to bring generative intelligence into the core of our operating systems, with large foundation models that power Apple Intelligence. This brought Writing Tools, Genmoji, and Image Playground across the system, making it easy to integrate them into your apps. If you’re using system text controls, you’ll get Genmoji support automatically. You can even use the APIs to make them appear right in your text.

The Image Playground framework provides Swift UI extensions to bring up the imagePlaygroundSheet in your app. And, for most of you, using the standard UI frameworks to display textViews, your apps were already set up to support Writing Tools. It's that simple. You can either use standard views or add a few lines of code to your custom ones. This way, your users can easily access Apple Intelligence within your apps with a consistent and familiar UI. But, what if you want to go beyond the default UI or need more control?

This brings us to the topic of ML-powered APIs that give you programmatic access to system models and capabilities. We offer a wide variety of such APIs. While some provide access to prominent system models with essential utilities, others expose convenient APIs for specialized ML tasks. Let's dive into these by revisiting how you can integrate image generation into your app. iOS 18.4 introduced ImageCreator class to ImagePlayground framework.

This lets you create images programmatically. Just instantiate the ImageCreator. Request for images based on some ideas. Here, we use a text prompt and a selected style. Then, you can show or use them in your app as you prefer. Also in 18.4, we introduced the Smart Reply API. You can let your users choose generated smart replies for their messages and e-mails, by donating the context to a keyboard. Let’s take a quick look at how you can set it up. To donate your conversation, configure a UIMessage or UIMail ConversationContext with your data, then set it on your entry view before the keyboard is requested.

When a user selects a smart reply from the keyboard for an instant message, it will be directly inserted into the document. However, in a mail conversation, the selection will instead be delegated back to your view’s corresponding insertInputSuggestion delegate method. You can then generate and insert your own longer replies appropriate for an e-mail. To learn more, check out “Adopting Smart Reply in your messaging or email app” documentation page. Note that this is all running on device and using Apple’s foundation models. In iOS 26, we are going even further with the introduction of: the Foundation Models framework.

It provide programmatic access to a highly optimized on-device language model that’s specialized for everyday tasks. Now it can power these features across all your apps. It’s great for things like summarization, extraction, classification, and more. You can use it to enhance existing features in your apps, like providing personalized search suggestions. Or you can create entirely new features, like generating itinerary in a travel app.

You can even use it to create dialogue on-the-fly for characters in a game. That one is my personal favorite! Prompting the model is as easy as three lines of code. Import the framework, create a session, and send your prompt to the model. Since the framework is on device, your user's data stays private and doesn't need to be sent anywhere.

The AI features are readily available and work offline, eliminating the need to set up an account or obtain API keys. And all of this, at no cost to you or your users for any requests. The Foundation Models framework provides much more than simple prompting for text responses. Sometimes you need an LLM to generate structured responses that you can use directly in your app.

This is easy with the Foundation Models framework. You can take existing types in your app and mark them as generable. Also add some natural language guides to each property, along with optional controls over their generated values. This lets you use Guided Generation with a simple prompt. When you indicate the response to generate your type.

The framework will customize the language model decoding loop and stop the model from making structural mistakes. Your data structure is filled with the correct information, so you don’t have to deal with JSON schemas. Just focus on the prompt and let the framework do the rest! The synergy between Swift, the framework and your custom types makes it easy for you to rapidly iterate and explore new ideas within your app.

When developing your use case, it's important to consider the knowledge available to the foundation model. In addition to information provided via your prompt and generable type descriptions, The model has a core set of knowledge derived from the data it was trained on. This data was fixed in time and does not contain recent events. While the model is incredibly powerful for a device-scale model, it’s not as knowledgeable as larger server-scale models. To help with use cases requiring additional knowledge from your app or over the network, the foundation model’s framework supports tool calling.

Tool calling lets you go beyond text generation and perform some actions. It provides the model access to live or personal data, like weather and calendar events, not just what was trained months ago. It can even let the model cite sources of truth, which allows the users to fact-check its output. Finally, tools can take real actions, whether it’s in your app, on the system, or in real world.

This was just a sneak peek of the framework's awesome capabilities, but there is so much more. For a more detailed introduction watch “Meet the Foundation Models framework” session. There you will also learn about streaming responses, stateful sessions, and the framework's tight integration with Xcode. And if you prefer learning by doing, we have a code along session for building your first intelligent app using the new APIs.

We also have a session dedicated to design considerations for your use cases. It focuses on best practices to help you write reflective prompts, AI safety considerations, understanding what is possible with a device-scale language model, and some solid strategies for evaluating and testing quality and safety. Be sure to check out “Explore prompt design and safety for on-device Foundation models” to learn more.

The new Foundation Models framework joins the suite of other Machine Learning powered APIs and tools you can use to tap into on-device intelligence for your app’s features. These frameworks each focus on a specific domain with highly optimized task-specific models. There is Vision to understand the content of images and videos.

Natural language to identify language, parts of speech, and named entities in natural language text. Translation to perform text translations between multiple languages. Sound analysis to recognize many categories of sound. And speech to identify and transcribe spoken words in audio. All with just a few lines of code. Let me highlight some interesting new additions to these frameworks this year.

Let's start with Vision. Vision has over 30 APIs for different types of image analysis. And today, Vision is adding two new APIs. Vision is bringing improvements to text recognition. Instead of just reading lines of text, Vision now provides document recognition. It can group different document structures, making it easier to process and understand documents.

Vision also has a new lens smudge detection mode. It helps you identify smudges on camera lens that can potentially ruin images. For more details on Lens Smudge Detection and the other cool new additions to Vision, check out the session “Reading documents using the Vision Framework” for more details.

Next, let’s talk about the Speech framework. The SFSpeechRecognizer class in Speech framework gave you access to the speech-to-text model powering Siri and worked well for short-form dictation. Now in iOS 26, we’re introducing a new API, SpeechAnalyzer, that supports many more use cases and leverages the power of Swift. The new API lets you perform speech-to-text processing with very little code entirely on device. Along with the API, we are providing a new speech-to-text model that is both faster and more flexible than the previous one.

You pass audio buffers to the analyzer instance, which then routes them through the new speech-to-text model. The model predicts the text that matches the spoken audio and returns it to your app. The new model is especially good for long-form and distant audio, such as lectures, meetings, and conversations. Watch the “Bring advanced speech-to-text to your app with SpeechAnalyzer” session to dive deeper.

Apple’s ML powered APIs offer tons of capabilities that your app can readily take advantage of! And many of these APIs can be extended or customized to your specific use case. The Create ML app and framework give you the ability to fine-tune the system models with your own data. Create your own image classifier to use with the Vision framework, or a custom word tagger to use with Natural Language. You can even extend the capabilities of Vision Pro to recognize and track specific objects with 6 degrees of freedom for spatial experiences.

So far we have talked about how you can leverage or extend the ML and AI powered capabilities built into the system. Next, let’s talk about bringing any model to device. When choosing and integrating a model into your app, there is a lot to consider. But it is made easy with Core ML. All you need is a model in the Core ML format. These model assets contain a description of the model’s inputs, outputs, and architecture along with its learned parameters. You can find a wide variety of open models in the Core ML format on developer.apple.com ready for use.

They are organized by category with a description of each model’s capabilities and a list of different variants along with some high-level performance information on different devices. Similarly, you may want to check out the Apple space on Hugging Face. In addition to models already in Core ML format, you will also find links to the source model definition. These model definitions are often expressed in PyTorch along with training and fine-tuning pipelines.

Core ML Tools provides utilities and workflows for transforming trained models to Core ML model format. These workflows not only directly translate the model’s representation but also apply optimizations for on-device execution. Some of these optimizations are automatic, such as fusing operations and eliminating redundant computation. However, coremltools also provides a suite of fine-tuning and post-training based model compression techniques.

These will help you reduce the size of your model and improve its inference performance in terms of memory, power and latency. These techniques are opt-in and allow you to explore different trade-offs between performance and model accuracy. Check out the “Bring your models to Apple Silicon” session from WWDC24 to learn more. Also, make sure to check out the latest release notes and examples in the user guide.

Once you have your model in the Core ML format, you can easily integrate it with Xcode. You can inspect your model’s key characteristics or explore its performance on any connected device. You can get insights about the expected prediction latency, load times, and also, introspect where a particular operation is supported and executed, right in Xcode. New this year, you can visualize the structure of the full model architecture and dive into details of any op. This brand new view helps you build a deeper understanding of the model you are working with, making debugging and performance opportunities incredibly visible.

When it's time to get coding, Xcode generates a type safe interface in Swift specific to your model. And integration is just a few lines of code. At runtime, Core ML makes use of all available compute, optimizing execution across the CPU, GPU, and Neural Engine. While Core ML is the go-to framework for deploying models on-device, there may be scenarios where you need finer-grained control. For instance, if you need to sequence or integrate ML with graphics workload, you can use Core ML models with both MPS Graph and Metal.

Alternatively, when running real-time signal processing on the CPU, Accelerate’s BNNS Graph API provides strict latency and memory management control for your ML task. These frameworks form part of Core ML’s foundation and are also directly accessible to you. This year, there are some new capabilities in BNNSGraph, including a new Graph Builder that lets developers create graphs of operations. This means you can write pre- and post-processing routines or even small machine-learning models to run in real time on CPU. Check out “What’s new in BNNS Graph” for all the details.

Finally, let’s talk about how you can keep up with the fast-paced development happening in machine learning and how can the Apple platform assist you. ML research is moving at a rapid pace, there’s advancements made every single day. New models and techniques are being explored and built at an unprecedented rate. There is a lot to try and keep up with.

It can be challenging without the right tools and resources. To keep up with the current frontier of exploration, one needs the ability to run large models, tinker with unique architectures, and learn from an open community. We have sophisticated tools and resources to help on your endeavors exploring the frontier. One such powerful tool is MLX.

It’s array framework for numerical computing and machine learning. It’s designed by Apple’s machine learning researchers and developed fully open source. MLX provides access to state-of-the-art models and the ability to perform efficient fine-tuning, training, and distributed learning on Apple Silicon machines. MLX can run state-of-the-art ML inference on large language models like Mistral with a single command line call. For example, here it’s generating code for quick sort with a maximum token length of 1024. This allows you to stay in-step with state-of-the-art research, thanks to the open source community working to make these models work with MLX.

The MLX community on Hugging Face has hundreds of frontier models readily available to you through one line of code. Check out “Explore large language models on Apple silicon with MLX” session to learn about how you can run DeepSeek-R1 on your Apple Silicon machine. MLX is designed to take advantage of the best of Apple Silicon. This includes a new programming model specific to unified memory. Most systems commonly used for machine learning have a discrete GPU with separate memory. Data is often resident and tied to a specific device.

Operations run where the data is. You cannot efficiently run operations that use data from multiple pools of memory. They would require a copy in memory. Apple Silicon, on the other hand, has a unified memory architecture. This means that the CPU and the GPU share the same physical memory. Arrays in MLX aren’t tied to a device, but operations are, allowing you to even run different operations on CPU and GPU in parallel on the same buffer.

Check out “Get started with MLX for Apple silicon” session to learn about this unique programming model and other features of MLX. You can even fine-tune your model with a single line of code and scale up as needed for distributed training easily. It’s available in Python, Swift, C++ or C, and other languages of your choice through the multiple bindings created by the open source community.

In addition to MLX, if you are using one of the popular training frameworks like PyTorch and Jax, we’ve got you covered with Metal, so you can explore the frontier without deviating from the standard tools that have been embraced by the ML community over the years. Lastly, developer.apple.com is a great resource for AI enthusiasts and researchers to get a peek at the latest machine learning resources from Apple. With that, we've covered our agenda. Let’s step back a little and take a look at everything we talked about today.

Based on your needs and experience with models, you can choose the frameworks and tools that best support your project’s Machine Learning and AI capabilities. Whether you want to fine-tune an LLM on your Mac, optimize a computer vision model to deploy on Apple Vision Pro, or use one of our ML-powered APIs to quickly add magical features to your apps, we have you covered. And all of this is optimized for Apple Silicon, providing efficient and powerful execution for your machine learning and AI workloads.

We are sure you will find the resources we went over here helpful and can’t wait to see the new experiences you create by tapping into Apple Intelligence. There has never been a better time to experiment and explore what you can do with machine learning and AI on Apple platforms.

Here we covered just the surface. I highly encourage you to check out the machine learning and AI category in the Developer app and on our developer forums to learn more. Ask questions and have discussions with the broader developer community there. I hope this has been as fun for you as it has been for me. Thanks for watching!