Video hosted by Apple at devstreaming-cdn.apple.com

Configure player

Close

WWDC Index does not host video files

If you have access to video files, you can configure a URL pattern to be used in a video player.

URL pattern

preview

Use any of these variables in your URL pattern, the pattern is stored in your browsers' local storage.

$id
ID of session: meet-with-apple-290
$eventId
ID of event: meet-with-apple
$eventContentId
ID of session without event part: 290
$eventShortId
Shortened ID of event: meet-with-apple
$year
Year of session: 2026
$extension
Extension of original filename: mp4
$filenameAlmostEvery
Filename from "(Almost) Every..." gist: ...

Meet with Apple • Session 290

Apple Intelligence Group Lab

2026-11-04 • 1:00:53

Connect with Apple engineers online to ask questions, get advice, and follow the discussion about App Intents and Siri AI, the Foundation Models framework, and more.

Open in Apple Developer site

Transcript

Hello. Welcome to the Apple Intelligence group lab. I’m Ernie. I’m part of the Evangelism team here at Apple, and I’m joined by a panel of experts from the Machine Learning Frameworks team. I’d like to start off this morning with a quick round of introductions. Louie, why don’t we start with you? Yeah. I’m Louie. I’m an engineer on the Foundation Models framework team here at Apple.

Hi, everyone. I’m Karina. I’m an engineer on the Core AI team. Hi. I’m Roman. I’m an engineer on the framework team. Hi, I’m Leslie, and I’m an engineer working on Siri AI. And my name is Michael. I run the core team. Awesome. Thank you, Michael. Thanks, everyone, for taking the time to be here today.

In addition to those you see on screen, there’s a team behind the scenes helping to triage all of your inbound questions. So please keep them coming. If you have code-specific questions or somehow we can’t get to your question today, we’d really encourage you to bring your questions to developer forums at developer.apple.com/forums. We’ll continue the discussion.

If you have a bug or a feature request, we’d encourage you to go to feedbackassistant.apple.com, as we really want to focus on the questions that are going to help you the most. What do you mean, bugs? We fixed them all this year, right? - Yeah, there’s actually bugs. - We don’t have bugs. What? Anyway. Yeah. Okay. Ideas are welcome. Yeah, I agree.

All right, so we just announced the new Siri AI developer preview. Michael, can you tell us more about what we launched, what it means for developers who already use App Intents? Thanks, Ernie. We’re very glad that all of you could join us live today. You are, after all, the reason we do what we do. We’ve been thrilled to see all of the adoption of our schema-based App Intents APIs from this year designed for specific types of apps.

With the developer preview we announced yesterday, we are taking the first step in expanding Siri AI’s new capabilities, like understanding personal context and natural language, to every kind of app. Yes, even yours. Using the new Tool Intent API, your existing app intents can become tools that Siri calls directly when people make requests. Siri can find your app’s tools if your app is in the foreground or if it’s named.

You can also create custom app skills for Siri now, bundles with markdown files that help Siri master even complex app interactions. If you already use App Intents in your app, you’re already most of the way there. Just make your intents into tools and fill in language to describe them and their parameters.

Siri can also find content from any kind of app and act on it. All you have to do is create an entity to represent that content and index it into Spotlight using the IndexedEntity APIs. If you’ve already adopted those, you’ve already done the hard work here, and we’d love for you to go test it out.

We’re also providing a powerful set of Xcode features for you to bring up and iterate on your Siri integration. The Siri Integration Advisor provides feedback on the shapes and descriptions of your tools and skills, and the Siri Evaluations framework lets you write scenarios to test and tune your tools and skills so they work no matter how people ask for them.

Xcode now even includes a set of coding agent skills to take your app from no Siri to full Siri integration in no time at all. We would love for you to try all of this out in the iOS and macOS 27.3 betas, since we’re still refining the APIs. We are really looking forward to getting your feedback on them. Right now, you can test ad hoc and distribute using internal TestFlight, with shipping to the App Store coming this spring.

Oh, and one last thing. You’ve told us that you want more control over the way your entities look in Siri AI. Well, we’re introducing a new Entity Snippets API, so the visual representations of your content can reflect the unique style of your app. You’ll be able to ship these snippets to the App Store when the 27.3 OSes are released.

We are so excited to see what you’re going to build, but first, time for questions. Awesome. Thank you, Michael. Wow, that’s exciting. I mean, what a big announcement. I mean, yeah, it’s so great to be in this world again here with us working with Siri AI. Okay, like Michael said, let’s go ahead and get going with questions. Our first question: Will users be able to provide their own written skills to influence how Siri AI handles specific tasks between multiple apps?

I want to take a moment. Sure. You should be able to provide written skills to influence how tools interact between each other in your app, but if your skill actually describes how it should be used with a different app, I think you should probably refrain from that. Good question on that. Skills as in, like, being able to guide a more complex workflow. - Exactly. - Specifics. Yeah. You should be able to stitch together multiple tools, work with each other. Yes. Yeah. That’s similar to, like, Claude skills idea as well, right? Like a generic skills in general. Yeah.

Yeah. That’s exciting. I can’t begin to tell you how many developers in our workshops have asked for this. So a lot of very happy developers, I know, out there that we’re now providing this. So that’s awesome. Nice. All right. Next question. The new tool intent in Siri capabilities look promising.

Do you expect this to replace the app schema domains? Do you expect the set of app schema domains to expand in the future? Very good question. - I can take that one. - Yeah. So as you can probably guess, we don’t talk about future plans. But what I can tell you is that app schemas are really great and really powerful for the domains that they do cover.

If you work on an app that is focused around messaging, adopting an app schema will get you really broad capabilities with Siri, and it will get you those capabilities very easily, because you don’t have to do things like evaluate a language that you’ve written for the model. All you need to do is implement the schema, and it will work very reliably and very consistently across different invocation points.

So they are not going away, and they’re still an excellent fit for lots of apps. And you can also use them together, right? Like you might have an app intent has a schema, but you could also create a new app intent. Maybe that doesn’t have a schema. That’s right. Right. Yeah.

And you don’t have to choose one or the other. It’s often a complementary thing. So if you are a calendar app, you might want to implement the calendar schema, but you might also write some additional app intent that takes that same entity, but does something custom with it, something unique to your app.

That is totally reasonable. Yeah, I have to say too, in the schemas, we’ve done all the heavy lifting, really, a lot of great engineering work has been done. So if a schema does work for what your app is trying to achieve, by all means, I think you’ll see a lot of saved time and value from using one of those schemas. So very cool.

Okay. Next question. Should @AppEntity(.generable) be used when the entity is the only parameter of an intent, and is it used anywhere else, or should its properties be added directly to the intent? In this case, very good question. I can try answering this one. So imagine that you have an intent that has all the parameters for creating a blog post. And in that blog post, you also have an option to add a poll.

So let’s think about what kind of parameters would you add to that intent? There would be a title and the body of the blog post, and the poll itself is going to be optional. So you don’t have to actually fill in what are the options of that poll and, like, when it actually ends. So imagine that you want to group all the poll information into its own entity and make it optional on that create blog post parameter. Sorry. Create blog post intent.

I’m curious, like, the general goal here is that similar to some of the Generable that we’ve worked with Foundation Models in terms of guided generation constraints? It is. Yeah. The name is not a coincidence. It has very similar… It has a very similar role, pretty closely, very similar name. And Siri will be able to actually fill in the properties of that entity.

  • That’s really cool.
  • Yeah. Yeah. Really exciting. And that’s available through, like, Core AI and Foundation Models as well, on, like, constrained decoding and kind of Generable in terms of outputting specific schema. So that’s, like, a consistent concept we have across frameworks. Yes. That’s great. Awesome. All right. Next question. For entity types that use a schema domain, does it matter if you add a custom title or description to a property?

I think, if I remember correctly, those are used in Shortcuts, but they are not used by Siri. So especially if you’re implementing a property that the schema owns, meaning that the schema defines, you can add them for kind of visual treatment in Shortcuts, but Siri won’t use them. So those are really strings to show in the UI. Right. Yeah, I see. Actually, for the new tool intent schemas, if you actually provide the title and description, they will be used in Siri.

Yes. That is different. Yeah. For tool intents. I guess, like, they are used by the model themselves, right? Not just as, like, identifying, but actually getting the model to be able to identify what tool to use better. That’s exactly right. That metadata is used by Siri to really understand what that parameter is going to be used for.

And so it is quite important to review that title and description, especially if you’re adopting the new tool intent traits. Yep. And you see the tools that we… The Xcode features we’ve added will guide you towards making those better, right? If you’re writing a tool and you’re like, maybe I’m not getting quite the behavior I want, often you can tweak those titles and those descriptions of the intent or the parameters, and they will begin to perform better.

Yeah. I think those are actually would be good recommendations for, like, bringing a custom model, like tool calls and guided generation for Foundation Models as well, because I don’t think we provide specific guidance when you bring, like, an open source model, but I think those are pretty standard, like, how do you define a tool so that the model itself can retrieve it better? So we could borrow some, like, you guys’ guidance.

Yeah, that’s an excellent point. Great question. This next question I’m really excited about, because we get asked it all the time. So question is, what signals does Siri use to choose between my app’s tools and a system app like Calendar when a request overlaps both, even with my app on screen? Siri went to Calendar. Is indexing entities with IndexedEntity the recommended way to get considered?

Leslie, do you want to? I’d love to. So I think there are two parts to this answer. So I think for schematized intents, like the ones that we released for Calendar in 27.0, the foregrounded app does play a role in us, like, routing the request to your app, and that will work regardless of whether the user mentioned your app name or not.

With the new Tool Intent API, there are a couple of ways that the foregrounded signal comes into play. One is if you’re foregrounded, then those tool intents are considered in context by Siri. And so Siri will consider your app’s, like, custom tools that manage those calendar event entities as part of when trying to process the user request. The second is when your app is in the foreground, you are able to expose entities on your views through the View Annotations API, and that will work now, whether you annotate those views with schematized entities or non-schematized ones.

Yeah. Excellent. Great. Thank you for that response, because definitely a question we get asked all the time is, you know, yeah, how does that decision get made? So thank you for that question. All right. Next question. So tool intent is only for when the app is in the foreground, right? Can Siri call into tools for apps that are in the background as well? It can. Yeah.

Leslie, feel free to tell me where I’m wrong here, but there are several conditions that will bring your tool intents into scope. One is the app’s in the foreground, two is the app is mentioned explicitly by name, three is that the app is currently running a Live Activity, and four is that the app’s entities are in context, usually from a search, right?

So if you use a custom notes app, and you index those entities, and you ask Siri to retrieve that note by its name or by something in it, Siri finds the note. All of a sudden, your custom notes app’s actions are also in play there. And if you want to archive that note or do whatever, you can.

Is that right? Yeah, that’s exactly right. I think maybe the only thing I’d add is that the same kind of rules apply for skills as well. And so I guess… That’s awesome. Really powerful. All right. Next question. How does Siri… Kind of similar to a previous question, but I think it has some unique aspects here. How does Siri decide between two apps that can handle the same request, and how does it rank our custom tools against Apple’s built-in apps, such as Notes or Mail? So similar to the previous question, but a little different too. So very common question.

Right? Yeah. So building, I guess, on the previous answer. So foreground is one signal that we use. So for schematized intents specifically, foreground signal is one that we use. Interaction history is one that we use for schematized intents, but for tool intents, we go through kind of the rules and signals that Michael already outlined just before.

And interaction history, it just means whether the app has been used, basically. And we look at both how often the user has launched or foregrounded the app. But we will also look at your app’s donations, which is a really important point. So interaction donations are basically, if you have an app that, say, opens a message, and you ask the user… Ask Siri to open that message, we’ll, of course, automatically record that donation to say, hey, that happened. And then we’ll use that as a signal later on to help prioritize your messaging app over another.

However, if that happened within your app, and the user’s tapping around and chose, by tapping around in your UI, to open a message, Siri doesn’t know it, and so it’s a great idea to let Siri know about the fact that that happened. And the way you do that is using the IntentDonationManager APIs.

That’s what she’s talking about. Yes. Are these recommendations all documented? Because that would be really useful to know the rules and the prioritization. Yeah. There’s some content in our dub dub talks from last year about this that’s definitely worth seeing. And then the intent donation APIs are also pretty well documented. So yeah, a lot of great videos out there too from last year’s dub dub. Okay. Next question. What’s the recommended method to interface with Siri AI without indexing app data via Spotlight?

That’s a good question. Let’s see. So I don’t quite know if that is a question for the new tool intent stuff that we are releasing or the older schematized stuff. I think, in general, definitely the case that indexing your entities with IndexedEntity is the preferred method here. If you can at all, do it. Highly recommend that you do.

For schematized intents, there are a number of domains which are designed to call your queries instead, because we’re aware that not everybody can always index all of their content, right? For example, if you’re working on a media app that plays audio, there’s a query you can implement that, instead of indexing all the audio, because you probably have a large catalog, and you can’t index the entire world into the user’s phone.

Totally, totally makes sense there. I think for tool intents, the story is a little bit different, where again, IndexedEntity is still recommended. If you can do it, you definitely should. But I think, lastly, we have support for some DynamicOptionsProvider and enumerable and stuff on those to retrieve entities as well. Right? That’s right. Do you support string query as well? Not currently. And so, to Michael’s point, we definitely recommend that you go through the IndexedEntity route where possible.

Yeah. I guess, like, to jump in here in terms of, like, the model can produce an output, but actually grounding it in something that is factual and, like, is the actual value, that those features all provide the grounding, essentially. Right? Yeah. This is like a, I guess, a foundational question, which is all about retrieval and, like, what you would call grounding. Yeah.

Yeah. I guess, like, you can bring in a lot of these open source models, which can do tool calling, but we don’t have the mechanism, like, you’d have to implement in the application logic layer to validate that this is, like, the valid value or, like, the valid entity that you’re trying to call.

So I guess you guys are basically have provided all the implementations. App Intents is very much a powerful layer over these models because it provides infrastructure, like queries and entities, to that creates that structure. So that structure doesn’t have to be built custom for every developer, for every tool, which I think is a big value add. Yeah. I wonder, can that integrate with, like, Core AI to bring even more powerful app experience?

Like, can I provide some of the, I guess, performance of the tool, but the validation of those tool parameters can be validated through App Intents framework? Yeah. Actually, that brings up a good point, that one of the things we realized as we brought up all of this tool intent infrastructure is you could do some really powerful things with it.

It’s more powerful than it initially looks. And, like, one of the things that you can do is you can take… You can create a tool that takes a string, which is basically a prompt, right? At one point, a colleague of mine named it tool, though that is not an official name.

And so once you do that, you can take that prompt, and you can put that into a foundation model, call into a query, call into a server, call to a different model, well, however you want. And that really unlocks a lot. That’s really… Once you realize you can do that. Yeah. - That’s awesome. - That’s great. Yeah. Great stuff. All right. We actually have a question, I believe, is not about App Intents and Siri. So. Oh. All right. Yeah. It’s quite wild and crazy.

Can a navigation app run ML inference on the Neural Engine while the screen is locked and audio is playing? Does the new background inference entitlement cover that? Yeah, I think that’s like a Core AI… Yeah. I think that’s a very specific question to the developer’s use case. I would say we’d want to check the background policies for the Neural Engine to comment on that. We’d have to basically look into this more.

We could follow up in a separate thread. Yeah, it sounds great. Great question. So please, discussions, come bring your question there. And we’re happy to follow up and get you a more detailed response. So thank you for that question. All right. Back to skills here. You keep mentioning skills. Is that SKILL.md?

Do we just add this to the app for it to be in context, same as the tool intent for an app? How do I reference this tool intent in the text so that Siri matches it up? Yeah, I’ll take this one. Really good question. So in Xcode, you go File and then create a new resource.

And then in that list of the resources that Xcode provides, a list of the templates, you should be able to select the new skills bundle. And that skills bundle is basically a simple, very simple structure, where in the Skills subfolder, you list all your skills and provide the SKILL.md file. That SKILL.md file has a YAML front matter and a Markdown body. What was the second part of the question? I think that… Let’s see. Sorry here.

Yeah, I think I moved on, unfortunately, from that question. So, Michael, the thing you mentioned at the start too, was that in Xcode, there’s now a skill to create those skills. Is that right? Yes. There are coding skills which you should definitely try out. That will basically take a draft of this for you. You can ask them to take your app from zero Siri adoption to fully Siri-ified, and we’ve tried this several times. It works incredibly well. It’s very, very cool and exciting.

If you are using the Xcode assistant, you will just get them. So nothing special you need to do. Just ask. But if you are using Claude or Codex or whatever other coding agent in the command line separately from Xcode, you do need to grab and install those skills. I can’t remember the name now, but there’s an xcrun command line that you can find, which will dump those skills into the correct directory for your coding agent, and it’ll just pick it up.

Yeah, I would actually add on that, to bringing in kind of the Claude skills we have brought into Core AI models. So back to the previous question, like, oh, specifically about the hardware there, there is some documentation, like skills we’ve put out as well, to basically help you, like, identically target a specific hardware in terms of the specific layouts of the tensors. So I think these are all toolings we’ve brought into our stacks.

Great question. I remember the second part of the question. The question was, how do you reference tools in that skill? That’s a great question. So if your intent is CreateBlogPostIntent, don’t reference it like that. Use natural language. So reference it as a create blog post tool. And don’t use the Swift name in that skill file. Excellent, excellent. All right. Next question here. All right. If two apps are open on an iPhone Duo, which will be used as the target for requests to be handled by tool intents?

The last used, or both? What if the two apps have been joined together explicitly by the user? Yeah. I think one thing to mention here is that a foreground is not, like, an overriding signal. So if you have multiple foreground apps, we’re considering both apps when we are deciding which tool to use.

Interesting. Excellent. Thank you for that. Next question. Here’s an interesting one. What protections does Siri add when an app’s own actions change data? So I think what we’d recommend, feel free to add, folks, but I think what we’d recommend right now is you… If you have an action that is potentially disruptive or changes data in an irrevocable way, right?

Or does something with a physical impact on the world, like opening a garage door, say. We highly recommend that you confirm in that action’s perform method that you want… That the user wants to do this. You can write the language for that, and there’s existing APIs, requestConfirmation, that you can call in the perform method to do so.

That’s just, like, one line of code. And then in the UI, when you ask Siri to do it, it will, like… Siri will basically pause, ask the user, get an answer, and then resume execution. Yeah. If you get a yes, and cancel it if you don’t, right? It’s, like, best practices is always, like, authorization happens at the application level logic. You can’t depend on the model itself to guarantee anything. Right?

Yeah. That’s good. Great question. All right. Yet another Siri question. Big surprise, right? When Siri shows my app’s entities as results and someone taps one, what’s supposed to happen? Well, my app is on screen, even though my entity has an open intent, a URL representation, and a Spotlight index entry, and none of them is called.

Interesting. So if this happens on the Siri canvas, so in the Siri app, I would expect that if you tap on it, it’ll open your entity, and that will mean it will run and open it. It will try to find an open intent that takes that input and run it. If it happens in the Dynamic Island, with Siri in the Dynamic Island, the same thing should happen there. If your app is open, I would actually still expect that it probably ought to run your open intent.

If it doesn’t, that’s probably a worthwhile bug to file for us. That sounds like something worthy for Feedback Assistant. Yeah, really good Feedback Assistant one. Great. Thank you for that question. All right. Another question here. I like this one because shortcuts, we know, have always played such a big part in the App Intents ecosystem. So how much should we invest in app shortcut phrases, compared to letting Siri infer intent from entity definitions?

Interesting. So I think at this point, the tool intent capabilities that we’re releasing are capable of doing everything that the app shortcut phrase system can do, just in a much more flexible, natural language kind of way, where you don’t have to craft the explicit words you expect the user to say, and instead, you just tell us what this thing does, and we will take what the user says and map it, right?

So going forward, that’s what I would lean towards, specifically for the Siri interactions of your app shortcuts. Instead of handcrafting the phrases, you can take that same intent that is hooked up to your app shortcut. You can leave the app shortcut there. You can take that same intent and make it a tool intent. Make sure that your entities are indexed and try it out.

Awesome. Yeah, they can be… Anything I’m… They could be a great foundation for some scenarios that you might write for Siri Evaluations. Oh, good idea. Yeah. If you are, you know, trying to port into a tool intent. - Yeah. - Great starting place. Those phrases that you’ve written are not wasted because you can directly turn them into scenarios. If you haven’t played with Siri Evaluations yet, I highly recommend it.

It’s a really interesting, very powerful set of APIs. And it’s great not just for doing kind of your first sets of end-to-end testing of, like, does this work in a golden perfect path, but really for making your integration solid and robust, because you can write many scenarios, including obvious things people say and including not obvious things people say. You can run them many times, and you could tweak your tools and your language to make them increasingly perform better and better.

I’m curious, like, can you also see, for instance, like, what are sets of tool calls that fit better for a specific use case that you’d want? So how granular I should make the tool be. In this, like, Evaluations framework, would you be able to get those answers or iterate?

Yes, I believe so. So the Evaluations framework can assert on sort of whether or not certain tool calls have been made for that given request that you write in the scenario. So if you’re trying to experiment with different tool intent shapes and different combinations, that’s definitely a great way to go. I see. Nice. And it’s very easy to get started with it. We have an Xcode skill for Siri Evaluations, and super easy, just to ask Xcode to write your evals.

Nice. That’s great. Yeah. And speaking of Evaluations framework, we’ve got some really great videos from this year’s WWDC. Really incredible framework for working with Foundation Models, and of course, now with Siri as well. So yeah, definitely something to take a close look at and leverage in your development process here. Okay, next question, a really good one. If someone is developing a new integration for an app that falls under the existing domains, should we adopt those or start fresh with this new setup? I very strongly recommend that you adopt the existing domains.

One thing you notice with the tool intent stuff is, like I mentioned earlier, and like the docs mentioned, there are a bunch of conditions that need to be true for the tool intent to get called. It’s not always in scope, right? Your app has to be named, or there’s four things, but we think that covers a broad swath of actual user kind of functionality. But the way that the schemas are called by Siri is different.

They’re callable at any time, right? Without any app mentions, without any app foregrounding, without even results to seed them from search. And therefore, like, very strongly recommend that if you are, say, a reminders app, you should start with the reminder schema. You can add to it, right? And we are very… If there are any challenges with that, we’re very happy to take your feedback on, kind of, in the area of these apps that want both custom functionality and schematized, but definitely do not skip the schema and kind of use tool intents instead, if you’re solidly in the domain of that schema. There are many reasons why that’s true. We’ve also done a lot of hard work to make sure that schemas work very well in Siri, so you don’t have to write your own kind of skills files or provide any, like, titles or descriptions. We handle this for you.

Yeah. Really great stuff there. All right. We touched on this a little bit earlier, but the question here is, how can we test Siri behavior? I’ve seen in my tests the same type of voice request does not always behave the same. Maybe something has changed in iOS 27.3.

Great. Let’s see. So two areas of testing that you probably want to think about for Siri. One is the App Intents testing framework, which we just released at WWDC. There’s a great talk on it, and there’s some good talks. It’s designed to make sure that the basics of your implementation are right, which means when you call the app, does it do the right thing?

Does it open the thing? Does it delete the thing or create the thing? It can also test queries. It can test Spotlight indexing. It’s really the kind of deterministic non-model part of the testing story. I highly recommend you start with that, because it’s going to be harder to bring up the rest if your foundation is shaky. So test that. Get that into your CI so it doesn’t break.

Once you have that done, you should test with Siri itself, obviously, like, interactively. But you should also try the Siri Evaluations framework. And that is really the basis for, like, these non-deterministic kind of challenges of, like, how often does this work? It works sometimes. Does it work nine times in ten, or one time in ten? Does it work 98 times in 100, or two?

There’s a big difference. But, you know, LLMs are non-deterministic fundamentally. And that’s why Siri Evaluations is built the way it is. It lets you run things multiple times. It aggregates data so that you can understand kind of at a statistical level how reliable this is, rather than just being a yes/no, green or red, you know, kind of answer.

Yeah. And a large part of that is also about taking natural language input from the user, right? Where sometimes it really is just ambiguous about what they really want to do, given the many intents that you might have. So making sure that your evaluation covers that broadly goes a long way, right, to get sort of that statistical signal of how well you’re handling all of those different cases. Yeah, there’s definitely a process to doing these evals, right?

And you kind of start with the golden path of, like, does this work at all? If the user says exactly the right, perfect thing, and exactly the way we want, is it going to work? Yes. Good. Okay. That means something is there. They won’t always say exactly… But that’s only the beginning, right?

Then you broaden those out to be like, what if they say it in an odd way? What if they, you know, say this and not that? Like, all the fuzzy testing you can do. Exactly. Yeah. I would also come in with, like, there’s the performance side as well, right?

Like, tool design isn’t just about what, do you get the right tools, but also, like, I guess, end-to-end model execution. Do you do more round trips if you define your tools more granularly? So I guess that you can also use Xcode Instruments to, like, profile the entire end-to-end latency, right?

Yeah, yeah, definitely. Great stuff. Definitely check out Siri Evaluations, a lot of exciting new testing support there. All right. Next question. Is there a way to use @AppIntent(.tool) on an existing app intent where it can still be used with older OS releases? Instead of using the macro, can we just add a conformance to ToolIntent? Yes. That works. Yeah, you can do that.

It works. Just answer. Answer is yes. Succinct. Yeah. No. Xcode has that feature to see what a macro… Because tool intent, it’s a macro, right? - It’s a macro. - And Xcode has that feature to show what a macro generates. Right. Because really, your macro in Swift is just generally… How they found it.

You click expand the macro. You can see what it generates. You can then… You could, like, do that conditionally on an extension or something with availability. Yeah. Yeah. Awesome. Yeah. Yeah. Good stuff. Great question. Okay. Next one. What’s an App Intents implementation that impressed you, and what made it work well?

I’m curious what separates a clever one from a merely functional one. That’s a good question. I’m sure we all have answers to that one. It’s a curve ball. Yeah. Like, give an example is the question, or… Yeah. What an App Intents implementation that impressed you and made it work well. Not one you wrote yourself, Michael. Yeah. Right. Yeah. Right. I don’t know, should we name third party apps? Is that… I would look at first party apps instead. Yeah. I would name third party by name, but maybe discuss. Yeah. Be my suggestion there.

You think about it? I guess in the first party world, you might want to check out Notes on the new beta. I know some of the folks that worked on that, and they did some really clever things. I think folks have already spelunked, so I can say this. There’s a skill in there which allows Notes to do text editing in a really interesting, clever way. And it works pretty well. And it’s very cool. That’s just one kind of recent example from the Apple side of it.

Yeah. I mean, the thing to highlight there, I think, is that the same APIs and tool intents, all those things that we now give to developers, we also use that for our own apps on iOS, right? So yeah, it’s really opening up how… - We make… - It good, right? Yeah, we do that for Foundation Models framework as well. Like, first party and third party, they adopt. Yeah, exactly. Yeah. Yeah. We, as much as possible, want to, yeah, dogfood our own things, right?

Yeah. For me, I’d say some of the experiences where it really reduces the friction of the user are the great experiences. Like, you’re running through an airport, you’re trying to find what your gate number is, that kind of thing. Your hands are full with, you know, suitcases or your kids.

And, you know, Siri comes to the rescue with Apple, right? So yeah. I can’t say, like, even without naming specific, like, apps that I’ve seen adopted, I’m already seeing folks send around adoption of tool intents, and it’s, like, super cool. And what I was hoping would happen is happening, which is you folks do things that we couldn’t imagine. That happens every year. That’s why this is so exciting. Yeah. So really looking forward to seeing all the cool stuff. Yeah, totally. That’d be really great. All right. We actually have a question on Foundation Models.

Louie. All right. Finally got you going here. I knew I had to come in today. There you go. Foundation model has a relatively small context window of 4,096. Is there a way to access a larger context window, or best practices for running tools that would have data that would exceed this context size in a single request? Is the context window size on PCC public?

Yeah. I mean, there’s a couple questions in there. I think starting with the good news. We increased the context size from 4K to 8K this year on most devices. So that’s the good news for the on-device model. For PCC, yeah, we publicly documented that gives you 32K context size. So a lot more you can do with that.

Now, in terms of if really you want to try something on device that requires even larger than 8K, I think bringing in custom models, potentially with Core AI, but still using the Foundation Models framework, right, as the API, I think, is a really natural way to do all sorts of custom things that you might want to do.

Yeah. To give an example, like, you can easily bring in Gemma 4 now, which supports up to, like, 100K contexts, depending on the hardware and kind of the variant of the model you bring in, and via Foundation Models, which we’ve hooked up with the protocol. You can basically look at what is my use case, what are kind of the tool calls what I want to bring in that would require, like, more context window, and then use one of our existing optimized hardware-optimized recipes to do that.

Yeah, no, that’s great. Good stuff. Okay. Next question. How does Siri determine which app intent to prioritize when multiple apps or system apps offer similar functionality? What factors influence its decision, and how can developers improve their app’s visibility and selection accuracy? So kind of similar to a previous question, but… I think, yeah, I think we sort of answered this in that for schemas, it is based significantly on user behavior.

When I did the talk on Tuesday at WWDC, I worked on a timer app as an example. And, of course, as I was developing it, I initially had to name it. And then I stopped having to name it, because Siri got to the point where I’m like, “Set a timer.” And it was like, “Yep, I’m going to use your app.”

And so something similar to what happened here. One thing developers can do to help is make sure you donate interactions that happen within your app. Because we… I should say, Siri can’t see those, right? They’re opaque to Siri unless you help out. And so by doing so, you essentially teach Siri how people use your app.

And you give Siri that behavioral information that allows it to say, hey, like, this user is constantly creating reminders in my app when they want to create a reminder. You really should consider potentially sending them to an app. Again, that is not a guarantee, but it is a signal that Siri uses and one of the ones that is more under your control.

Excellent. Good. Thank you, Michael. All right. Another Siri question. How can Siri recover from failed interactions? A big problem with Siri is that it will just throw out requests when an app doesn’t support some aspect of the request. I asked the question, the route fails, and Siri offers no recovery. My request is just dropped. - A really good question. - Yeah.

Let’s take that one. Yeah. Of course. So I think with, especially with the new Tool Intent API and integrating with Siri AI, I think when a tool throws an error, the model will try to retry and find another way to succeed. Obviously, if the tool itself is throwing the error with a particular error dialog, we will respect that dialog and tell the user directly.

But one thing that you can try is try to throw errors that maybe can suggest what should happen next, or whether there’s an alternative way to do this. Skills are also a great way, if you’re starting to see that the way that your tools have been exposed to the model, like, Siri is not behaving in the sequence that you expect.

And so I would definitely try out a couple of those avenues first. Maybe two things to add here. Roman, do you want to maybe talk about error throwing in App Intents? Because I don’t know that everyone knows the right ways to do those. Yeah. There is a protocol called CustomLocalizedStringResourceConvertible.

And if you conform your error to that protocol, you should be able to provide the LocalizedStringResource, which represents the user-visible error. And if you throw that error, Siri will actually display that error to the user. I see. And then just one thing about skills in general, you should only use them if you need them, right? Which means you should really try to get your tools right first.

It’s not great if you’re using a skill only because your tool is poorly designed, right, frankly, and the model isn’t doing a good job calling it. You may want to consider, like, iterating on the shape and structure of your tools and on their language before you introduce a skill. But definitely use skills when you need specific things, like coordinating multiple tool calls into a workflow, right?

As an example, Finder has a cleanup feature now, where you can, you know, be looking at a folder and ask it to clean it up. That is built using a skill, because it’s coordinating multiple tool calls in this complicated and opinionated way that is kind of designed by the Finder team.

Interesting. Excellent. Great question. Thank you. All right. Next question. Also on Siri. When Siri says my app doesn’t support that, even though a matching tool intent exists, how can I see which of my tools Siri actually knows about, and how do I force it to pick up my tools again during development? That’s a good question. I hear that a lot.

I don’t know. I don’t think we have a way for developers to dump that, right? I don’t believe so. The Siri framework, I think, only asserts on whether the tool is called at all, not whether or not Siri was looking at that particular tool. And so I think what I would recommend is definitely running the Xcode Siri Integration Advisor first. Let’s see if there’s anything that it flags. Maybe it’s metadata that’s missing. One thing you can also try is, especially in local development, just prompting Siri saying, like, you want that particular tool, to see if we’re even able to call it.

And another thing that you can try is to maybe add some of the search keywords to the intent description. One thing that we see is, like, there may be many different synonyms or different ways that keywords that people tend to use to invoke that particular action. And so you might want to add a couple of those variations there.

One thing I would also do, a little bit of internals, but something that will help you is look at the metadata in your app. If you open your app bundle, you see that there’s a folder in there called Metadata.appintents. And inside there is something called extract.actionsdata. If you open that, it is just a JSON file, and you can just read it.

And so sometimes if there’s a problem with the app where something is not being extracted, or for whatever reason, we’re not able to detect the tool, maybe it’s not being compiled where you thought, right, or something funny like that, or maybe the availability is weird, sometimes you can tell by looking at it there. That is kind of the first thing to check. And Siri Integration Advisor should be able to do the same thing, kind of, right? Because it looks at the metadata itself.

Yeah. Great. Great points, team. I was going to suggest, option two is filing a feedback request. If you have some ideas yourself on how to approach this, we’d love to get your insights on that. So thank you for that question. All right. Next question. I want to replace my old SiriKit Intents… SiriKit, here we go. I love this. And CustomIntentMigratedAppIntent with new App Intents for users on 27.3 and later. How do I make isDiscoverable version dependent and apply the same behavior to migrated SiriKit Intents, so they no longer appear in the Shortcuts app? Interesting. You want to take… A really good question.

For the custom migrated intents, I think… There’s a protocol where you can say it’s DeprecatedAppIntent. It’s not going to appear in the shortcut search. And if the user has the intent in their library, it will show up as, like, this intent is deprecated. And I think there is a way to provide a message when you say… Yeah. Would availability help them here?

Like if they set a lower availability? I’m not quite sure, but… Okay, this… There’s a very specific question. If the advice you’ve gotten so far doesn’t help, a forum post would probably be a good idea, and we can see if we can zero in on that for you. Yeah, that sounds great.

All right. Next question. How do you imagine internal visibility parameters being used? I can see it being useful for passing data between app intents, especially when you mentioned by a skill. Are there other use cases you had in mind? It took me a second to understand what they were talking about, but I think I understand.

Roman, you want to take this? Sure. The skill is a good example. Essentially, you want to stitch two actions together, and one action passes the entity, and another action actually takes the entity. And this is the way to preserve actually the data in that entity. Michael, do you think what would be the other good use case for internal?

I don’t know, do they mean parameters or do they mean properties? - Properties are the… - Entity. They mean properties, right. Okay. Because I don’t think there’s a public way to do internal visibility for intent parameters. I think the entity properties, yeah. So these are really good for, like, widgets, right?

You might have some information that you specifically want available if that entity is used in your widget, but, like, as a parameter, like, for configuration or for interactivity, but you don’t want Siri or Shortcuts or other features to be able to see it. That’s a lot of what that’s for. It’s essentially a way of saying, hey, this is for… This is a field in my entity that is for me communicating with my code.

It is not meant for Shortcuts, Siri, or other system features that discover, write, or just look at the whole thing. Please hide it from there. That’s kind of what that API… But it also works in Shortcuts. So if you have two actions, and again, you pass the entity around between those two actions, you should be able to pass the data and see that you will see as a developer, but users in Shortcuts won’t be able to kind of choose it. Yeah. Right. Yeah.

Excellent. All right. Next question. How do you recommend testing Siri interactions across languages and accents, especially in multilingual markets like Southeast Asia? That’s a good question. That’s a really good question. I don’t know if I have great answers for that at the moment. Does anyone have thoughts? I think all LLMs are capable of using different locales and languages, and they should be able to understand this fairly easily. That’s my opinion.

Yeah, I think we have found, you often… You can localize the language in your app intents, but that’s only necessary if you’re showing them to users, like, if people will see them. Because if it’s really just for a model, models are very multilingual, right? Like, they don’t really have a problem reading something in French or German or what have you.

In terms of accents, I think that’s probably at a bit of a different level, right? That isn’t something you can probably control. That is more like an ASR recognition thing where you probably want to file feedback requests on us if you’re seeing problems with an accent, rather than it being something that you need to fix, right? What’s in your control is just handling different… Writing different scenarios in different languages and with different kinds of phrasing, right?

If you think that even, say, among the English-speaking countries, folks in one country might phrase things differently than the other, you can write scenarios for all three and make sure that your language work, that your tools and skills and everything work well for all of them. Excellent. Great question. Thank you for that question. Okay. Can we dynamically create skills or tools, allowing users to define or customize what’s available?

Can I take it? No, it’s not currently possible. And you need to provide all your skills and tools at the app build time. That is right. There’s lots of very strong kind of safety, security reasons why that’s the case. However, you do have flexibility in how you shape your tools, right?

And often you can get surprisingly dynamic behaviors, even if you cannot literally completely recreate the tools, because tools can take entities. Entities can be indexed and can be dynamically retrieved. There’s a lot you can do, even if you can’t dynamically change the tool shape itself. Excellent. Great question.

All right. Next question. Does Siri AI read tool titles, descriptions, and parameter descriptions in the device’s language or only the app’s development language? Must we localize tool and entity metadata for Siri to pick the right tool in other languages? Does description quality per language affect tool selection? Wow. A lot of good questions here.

Yeah, we do pull the intent title and description in the system language of the device. And so as folks have mentioned here, the LLM is, you know, surprisingly great with many languages. And so we are able to handle that, I believe. There was a second part to that question. Yeah. So must we localize tool and entity metadata for Siri to pick up the right tool in other languages?

Oh, I see. It is a great idea to localize, especially your intent title and description, because one of the things that we look at when trying to decide which tool to select is user’s request. And so if the user’s request is in a given language, it’s great if your intent title and description are also in that matching language. Yeah. I mean, in general, I think, like, more broadly, right?

Like, with LLMs and tool calls, if we use, like, chat templates for some of the open source models in terms of deciding what tool calls to grab. So if, for instance, it’s, like, in a different language, and you put it in the chat template, then it’s more likely for the model to behave correctly, right? Yeah. Nice. Yeah. And also, you need to be mindful that your app intent can show up in multiple surfaces. So for example, if your tool intent works in Siri, you should also see what it looks like in Shortcuts. - Absolutely. - Yeah.

And there, the text is user-visible, right? So there’s a strong incentive there to localize. Visible and localized. Yes. Yeah. That’s great. Good question, though. Wow. I’m loving these questions. Giving us a workout here, which is what we love. Can we apply ToolIntent slash @AppIntent(.tool) to existing app intents that run on older OS versions?

I think we took that one, right? Yeah. It’s possible. You can just conform to… Check the video on demand. But yeah, you can. Yeah. Sounds good. Okay. I saw somewhere that Siri will be able to understand and pre-slot multiple app intent parameters at once, instead of just one at a time, like the current behavior with app shortcuts. How does one opt in to this behavior?

Pre-slot multiple app intents. Yeah, I’m not sure how to parse, but yeah, just when you use a tool intent, that is how it will work, right? Siri is no longer limited, like the app shortcuts system is, where with an app shortcut, you can only have one parameter, and it needs to be kind of enumerable. We need to know all the options ahead of time. That’s no longer the case. So Siri can fill in as many parameters, multiple parameters, and can fill them in generatively, right? It can write a string. It can do anything it needs to there.

Obviously, tons and tons of parameters, especially tons of required parameters will probably start to not work so well, which is why something like the Siri Integration Advisor that you can run in Xcode will give you feedback if your tool has a lot of parameters. That doesn’t mean that it definitely won’t work. But when you have a lot, it starts to be like, it may not work well.

So that’s why we’re giving you that advice. But yeah, you’re definitely no longer limited to one, and you’re definitely no longer limited to specific list for that parameter, and nor do you have to opt in, especially, to this behavior. You just make use a tool intent, and that’s the behavior you will get. Yeah. Is that because of kind of, like, the inference workload as well? So if you have, like, way too many parameters, you have to pack that all into the context before you even can do a model inference. Is that part of the consideration?

Definitely with the many parameters, that’s one of the concerns I see. Yeah, I see. And if you have, by the way, a group of many parameters that you are repeatedly using, right? That’s where generable entity can help, or one of the areas where it can help. So pack those up into a generable entity, have that be one parameter, and then Siri will make the entity for you. And that’s really useful if that same set shows up in, like, five different places, so you don’t have to keep repeating.

Awesome. Really good stuff here. All right. Next question. You can say the app should be named in the Siri request. Can we use synonyms if our app name is difficult to say in foreign languages, for instance, or difficult to identify by Siri? Currently, we don’t support the app’s synonyms for tool intents. So not in 27.3.

That would be a really good feedback request. Yeah, I was just going to say. So, yeah, great question, though. Awesome. Look forward to seeing your request on that. All right. Next question on onscreen awareness. When using onscreen awareness, Siri used to not be able to understand custom non-schema entities, even though they were indexed. Is tool calling the route to improve understanding what’s on the screen?

Interesting. So I think as part of this preview, Leslie, correct me if I’m wrong, that should change. You don’t need tool calling for it, just by having an indexed entity, right? Siri will be able to understand it, find it in Spotlight, and understand it if it’s on screen.

All you need to do is adopt it, run it on the new beta, recompile against the new SDK, and run your app either ad hoc, locally in Xcode, or with the internal TestFlight. That’s it. That’s exactly right. There’s no more of those schematized entity restrictions for onscreen and Spotlight. That’s great.

All right, next question. Can we discuss how many app intents can be included in an app, and what the relationship is between intent tools and how a user interacts with these systems in Siri AI? Ooh. You should maybe take that first. But yeah. So I believe we do have a maximum limit on the number of tool intents that your app can ship. I believe that’s 40 right now is where we landed. Yeah. And what we do recommend is that you target your key capabilities of your app first and iterate on those, the shape of those tools, run them through the framework, as we’ve been saying before.

You start expanding to other capabilities in your app, since you do only have 40. So it does pay to prioritize there. As for the relationship, in our system, tools are intents, intents are tools. They’re one and the same. When a user has an interaction with Siri, the assistant, Siri, may run many tools slash intents in sequence to achieve a complex goal, right?

I see. So, like, the more tools you might define, Siri might try and do more things, which kind of can possibly make the inference a lot longer, like time to first token. Yeah, there’s that, with the number of tool calls that you chain. And also just having that number of tools shown to the model definitely is going to, in some cases, may confuse the model, especially if they overlap, or if there’s something about the description that is kind of vague about why we would want to call that particular tool.

So that’s why I say, like, it’s definitely better to go with an iterative process, so that it’s, like, clear to the model first, and you’re confident that the model knows how to use that tool before you start adding new ones. Yeah, that’s part of why Siri Evaluations is so important, right?

Because it is designed for you to be able to iterate not just on the language, although you will certainly do that, but on the entire shape of your toolset, right? You may start out with 50 things, and then you may be like, actually, I can reduce this to three. And then you might be like, well, that’s reducing too much.

I actually need five. And maybe you ship with ten, and that’ll be the thing that works, right? Like, you have to try these things, and writing scenarios with Siri Evaluations and trying it is a really good way to get that done. Yeah. Totally agree. All right. Next question.

How do you suggest we educate the user about the availability of tool intents or app schema domains for our app? Very good question. Really good question. We hear a lot of these kinds of questions in our workshops. Yeah, that’s a good one. I think that is one of the challenges of voice interfaces, right? It always has been. It’s no different now. I think we have seen apps do onboarding of various kinds, saying, hey, we now have the ability to X. Here’s a couple sample phrases that you might say.

That’s something you should consider trying out. If you run into problems with doing that, like you’re missing any information, please let us know. That’s a good, again, good feedback request, but that’s kind of what I’ve seen most of the third party apps do. You can also ask Siri, right, what an app can do. You ask Siri, like, “What can the Notes app do?” And it will tell you, I believe. Yeah, that’s 27.3. You can just ask Siri. Yeah.

That’s great. Great question. All right. We have a Core AI question, Karina. I love it. Nice. Okay. Core AI seems like a big opportunity for the future. In our current testing, .aimodel and .aimodelc files appear to be compiled on device after download before use. That requires 40GB of free space for a 4GB model and takes minutes.

Is there a way to precompile architecture slices to prevent this? Yeah, that’s a really good question, actually. A lot of the kind of bottleneck we see with shipping large language models is that the kind of compilation process takes a lot longer than previous, like, classical machine learning models. So what you can do is ahead-of-time compilation. That’s actually a new feature we have with 27.0 release. So you can basically specialize for the hardware ahead of time, even before you bring it to your device.

So I know we’re running out of time, but there’s a lot of documentation. We have a WWDC video where we share more about that. But yes, exactly. Like, there is a lot that you can take advantage of in Core AI. Yeah, plus one on the great videos on Core AI from this year’s WWDC. Really gets into the nuts and bolts of Core AI. So really exciting stuff. And like you said, Karina, unfortunately, folks, we are almost out of time. I know time flies when you’re having fun. A lot of great questions.

But again, you know, we’re thankful that you’ve all joined us today. We hope that you found this to be really helpful. I also want to special thanks to our panelists. Thank you. Your very busy schedules, as well as all the folks working hard behind the scenes to make this happen today.

So as we mentioned earlier, if you didn’t get to your questions, please visit us at the developer forums at developer.apple.com/forums, where we can continue the discussion. And of course, Feedback Assistant, file any bugs or feature requests. We really do appreciate your bug reports and feedback. Look forward to seeing what you develop with the new developer preview. Have an awesome day!