Video hosted by Apple at devstreaming-cdn.apple.com

Configure player

Close

WWDC Index does not host video files

If you have access to video files, you can configure a URL pattern to be used in a video player.

URL pattern

preview

Use any of these variables in your URL pattern, the pattern is stored in your browsers' local storage.

$id
ID of session: meet-with-apple-291
$eventId
ID of event: meet-with-apple
$eventContentId
ID of session without event part: 291
$eventShortId
Shortened ID of event: meet-with-apple
$year
Year of session: 2026
$extension
Extension of original filename: mp4
$filenameAlmostEvery
Filename from "(Almost) Every..." gist: ...

Meet with Apple • Session 291

Apple Intelligence Group Lab

2026-11-05 • 1:01:41

Connect with Apple engineers online to ask questions, get advice, and follow the discussion about App Intents and Siri AI, the Foundation Models framework, and more.

Open in Apple Developer site

Transcript

This transcript was generated using Whisper, it may have transcription errors.

Welcome to the Apple Intelligence Group Lab. I’m Shashank. I’m part of the evangelism team here at Apple, and I’m joined by a panel of experts from the AI and machine learning framework teams. Let’s start off with a quick round of introductions. Matt? Hi, I’m Matt. I’m part of the Foundation Models Framework team. Hi, I’m Karina. I’m a part of the Core AI team. I’m Kevin, part of the Siri AI team. Hi, I’m Justin, an engineer on App Intents. Hi, I’m Michael. I run the App Intents core team.

Thank you. Thanks, everyone. In addition to those you see on screen, there’s a team behind the scenes helping with the triage of all your inbound questions, so please keep them coming. If you have code-specific questions or somehow we can’t get your questions today, we highly encourage you to head over to the developer forums and share them there. If you do have a bug or feature requests, we’d like to encourage you to head over to feedbackassistant.apple.com, so we can focus today’s group lab on questions that helps the broader audience.

All right. So we just announced the new Siri AI Developer Preview. Michael, can you tell us more about what we launched and what it means for developers who are already using App Intents today? Sure. So we are so glad all of you could join us today. We have been thrilled to see all the adoption of our schemas-based App Intents APIs from this year that we designed for specific types of apps. With the Developer Preview that we announced this week, we are taking the first step in expanding Siri AI’s capabilities every kind of app. Yes, even yours.

Using the new tool intent API, your existing app intents can become tools that Siri calls directly when people make requests. Siri can find your app’s tools, if your app’s in the foreground, or if it’s named. You can also create custom app scales for Siri, bundles with markdown files that help Siri master even complex app interactions. Siri can now also find the content you have indexed into Spotlight, even if it doesn’t conform to a schema. If you’re not already using the existing index entity APIs, now’s the time to start.

To help you refine your Siri AI integration, we are providing a powerful set of Xcode features for you. The Siri Integration Advisor gives you feedback on the shapes and descriptions of your tools and skills. And the Siri Evaluations Framework lets you write scenarios to test and tune your tools and skills so they work no matter how people ask for them. Xcode now even includes a set of coding agent skills to take your app from no Siri to full Siri integration in no time at all.

We would love for you to try all of this out in the iOS and macOS 27.3 betas. Since we are still refining the APIs, we’re looking forward to getting your feedback on them. Right now, you can test this stuff at desk and distribute using internal test flight, with shipping to the App Store coming this spring.

Oh, and one last thing. You have told us that you want more control over the way your entities look in Siri AI. We’re introducing a new entity snippet API, so the visual representations of your content can reflect the unique style of your app. You’re going to be able to ship these snippets to the App Store when the 27.3 OSs are released. And now, let’s get to some questions. ANIL ANANTHASWAMY: Thank you, Michael.

Thank you for all the questions, everyone. So let’s get started. Here’s the first one. Does the number of app intents exposed to Siri AI by my app hurt how well it can pick them? What’s the recommended maximum? Kevin, want to go for it? Yeah, for sure. So Siri will basically use the app intent that makes the most sense with the request. The limit of app intents is 40.

We do recommend you to really think about the tool shape to be sort of like, of course, make sense with your app, but also be sort of like a small unit that would make sense of like if you’re an app that sells products, matching, searching a product, adding it into your cart. It’s sort of like the logical action that a user would take. Now, to come to the second part of the question, would that hurt, we recommend to sort of like have the least amount of tool to sustain your user actions.

One thing to add here, definitely try out the Siri Evaluations Framework. It is not only just, I think, an extremely cool API and kind of design, but it’s very helpful to you because it lets you refine the way this stuff works. So what often happens is you might come up with a way to break your tools down, like Kevin said, and you may find that you hit it right in the first time or you may find that you didn’t, right? Siri Evaluations lets you write a bunch of scenarios, try them out, potentially repeatedly, and see how well it works and then reshape not only like the language on your tools, but the actual count and number and organization of your tools, right? So that it gets better.

It is a very powerful ability to be able to like eval Siri in this way. That’s an excellent point because you can really sort of push the boundaries of how your app and your tools are going to be used and iterate on those shapes of the tools. Exactly. Be prepared to iterate. Don’t assume you’ll get it right the first time, but you will get it right with a little bit of work. So as few as you need, but test them with evaluations to make sure that’s exactly perfect. Thank you.

All right, next question. Is there a particular reason slash intention behind app intents not being opted into Siri AI by default, especially those that are registered as an app shortcut as clearly intended to be used with Siri by the developer? Why not opt them in by default? Okay, so this is a great question, actually.

A lot of it is that you need to test your app intents with Siri, right? This is, I was just talking about Siri evaluations. You need to see that it actually works, right? If we were to say, just kind of do something automatically for you, what would happen is it may work sometimes in some golden path cases, but the minute someone said anything slightly different than the perfect way of saying it, it may not work, right? We really want you to take the opportunity to do that evaluation.

Try writing scenarios for different kinds of phrases, both the easy golden path ones, as we call them, and the off the beaten path, like, oh, this is an odd way to phrase it, but it should still work. We want you to try both. That’s why this process of opting in with two intents is an explicit one. And at the end of the day, it’s also to make sure that users have a high quality experience with Siri. Exactly. In the end, that’s the goal.

Because neither us nor you would want to ship something that just doesn’t work well for people. I mean, in the end, you’re kind of now having to map from natural language to specific tools, right? And that is something that we’re adding. That’s a new feature this time. Yeah. The opting in is fairly easy, right? You just, it’s a single line of code.

Oh, it’s super easy. I mean, we have a macro for it. Right. Really, if you already had an app intent, it’s as easy as writing one line of code. One line of code. So it’s easy to now just evaluate, decide if you need it. If you do, just add it or remove it. Of course, that doesn’t mean it would also just work out of the gate. You have to test it, like Michael has said.

There’s also a great tool in Xcode to go ahead and help you integrate. So it’s the Siri Integration Advisor. Oh, nice. And that way, we’re making sure that everything is in line with our standards and making sure that you have enough detail so that the model can answer those requests. And also, if you’re using coding agents, make sure that you try out the new Xcode skills that we’ve added that really guide you through this process. If you are using Xcodes, a coding agent interface within the Xcode app, they’re just going to work.

If you’re using Quad or Codex or something out in your terminal, that’s fine too. There’s an XCRUN command line you can use to grab them from Xcode, install them into the right place, so your agent picks them up. They make this a pretty easy process. So no excuses to not do evaluations, right? All the help that you need. I actually wonder, is the evaluations also connected with some of the Xcode profiling, where you can not only look at the tool calls in terms of their quality, but also how do they affect the end-to-end latency of the end-user experience?

Yeah, so the Siri evaluation really focuses on the trajectory that Siri will take. So you can make sure that you have assertion of the tool that you expect to be called, on the scale of skill, that your skill is indeed used for the turns. It’s not specifically looking at the performance of the latency. I see. But any failure or retries would also be documented as well, right? That would be super helpful. Awesome. All right, let’s move on. There are a lot of questions. Thank you for sending in your questions. Please keep them coming. We love questions.

Yeah, it is so cool, by the way, how quickly we’re getting great questions and how far you guys have already gone with all of this, given how new it is. Really awesome. Awesome. Let’s move on to the next one. Can app intents or Siri AI trigger an action in an app that’s in the background without bringing it to the front?

What response time can we realistically expect? Can we take that? Yeah. So the short answer is yes. You can use your tool intent by when the user essentially mentioned the app. That’s one way. If the app is on screen is another way or if there is a live activity showing.

There is also another way where if the user triggers a search and your app entity is in this search, you can expect then your tool related to your app to be called for the second part of the question you should expect like normal latency into calling you know calling the the intent i don’t know if you could add some more color on like the app is not running so there might be a little overhead of launching yeah yeah there’s app launch but uh one one thing that’s cool is app intense support being in your app or being in an extension um and then you could even put them in a framework and put them in both. And bringing up an extension is usually faster.

So if it’s a background action and your goal is like, oh, I want the quickest possible way to check off the task, or something like that, an extension is great. And then if you put it in a framework, we have some smart policy that actually says, oh, whatever is already running, we’ll use that first. And if it’s not running, we’ll prefer the extension, because it’s probably faster to spin up. We have all kinds of things there to make this perform well. FRANCOIS CHOLLET: It’s actually a good motivation to add up an extension. Because then you get sort of snappier experience.

  • We will get, yeah. For background things, that’s a snap way to get the fastest performance.
  • Good question, thank you. This is an interesting one. This is, if an iPhone app is the remote for a Mac app, where should app intents live? iPhone, Mac, or both? Can a request on one device run an action on the other?

SETH VARGO: Interesting. So no, right now there is not an automatic way to say ask for an action on the iPhone and have it automatically invoked in your Mac app. You probably want to put those app intents in both native apps so that they can run correctly on both platforms natively. And then it will sync from there since you have a pair of apps. SETH VARGO: Awesome. Thanks, Michael.

Next question. Is there a particular reason, intention behind… Oh, we did this, the app shortcuts. Okay. Can we expect Siri via these APIs to capture or inform multiple variables in one request? For example, I lifted 350 pounds, eight reps, or log 18 ounces of water without saying fluid ounces.

You want to do this one, Jess? Yeah. I mean, I believe the answer is yes. You can really take advantage of skills, which are new, and they should do a good job of kind of making sure that all your requests are fulfilled. Yeah, I think like, Kevin, if you’re going to correct me here.

So, yeah, app shortcuts, which we were asked about before, are kind of limited, right? In this that they can only take one parameter and you have to have like an enumerated set, right? That needs to be a fixed set of possible values, which we then expand out. These new tool intents don’t have any of those limitations.

They can fill many parameters. They can fill freeform parameters that are like a string. People have even done natural language tools where they make a tool intent that takes basically a prompt, which itself internally works with foundation models. Hi, or Cori. I mentioned this yesterday, too, where you can really bring in your own implementation for the freeform. That’s right. So all of those limitations of app shortcuts are gone. We’re very excited for you folks to port over your app shortcuts over to 2 and 10.

And then, what was the other part of the question? I thought there was something else. Multiple variables, and can it infer sort of the units, for example? Yeah, yeah, it should be quite good at that. Again, you have control over your titles and descriptions of your parameters, and it can guide the model if you do the right thing, and it will. Is this somewhat related to the generable entities at all? Yeah, so maybe do you want to talk about, Does anyone want to talk about generable entity and how to use it? I can talk about what it is.

It’s a transient app entity. We have a new API. We have a new trait on the macro for app entities. And that would make it, you would mark it as generable, which would allow Siri to go ahead and make these on the fly. So if you had, for example, like a flight search, the user might say, oh, find me a flight from San Francisco to Taiwan. and for this date, and that should be able to generate that search request.

Yeah, they’re really cool, especially when you have a set of parameters that you reuse in multiple places, where you have multiple cases of that same set going into an intent or going out of an intent. Wrap those up in a generable entity, and then it will make all this much more readable and clear in your code. I’ll do a quick plug to Foundation for those who are familiar with the framework. So there’s generables there. When you want to generate structured outputs, your marketer’s generable. This is very similar.

So if you want an int, you get an int, right? If you want a string with specific guidance, you provide it. Very powerful. Awesome. All right, moving on. How does Siri pick which app handles a generic ask like get me food? Can users set a default? And how much do donated intents matter?

Yeah, so donated intents are really important for schematized intents. Like if you, say, have a create reminder intent, right, and there might be four reminder apps that could do that on the system. A lot of the way that Siri determines, if you don’t name one, right, if you name one, we’ll use it.

But a lot of the way Siri determines if you don’t name one, which one to call is actually based on user behavior. And that’s tracked by donation. So donations are like we have a transcript of these things. And if Siri calls your intent, the transcript will know it. We’ll obviously automatically record that.

But if the user makes a reminder in your app, we can’t know that. And so you very much, you need to adopt the intent donation manager and related APIs to kind of tell us, like, hey, this action that is equivalent to this intent happened in my UI by the user tapping around. And when you do that, we use that information as a signal to determine like, oh, that probably means that this is the app we should use when the user asks to create a reminder.

For custom, for these tool intents, the situation is a little bit different where because of the gates that Kevin talked about, right, those gates are going to determine what runs, right? So if you are looking at your favorite food ordering app and you say, I want to order a burger, it will call that app first.

If you are not looking at an app, unless you found a search result, which probably wouldn’t be the case for food ordering, you have to name the app by name. And so that avoids a lot of these ambiguities. Sounds like these are perfect examples for Siri evaluation and trying out all these different ways. But it sounds like the context is what matters the most for these.

And that’s probably the best sort of flow to see how things work. Yeah. Awesome. All right. Next question. How do the usage limits work for Siri evaluations and for app intents we expose to Siri AI? Are the usage limits tied to the developer or the individual user Apple accounts? Thank you. - Ooh, anyone wanna take that?

I don’t know if we have the right expert here. That might be a good forum question, actually, for folks to take. Yeah, let’s do that. Awesome. All right. Thank you. Thank you for the question. And, yeah, we can get back to you in the forums. Please share these there so multiple people can benefit from our responses. Thank you.

Moving on, is there a limit to how many tools in Tense City AI can chain to perform a more complex workflow? Can this work across intents from different apps? Oh, that’s a good question. Two very good questions, which sort of relates to how skills work. So the answer is yes. Siri can do up to 32 calls within a skill. So I think there’s a couple of advice there.

Back to the first question on making sure that you have fine-grained or the good sort of fine-grained tools, I guess, is one thing. The other is if your skill actually operates on a number of entities, we do recommend to have intents that are batchable. So basically, like, I don’t know, if you’re, you know, operating on a set of photos and one of your tools is favorite a photo, obviously, you know, if you have now like 40 photos to favorite, you’re going to go behind those limits.

Right. So I think one really good practice is to actually have batch support. This is not only important in the context of skills, but also just serial orchestrating would be faster because it’s one turn of a single tool call versus N. Don’t make the planner loop. You should loop, if possible, within your intent, which means taking a list parameter.

And the most efficient way to do that is to use a type we have called entity collection, which allows us, instead of sending you a list of entities, If you don’t need those, if you really just need the IDs so that you can go ahead and make a ref server call or hit the database for some giant batch query, use entity collection because that makes it even more efficient.

Because then we don’t have to call your query beforehand to get you the whole thing. And that way, Siri can handle large quantities of entities. For a specific workflow that the developer wants, is skills the right way to tell Siri, here are three tools you use in this sequence. Is that the… So the answer is nuanced because Siri is very capable at chaining tool call on its own. By itself.

Exactly. So I think what is really important is to, you know, back to the Siri eval, is to sort of push the boundaries on how users are going to use the app and see when there is, you know, either trouble or inconsistencies into, you know, what sort of tool calls are made. Skills are really good to teach Siri about your particular domain.

And so there might be like a certain order of tools that would work better or like exchanges of information that make sense in a particular order. So that’s where like skills are very useful. Get the ingredients first, then cook. So those would be the sequence to it. But I think what’s important is to not default to a skill and really sort of like push the boundaries of your tool. Actually, having to use a skill could be an indication that the tool can be sort of rethought. Can be improved. Yeah.

And like first iterate on like the actual description of your tool. And the shapes. And the shape, the parameters, et cetera, so that it’s really well understood on its own. Yes. It’s not ambiguous just by itself. Exactly. A skill is not required to understand that. Exactly. And keeping… Oh, go ahead.

Yesterday we were chatting about how in the end skills are kind of instructions for the model, but it’s not guaranteeing anything. So you have to think about what is that kind of deterministic layer for the model, but also like things that you can help the model guide it. That’s right. Skills are, yeah, definitely don’t default to skills.

Don’t use skills to replace having well thought out tools. Keep in mind that tools can, Siri might hit on a tool without your skill or with your skill. So that means that tools should stand independently. Not just, oh, this is going to totally break unless you use it in the exact way the skill says.

And also, don’t use skills for very deterministic processes, right? If you’re in a situation where this must 100.0% of the time, first call A, then B, then C, then D, probably not necessarily the right place to use a skill. You may actually want to wrap those up in a single larger tool to make sure it’s always done.

But if it’s more like, well, interview the user first, maybe ask for this first, and then that, and then combine those, and then maybe call other tool that sounds like a like a skill uh kind of flow to me i was curious with skills like do you have a recommendation of how to write that in a certain way like is there how to write like the tool references yeah exactly like the skill itself yeah is there certain things that you yeah yeah so um we found that it works well to reference tools in skills informally right so like don’t don’t try to figure out how we mingle your tool name right uh don’t name it exactly like the app Don’t say, call the get car app intent. Instead say, call the get car tool or get the car. And Siri will figure it out from there. Got it. I think the key is to use natural language. It’s easy to write too. This is what we do.

It’s almost like you’re instructing an admin or something like that. Think of it informally and not as code for a computer, but as code for something that understands language. Got it. I think one last important thing is, think about the description of your tool when you are actually writing your skill, ’cause all that should be interconnected. - Got it.

  • And actually, the way we show skills to Siri is working exactly how we show tool intents. Whenever the app is in context, or there is search results, et cetera.
  • The same four rules, right?
  • Exactly, so you’re gonna have your tools sort of sitting alongside the skill.
  • Got it.
  • Makes sense.
  • Thanks.
  • No, good discussion. I think the key thing I took away was, skills is not a gating thing. the Siri can use skill or call tools directly. So don’t treat it as like gospel instructions for the app, but treat it as a guidance. Yeah, let’s put it this way. You may not need a skill.

It’s very, very possible that you can do everything you want and not need a skill. Awesome. All right, next question. Why does an open intent require an app entity target? I wanted to use a union value, but instead I had to create an open intent for each type.

Yeah, so I think probably that’s maybe just a feedback request for us in terms of, like, I think right now that’s how it’s structured, where you would write separate intents. I can see how it could be useful where if all those implementations are somehow really similar and share a lot of code, it could be useful to, like, allow you to wrap those up in a union value and then go ahead and call it.

Do you know, would the system.open schema work for that? I’m not sure, but again, these are the types of things that I’m sure developers have tried, and this is why the question’s asked. Yeah, yeah. So maybe there’s an implication that you’re reading. I would definitely file the feedback. Definitely an interesting problem that we want to solve. Yeah, try out using system.open directly. If that doesn’t work, definitely file a feedback.

Here’s a secret. We love feedback. Please file them. We read all. We look at them. Yeah. Thank you. Next question. When Siri calls a custom tool, what leaves the device? The tool’s definitions, its arguments, its results, can it be limited to on-device only? I think that’s you again, Kevin. Yeah.

So with tool intents, that’s a feature that only works with Siri on the server. So that uses the PCC, private cloud compute servers, where essentially the request is encrypted. Part of the request is indeed the system prompt, the details of the tool, as well as the parameters. But at no point, Apple can actually access that. It’s encrypted end-to-end. I will point out we do have a technical paper, I believe, on the infrastructure of PCC and privacy guarantees, verifiable privacy promise, I think we call it. So, yeah, feel free to check that out.

Maybe like one interesting thing here is that, so if your device has no internet access, Siri can still work with the on-device model. But in this context, tool intents are not supported. I see. Good to know. - So the tool resolution is where it reaches PCC, but in some cases, some of the tasks can be done on device. All right. Awesome. Let’s move on. Here’s another question. Are there any limitations on the older hardware, such as iPhone 15 Pro or 16 lineup, in terms of capabilities with the new app skill? Or can we expect functionally same behavior aside from device speed?

Do you want to take that from the App Intent? On the App Intent side, no, there aren’t any limitations. Yeah, I think it’s any Apple intelligence device. Which is 15 Pro and later. And if it’s hitting PCC, I think it’s virtually indistinguishable. But I will mention, I guess, from the Foundation Model Framework side of things, models have differed between these devices. The on-device models, right? Yes, on-device models. Yeah, specifically context size.

We have some larger context size of 8K versus 4K. Yeah, exactly. Yeah, but PCC available at every Apple television. Yeah, and recent devices have a different model. And we have a doc page now as of this week. Yes, exactly. It’s a documentation page describing which devices support which type of foundation models. And when it hits the on-device model, so there might be some differences there.

Sounds good. Thank you. Thank you. Let’s move on. What’s the best way to test our Siri intents at scale using an LLM as a judge? And how often we should retest as Siri’s model evolves? Ooh, I love this. I know Evaluations Framework has an LLM as a judge capability. Yeah, the Siri. How does it work on the Siri evaluation side of things?

You can think of Siri evaluation more as almost like unit test evaluation. So there is no sort of concept of LLM as a judge, but I think it’s good feedback. I’m curious to… Yeah, the evaluations framework does have that feature where you can try different LLMs as a judge with that.

But yeah. Somewhat of a key difference there is you control the session, you control what goes into the model, you control what comes out of the model, so you can use an LLM as a judge, including PCC, if you want, to judge it. But on Siri, it’s a little more controlled workflow, right? So good question.

Actually, before we leave testing, it’s worth kind of talking about the layering a little bit. So we talked about Siri evaluations, right? And what it’s for, I think quite extensively already. But kind of before you get there, you probably want to use Appentence testing. Do you want to talk about what that does and why? Because I think that layering is important and actually getting it right will help developers be successful. Yeah, absolutely.

I think app and test testing would kind of be the first step before you even go into testing your tools. You know, there is a set of determinism that you want to make sure you test. And app and test testing is really able to test all the things app intents, your queries, your intents, and so on. And there’s a really great WWDC presentation on this that my coworker at Venkatesh had done.

It just really goes in and deep dives into that if you are really interested in trying to test that with that and test testing It’s really good. It’s um, you can you can not only can you test your intents in your queries? You can test stuff like did I index into spotlight correctly?

Did I tag the entity on the view right you can get all this ratio? It’s basically like almost like unit testing your app intense some implementation Before you get to Siri, right and the nice thing about getting that right is then when you start testing with Siri you’re not also going to be dealing with random bugs in your perform method. Where I’m like, oh, I forgot to index that. Oh, I didn’t.

That layering really helps, I think, at that rate. I mean, it’s essentially like model reasoning of the tool calls, but then also like the tool calls and what parameters they have to be hot. So you want to compartmentalize them. And if you want to check if Siri actually is reading the skill and stuff like that, that happens on the Siri valuation side. Is the intent even working? As I expect, that’s on the app intent testing. Exactly.

If there’s a bug in your app intents, it might be hard to find if you were trying with just the Siri evaluation. Yeah, that’s the thing. Because you’re dealing with a non-deterministic model that’s going to, like, there’s just a lot of factors in there. Yeah, I would add that if you’re trying anything with, like, tool calling with foundation models and query as well, those are all the testing you have to do because it’s not deterministic if the model’s able to call the right tool versus, like, what the tool is actually able to execute. Yeah. Yeah, check if the tool is working first. Exactly. Then if the model can call it. Awesome. All right, thank you.

Moving on, next question. The context window of 4096 tokens per language model session is pretty constraining. Is there plans to expand it? I think the question is right for you, Matt. Yeah, I mean, as we mentioned earlier, there’s 8K with some newer devices. But you can get a lot done with that. I think also Core AI. We have the language model protocol that came out in 27.0.

And I don’t know, but probably you can talk more about Core AI with its other models that you can provide. Yeah, yeah. So exactly, as Matt said, if you need support for longer context, you can definitely bring a custom model. So to give an example, like on Core AI models, which we mentioned yesterday as well, you can now bring in like Gemma 4, which supports up to like 100k context, for instance. for E2B, E4B for Gemma 4.

So depending on your use case, if there’s something that’s not even supported, you can file an issue and even bring in your own model through Core AI. Awesome. So there is the new upgraded context window. So it’s large enough. And I will also mention that we did publish an article where we discuss how to manage context window. So it’s in our tech notes.

Maybe there are things you can do to chunk your data or use better use of generables where you provide structure guidance through generables, not through prompting. All of these can be used to better take care of. Yeah, we also have in the foundation models utilities open source package some modifiers when using like dynamic profiles.

And we also talk in the DubDub talks about like different variations of trimming the transcript, like removing tool calls, dual outputs, that sort of thing. Yeah. Yeah. I mean, in general, for like open source models, you bring in through foundation models, we represent the tool calls in kind of the chat template. So if you like reduce some of the tool calls, you can get a lot more context for the actual useful information.

So there’s a lot of like hacks you can do for sure. It’s so interesting how like years ago we were managing memory. Yeah. - Right, we were MRR, manual retain release, and now we’re still managing, we’re just managing context. - Yeah, yeah. - Yeah, yeah. - And actually-- - About context engineering. - Yeah, and that’s actually also important for Siri. So there is also, for Siri, you can impact how much context is used, depending on the description of your tool, as well as the description of the parameters.

Maybe one thing that could be interesting to point out is how for app entities, you can choose to show or not certain property to Siri. That’s actually a good way to optimize what we show and to make just action go. That’s right. You can have private and internal properties.

And those are cool for like, well, I want this on my entity because I’m using it elsewhere, like say in a widget or as a parameter. When I get that entity as a parameter, I want to read it back. but I don’t want the model to see it. So that’s when you market private.

  • Yeah.
  • This reminded me, even with skills, I believe Siri will decide reading the front matter if it should load the whole skill or not, right? So it’s managing its own context that way.
  • Yeah, like one of the cool things about App Intents relative to like some other approaches to models is like the type safety, right? Because this is Swift, FM does this too, right? In its own way.

But basically like if you, like someone had example earlier of like a unit, right? Of like kilograms, right? - Yep. you can try to use a string, and then you can spend a whole bunch of time explaining that it has to have KG at the end of it, and it can’t be capitalized, and waste all that context. Or you could say this is a unit. And then we’ll use constraint generation to be like, no, it needs to look exactly like this. And it’s all generable. Yeah. Unification here. Awesome. Thank you. Next question. How should customers learn what they can go with Siri and understand its limitations?

Seems like a broad question. What they can do with Siri, probably. Okay. And understand its limitations. Yeah. So you can actually ask Siri, what can I do with this app? That is a new thing. It’s really cool. Try it in Notes. Try it in Finder. Try it in a couple of the different places where we have some new customer Siri functionality, and you’ll see it. And then if you try it with your app and you have some tools, you’ll see that they just magically show up. It’s very nice. It’s actually a cool, quick way of determining, too, hey, did Siri even find my thing? It isn’t there.

Well, just ask. Yeah. Siri already knows what tools are there, what workflows are supported. So yeah, ask Siri. Awesome. Next question. With schema-conforming app intents, Siri AI can only reason over intent properties matching the predefined shape of the schema. Is this still the case? For tool-conforming custom app intents, can Siri AI reason over all your custom properties? Yes.

It can. Next question. Nice. Next question. Yeah, it can. That’s kind of one of the big differences between schemas and tools is that with two intents, Siri can reason over all your parameters to an intent. And unless you’ve marked them private, Siri can see all the properties of your entity. And that is very powerful.

But keep in mind that if you are in a domain where there is a schema, like if you’re a reminders app or a calendar app, you should start by adopting the schema. I can’t emphasize this enough because schemas give you a lot of benefits, like being able to have reminders created in your app without your app being mentioned or foregrounded. You can’t do that with a tool intent.

Schemas have this globalness to them. And so if you’re a reminders app or a calendar app, start with that. If you fit a schema, start with your schema. And then you can always add additional tool intents, even ones that use the same entities. And they will work, and you can kind of beef up your functionality that way and handle some of the things that are unique about your app.

That’s really cool. I think to add some more color, definitely try and have your tool intent sort of differ from the actual schematized capabilities. But when those tool intents will be available, Siri should favor essentially your custom intents over the schematized one. And so again, back to Siri evaluation, that’s a good sort of boundary to see.

Yeah, I am a schematized entity. I have those extra functionality. What they should call. Exactly, yeah. What to say. Makes sense. Thank you. Next question. This is related to something we received earlier. Which Siri AI requests run fully on device and which needs server? Does an app intent still work when the limit is reached or Mac is offline?

I think we answered some of this, where for tool intents, it does reach out to PCC. Is there a limit that we have discussed? Yeah, I’m trying to see what kind of other details I can add there. Yeah, I think essentially when PCC cannot be reached, that’s where we’re not going to be able to call into the tool intent. Yeah. It’s also a good thing to run through the evaluations to see what exactly gets invoked and what it’s doing too, right? Yeah.

That way you can test your app in all scenarios, including because, let’s face it, people are going to be in places where there’s no internet connection. That’s right. Right? What is Siri going to do in those situations? Wouldn’t it be nice if you can test it? Yes, it would be, and we have a framework for that.

Sounds good. Let’s move on. How does CDAI work with custom tool calls that are long-running app intents? Will it cut off those intents early? Is there a way to communicate to the user that the task will take longer than a typical request? That’s a good question. Oh, to the user. Okay. Yeah, you probably want, we have an API called long-running intent, which shipped quite recently.

I think this past WWDC is probably the first time you’ve seen it. And it’s very cool. What it does is it allows you to get a significantly longer period of runtime. I don’t know if Siri has a timeout. It might not be indefinite. It is a long time of 30 seconds. Yeah, but it’s much longer than the normal 30 seconds. Yeah.

And the compromise is it has a very explicit UI associated with it because we always want the user to know if you’re doing something long in the background. and it also has like cancellation built in so that, and the ability to report progress, right? Those are all really important parts of a user experience if you’re having something that’s going to run for five minutes or really anything longer than a pretty short 30 second type period of time, you’re going to want to respond to, hey, the user asks you to stop, please stop. And you’re going to want to show progress. Yeah.

I kind of feel like it’s the responsibility of the developer to kind of design tool cause in a way where it doesn’t take that much that long or like you have that much like model round trip back and forth right so with the recommendation here being like having more granular and like more precise tools where you do one thing and then you can go go to the next thing yeah that’s that yeah that’s a good uh excellent point think of the internal experience not just the individual this next question is actually i think aim.u matt and karina can an app use Apple’s on-device models for conversational driving assistant? I believe driving here means driving a vehicle. While keeping location and trip data private, what are practical limits? That’s an interesting question. Apple’s on-device, I’ll read it again.

Can an app use Apple’s on-device models for a conversational driving assistant while keeping location and trip data private? What are the practical limits? I think there are a few different questions within this question. I’m pretty sure we can because the on-device AFM, you’re not sending any data to the cloud. When you’re using that, all the processing is done on-device. Completely on-device. Yeah, so by nature of that, no data is exposed to Apple or the cloud.

Right. Is there a second question? I think the question about what are the practical limits, I think this hits the context window question, wherein if you want to have an ongoing sort of session. So how do you manage a context window? - Yeah, I think there’s a few, I mean, like we said with the context management with different ways to trim the transcript and that sort of thing.

I think also evaluations here, like try now, seeing how things might work for that sort of scenario. I think also the FMCLI we can plug, which you can quickly try with the model and see how things work together there. - Which is on the macOS terminal? - Yep, built in. And so I think those are some areas that you can try.

That’s what I would recommend. - Yeah, I would add to like with Xcode profiling, you can use, there’s a very granular kind of multi-turn, like pre-filled decode profiling you can do to see like, oh, how long can you keep the conversation going and like how does that translate to the user experience in the end?

So we have a lot of tooling on Xcode. Yeah, no, the only thing I’ll add to that is if the on-device model, again, is not serving your purpose, then you have an option to bring in your own model with a larger context window, maybe couple that with a Core AI model that does text-to-speech or whatever capability you want.

You can bring in pretty much any model you want on-device. I think what’s really nice, too, is you can start with your code with a system language model and do everything that you think is right in terms of what you want to set up. And then if you’re thinking, oh, I want to try a different model, it’s one line of code change, right? Yeah, exactly.

You can build a lot already and play around with different models as well. Yeah, and we touched a bit on the general goal, but if you were to bring a custom model that’s hooked up with foundation models for a driving app where you need to, for instance, show anything that needs to be a schema or a constrained structured output, You can also kind of define that and use all the capabilities we have hooked up with foundation models to support that as well.

Just to super clarify, when we talk about the integration, the foundation models framework is a single framework for multiple model backends, right? You can have a core AI model. You can have the on-device model. can have the PCC or bring your own model. And you can have a consistent interface and be able to swap between these.

That was new this year. Yep. And Matt and I both worked on that. It’s kind of like you have the right API boundary of the entry point where you kind of work on what is the high-level language model protocol that you call into, but then whatever runs on device, if you want to bring a super custom model, you can feel free to do that, but you’ll have a very simple interface. to work with. Yeah, same exact entry point, which is really nice. And this by saying with dynamic profiles, you can have agentic workflows where you swap between these models too based on giving them separate tools and separate instructions and all of that. Yeah, exactly.

Awesome. Thank you. Good question and good discussion. Thank you. Next question. Back to Siri. How can Siri use context from an active trip, such as the destination next turn or ETA without exposing more location data than necessary? Seems like on the topic of transportation and navigation, but Siri this time.

Yeah, thinking about this, I’m not quite sure what exposing means. You mean exposing to Siri, I guess? Yeah, that part is not clear. Yeah, I think maybe touch a little bit on how we mentioned you can customize what is shown from an entity. Oh, yeah, yeah. So the private entity properties, you can do that, yes, to basically make it so that Siri does not see that information. That’s for the visibility, right? That’s what? Visibility? Yes, it’s called visibility. In terms of privacy, everything is still in the PCC world and viable. Yeah.

Maybe it could be a good segue to talk about the authentication policy of a property. Oh, yeah. That’s what I’m talking about. So, yeah, there are authentication policies on intents and now on entities and on their properties. And the goal of those is really to control what happens when your device is locked. So there’s a couple values, but the crux of this is really that there’s some intents that you want to allow on the lock screen because they’re considered safe. Maybe they’re adding a reminder in a way that isn’t really going to cause problems if someone grabs your phone.

But then there are lots and lots of intents and lots of kinds of content that are very sensitive. So if you, say, work on a banking app and you have an entity representing something about the account, maybe you can expose the last four digits, and that’s fine. So you might set an authentication policy there that says always allowed, but on a property that represents the balance, maybe you perhaps don’t want to do that.

I think it touches, again, like best practice for developers always trying to validate when it touches more sensitive operations or deterministic operations. You can’t expect the model to do anything like that and expect anything outside of your application logic to do that kind of validation. Yeah. I think that makes sense. Thank you. All right. With iOS 27.3’s new Siri AI, can developers generate complex validated configurations entirely on-device or is Siri still limited to triggering predefined actions?

Thinking about how to interpret that. I think predefined actions really sounds like schemas to me. And the answer is like, no, Siri is not going to be into schemas anymore. But if you zero into the on-device part, it sounds like, Kevin, you’re saying the use of tool intents does require PCC.

So although it is very safe, again, CR security guide, it is not entirely on-device. Yeah. Not technically. Sounds good. Thank you. Next question. How would you recommend we surface skills to our users? Old APIs, we could surface the add to shortcuts button via intent framework. Is there something I haven’t discovered in documentation that helps with skill discovery? Ooh, that’s a good tip.

Does Siri app show app skills like the shortcut app shows automations? Yeah, so definitely try what can-- Siri, if the user asks Siri, what can I do in this app? It will find it, which is very cool as a user discovery feature. I think we are definitely interested.

Again, this is part of the we’re looking for your feedback bit. But we’re definitely interested in supporting developers who want to help users discover what is possible. Obviously, that’s a very common challenge with all kinds of voice interfaces. How do I know what I can say? And it’s definitely worth, like, developers playing around with this on the design side of, like, is there an onboarding screen that would make sense?

Can we prompt the users in a way when a new version is set up to help them discover, like, hey, you can now say something like this or like that. And unlike with app shortcuts, it’s no longer an exact phrase, right? So it’s a little bit of a different kind of onboarding story than app shortcuts, where it’s like, here’s literally what you have to say. It has to be literally this.

Instead, this is more like, here’s a thing that. you could say that would almost certainly work. And you could say lots of things like this that will also probably work. Yeah. And I think the integration has to be thought as in, like, the user shouldn’t have to think about skills. That’s the whole point. You should just, you know, think about, like, what are the cool usage that would unlock, you know, people to use Siri with your app. And it just should naturally flow.

And so, yeah, that’s where skills sort of, like, enables those more complex workflows. But from a user perspective, it’s going to be-- FRANCESC CAMPOY: You should be telling your users, here are some skills. You should be just having good design guidance in your own app that tells users about capabilities. MARK MANDEL: It’s about capabilities. It’s not about telling users how to invoke them anymore. It’s about Siri and my app can now collaborate to do x.

For example, the Finder feature we shipped, where it’s like, you don’t even know how to tell Finder to do this. We just told you, hey, find your canal, clean up your folder. The idea is that almost any way you can reasonably think of to ask finder to do that, it will. That’s so cool.

Funny thing is in the ML community, skills is such a common word. But in general, it means nothing to a lot of people. That’s true. Yeah. Awesome. So here’s a challenge. We have 10 minutes left. We’ll try to get to lightning round. Lightning round? Yeah. But with good intention, let’s try to answer. All right. Tool intent is unavailable on watchOS. In our mirrored watchOS workout app benefits from a live HK workout background runtime. What experience can we expect for watch users who invoke Siri on the watch while leaving the phone face down on the ground?

Oh, interesting. Yeah, I think that is a great use case and a great feedback to follow for us. Again, I will emphasize, this is a developer preview. A lot of the purpose of this is these are incredibly powerful tools we’re giving you. We need feedback about how you’re using them, how you want to use them, what’s possible, what isn’t.

We think we’ve enabled a lot right now, but it’s not everything. And we are excited to learn from what you guys attempt and to see what you guys do. I know we’ve said this before, we’ll emphasize again, feedback requests, please do file them. And if you’re seeing something not as you expect, then forums are a great place to share them in addition to filing feedback. Thank you. Next question.

Can Apple’s Foundation Models framework serve as an on-device inference backend for a third-party coding agent where the application independently manages orchestration, tool execution, permission gates, and audit logging? And can developers guarantee the inference remains entirely offline. Yeah, I mean, absolutely. From the Swift API, you can build that right away. In terms of other models, in terms of larger context size and capabilities, that sort of thing, I think, again, echoing the whole core AI thing, I think. I mean, if you know more about those capabilities. Yeah, exactly. Check out core AI models.

Repository actually brought some models of that size and of those capabilities, and you can use directly with foundation models for your work. I think this question is specifically focused on the Apple’s foundation models framework on device. So, yeah, they are not maybe specifically talking about the Apple foundation models. Sure. But if it’s bring your own, in either case, I think you want to make sure that the model is actually good at coding. Yes.

So we don’t recommend using our on-device model for coding use cases. right but if you were to bring in your own through core ai 100 you can do that then 100 that’s a valid use case yeah like in um some of the mac platform especially where you can bring a lot more capable and like larger models we totally have models that are great at like coding coding already and um as i was mentioning like a lot of these models are already great at tool calling um and like kind of all these like agentic workflows already um right where you you just bring them in, you convert them through Core AI models, and then you use them with a foundation models framework. And I think with privacy, too, at the end of the question, yeah, it’s on device. It’s fully on device.

And I’ll just add that I actually do this across using MLX and Core AI. You can hook it up to your popular coding harness, and it works beautifully. So, yeah. Awesome. Thank you. Next question. For a navigation app, how can a driver make a hands-free request like, find a gas station along my route and have the app carried out using Siri or Apple intelligence.

How can a driver… Interesting. Find a gas station along my route would be the request to see. Yeah, yeah. So I would do two things. I don’t have the navigation schema memorized, but I would start with that and see how far you can get, right? Again, as I said earlier, make sure that you don’t start by overlapping schema capabilities. Instead, you start by using the schema. Once you have tried that, if you’re looking to push, if you’re a navigation app and you want to push beyond, what the schema can do. That totally makes sense. That’s when you would start adding to intents.

And I think you can do nearly everything you can imagine with the two-intent infrastructure now because of how flexible it is. SETH VARGO: Awesome. Next question. What on-device Foundation Mods latency should we expect? And on which chips can we warm up the model so the first request is fast? SETH VARGO: Oh.

SETH VARGO: Well, we have a great API called pre-warm for this exact sort of, well, pre-warming. I’d also take a look at instruments. We did a great collaboration where You can look with your requests, the latency, different turns you have, tool calls, that sort of thing. And you can really see how things are working for your specific use case.

We have a great WWDC video on instruments pre-warming to speed up. As far as what latency should we expect, that is something you can see on the instrument. Yeah, I’m not sure off the top of my head, but that’s a great way to play around with it and see where you’re going. Yeah, I would add for any custom models, a lot of that first-time load, If you want to offload that to build time, we have ahead-of-time compilation. That’s a completely new feature.

So the kind of first time-to-first token user experience would be a lot faster when you were able to do that offline. Perfect. And the reason we want to discuss this in the same context is because a lot of people will be using Core AI through the Foundation Malls framework. Exactly. Which means you also get the benefit of using the instruments.

Exactly. Checking all of those chunks. I think it’s very possible. we get some really cool stacks of these frameworks where essentially you have app intents, right? Siri talking to app intents, talking to FM, talking to core AI in this like, to create these like really cool flows. - Absolutely.

  • The world is your oyster.
  • Next question. How much control do developers have over combined app intents? Can we explicitly define dependencies, execution order and constraints between actions or does Siri dynamically determine the workflow? Additionally, how are partial failures handled when some actions succeed but others fail? Oh, interesting.

Let’s see. Maybe the first part of the question. Combined app intents. Yeah, so there is, you know, depending on our use case, there might be like a natural flow where, you know, you first need to create an entity. And then that second app intent would take in that entity as a parameter.

So I think, again, it goes back to testing out the boundaries of your use case and see, can Siri naturally flow those tools? And if that’s the expected outcome, if not, that’s where you can have your own skill that would explain more that domain-specific and how it should be. Yeah, I think you have a couple layers, right? You have like, just break it up into tools, define the tool structure well, and then define, you know, work on your tool descriptions and tool titles, and then see what happens. Do that first, right?

Then if that doesn’t work well, see if you can write a skill, right? That basically like prompts you for how you want those tools orchestrated. But keep in mind that won’t be deterministic. It will probably work very well much of the time, but it is not a 100% hard-coded software solution, right?

And then if it is not working well or if you find a case where you absolutely must enforce a hard dependency, that’s when you might want to try combining the tools instead so that the model can never separate them, right? So you never get partial failures in that case. But I would go in that order, right, rather than jumping.

Okay. We have a couple more minutes. Let’s see how many we can get through. With Siri AI now supporting custom app intents and skills, how can we make our app discoverable when users make generic requests such as summarize this file or convert this file to without explicitly mentioning our app’s name? How does Siri choose which third-party app should handle the request?

Yeah, yeah. So the answers there are a little different for schemas versus not. For schemas, I think we talked about this earlier. It’s really about user behavior and donations, right? So if we know through Siri’s own behavior and through your donations from your UI that I use a certain reminder app, that’s the reminder app that will be chosen if I were to ask Siri to create a reminder.

For two intents, it’s a little bit different. There are those four conditions that Kevin mentioned. App named, app in the foreground, live activities running, or we’ve retrieved content from search. And so that transcript contains an entity, and then that will enable your app tools if the context contains an entity from your app.

Exactly. Awesome. I think that’s all the time we have for today. Any final words? Any single key takeaway for our audience today from each of your frameworks? Oh, man. Play around with the language model protocol. I think it’s really cool. It’s awesome to play with the different models. Yeah. Maybe the same for you. Yeah, 100%. But also, you can build such intelligent experiences with all these frameworks together, like custom models through foundation models, framework as the perform method to App Intents, for instance. That’s just really cool.

Yeah, and yeah, really excited to see what people are going to build with all those technologies and yeah. Thanks a lot for all the questions. Absolutely. Yeah. We’re already seeing people post like really cool examples and like, honestly, the best part of this job is looking at some, you know, social media posts or someone showing a video and be like, they did what? Wait, how did they even, wait, does it even do that? I guess it does. That’s cool.

That’s like the most fun. Yeah. It’s kind of why we do what we do. Absolutely. Thanks everyone. The only thing I’ll add is Xcode has great agentic capabilities to do all of this. Oh, yeah. Several skills, so you’re not alone. So awesome. So that’s about all the time we have for today’s group lab. So we’re so thankful that you were able to join us today and hope you found this really helpful.

I also want to say thanks to our panelists as well as the folks working hard behind the scenes to make all of this happen. As we mentioned earlier, if we didn’t get your question, please head over to developer forums and ask your questions there. We can continue the discussions. And feedback assistant is your friend. Please file feedback. We love to see those. Thank you. And we really do appreciate you tuning in. So until next time, thank you. Thank you, Max. ♪