Good morning. Welcome to the Apple Intelligence Group lab. We’re so glad to have you here. My name is Ernie Sabella. I’m an AI machine learning evangelist here at Apple. There have been some awesome announcements at Wwdc for Apple Intelligence Foundation models framework, private cloud compute, visual intelligence, and a whole lot more. And of course, app intents and Siri.

I am joined by an awesome team of engineers who are going to help you with those questions. And why don’t we begin by some quick introductions? Yeah. I’ll start. My name is Matt and I work on the Foundation Models framework. Hi, I’m James, I work on App Intents. And I’m Rob. I work on the evaluations framework.

I’m Louis. I also work on the foundation models framework. I’m Dan I work on Siri and appetites. Awesome. Thank you all. In addition to those you see here on stage, there’s a whole slew of engineers behind the scenes working to answer your questions, please submit those questions. If you have a code related question or issue. Definitely connect with us on developer forums. That’s a much better place for us to get into specific coding issues, but we really look forward to answering your questions today about all these exciting new updates.

And if you have a bug or feature request, of course, go to Feedback assistant.apple.com. And yeah, we’d love to get your ideas and feedback on any bugs, although I know bugs in our software. I haven’t seen anybody anyway. All right. We have no bugs. No, not this year. On occasion, that might happen. Anyway, with that, we have a lot of questions. Let’s get started. The first question we have today is from Mario Gravina. Thank you, Mario, for the question.

The question is, what are the most useful Apple resources for learning and applying the new version? 27 APIs and design changes in real apps, updated sample projects, migration guides, etc.. That’s a really good question. Thank you for that, Mario. Yeah, well, of course, there’s Wwdc videos, lot of great documentation, a lot of sample apps. Any other suggestions you guys. Dedicated sample section of the documentation page?

Yeah. And there’s definitely a lot of great technical articles as well. So a lot of great resources. Yeah. So I think on App Intents this year, I think we had five sessions. Most worked on a few. Like I think there was a few different sample apps that we put out. A lot of the sessions used them in the talk. So you can kind of see how they were built, how they’re used. We also did a big push on our documentation this year.

No for opinions. I think for foundation models as well that really show we built a lot this year. Yeah. And to kind of how what you want to understand it, the docs are not just here is the API, it’s how to use the API, how it benefits your users.

Yeah, I think you made a good point. Like if it was in a session video, it will show up in that sample section. Yeah, no great points. And obviously WWC from last year, previous years. Lots of good material too, especially if you’re starting out like say with app intents, a lot of great foundational material as well. So definitely check that out. So awesome. Well, thank you for that question. Next is from username looks safe. Hopefully I’m saying that right.

Nice name by the way. My app doesn’t fit any of the predefined schema types. I think we’ve heard this one a few times. Is it better to try to use the closest related types, or should I stick with not supporting the schema? For now, my app is for tracking time. There are times types for timers and for calendar events, but they don’t quite fit within my app offers.

That’s a very good question. I can take that. So new this year is the introduction of app schemas. These are schemas that are fine tuned and designed by Apple to integrate with our great system experiences. The cool thing about them is that as a developer, you can simply pick and choose the schemas that best match your app’s capabilities.

So you mentioned, you know, timer functionality, calendar. We have schemas for those actions. And you can simply pick and choose the actions that match your app’s capabilities. If your app has functionality that is outside of the schema, well, you can also combine it with custom app intents that you can write for those actions. And when you write an app intent, that action becomes available in many places across the system. The shortcuts, app spotlight, and more. Now, it’s important to note that for the new Siri AI integration for integration with that, you do need to adopt one of our app schemas.

Now, the other cool thing too is a lot of our schemas are sort of designed for certain specific types of actions, but we also have a domain called the system domain, which is more generic and broad. So if you have functionality in the search space, for instance, apps like e-commerce or maybe food ordering, well, you can integrate with system dot search.

And what that will do is it will allow your customers to use Siri to basically integrate directly into your app search capabilities. So you can say something like, you know, order me a bicycle on my favorite marketplace app and it will deep link to your app directly into your search capabilities so that you can continue your ordering flow through your app.

We’re really excited about this. There’s also another schema in this domain called system dot open that allows you to open specific entities that can be donated to spotlight, and Siri will be able to open those entities directly in your app. And if none of these schemas satisfy your needs, well, we also have app shortcuts. James, do you want to talk a bit about app shortcuts? Sure. So App Shortcuts is an API that we’ve had for a while.

It allows you to add phrases for an app intent. And when you do that, the action becomes available in the system. So it’s automatically available with Siri, shortcuts, spotlight, and a few other places. You can even use it to on the iPad if you want to tap on an Apple Pencil. It all works with app shortcuts, and it’s a great feature.

As Dan mentioned, when you have a specific action in your app that you really want users to be able to do, and it has a memorable phrase. So app shortcuts does require a phrase, but for these very specific actions, these kind of hero use cases in your app, they are super impactful. And then yeah, Dan, you mentioned a bunch of other things. We have a ton of APIs. We have a ton of different experiences.

There is, it’s kind of a choose your own adventure. Start with the schemas. If something makes sense, use it. And then there’s a ton of other experiences as well. And the documentation this year is great for finding out what functionality is available through app schemas. Also how app shortcuts work. Please check out the developer documentation to find out how you can integrate with Siri. That’s awesome. Great. All right. Well thank you. Great question. Moving on to our next one. This is from Josh. Let’s see.

Hopefully I’m saying that correctly. The message or the question please. Is my app’s core entities and actions don’t map to any predefined schema domain like the apps or like the apps of many others. What’s the best way to bring them to new Agentic Siri today? What should I do now to be ready for Platt as the platform evolves? And how do you. How should we think about their place in the Agentic experience? Good question.

I think we I think we covered a little bit of that. In the last one. Right. Yeah. Yeah. You know, I think we have these. Take a look at the schemas. See if as Dan mentioned, there are you know, you can kind of pick and choose parts of these, you know, domains.

If something makes sense, implement it by conforming to an entity schema and kind of donating and interacting with spotlight. You do get a lot of other benefits, even if maybe one of the intents doesn’t make sense in your application, you still can use Siri AI to ask questions about the content of these, or take action on it using any of our new view annotation API. So there’s, there’s a little bit there, right?

Yeah, no that’s great. Thank you, James, for that. All right. Next question from Indigo J. Another nice username exposes an app’s actions to Siri shortcuts. But there’s no API for a third party app to invoke another app’s intents directly. Siri shortcuts is the orchestrator. Can an app act at its own orchestrator and drive other apps, or does cross App Action still have to route through Siri shortcuts? That’s a very good question.

Thank you. So new this year, we have a system orchestrator that can take action from app intents from many different apps across the system. Now, one of the important things for us to consider with this is when it comes to privacy and safety, one of our key concepts of Apple intelligence is that it is AI, personal and private to you.

When you integrate with App Intents, you’re basically making actions from your app available to the system, and you’re also modeling content in the form of app entities. Now, the cool thing about our semantic index and our spotlight integration is when you index content from your app, you’re making it available to the system and to the system alone. So, you know, it’s not like you’re sharing data from your app that could potentially expose other apps from accessing this data. It’s all sandboxed to you and the system and the system alone. Now there is also a concept this year called transferable.

Transferable is an API that you can adopt as a developer that allows you to control. How you can share data from your app to other apps in the system. The API essentially has. Bidirectionality support, meaning you can choose which data you want to export out of your app. And which data you want to import from other apps. So it’s a really neat API that makes the boundaries clear between what is data that your app owns versus data that you can import from somewhere else.

Excellent. Yeah. Great stuff. Yeah. Great year. A lot of great new stuff. Great question. All right. Next 1NDT in the question is with the new Siri in iOS 26. Can my third party app take part in a multi-turn conversation? Another great question. Or am I still limited to the discrete actions and and entities I exposed through app intents.

I could take this one. The answer is yes. This is what the schemas provide. You can take action from your app. You could ask questions about your entities. You can do follow ups. It is. It’s I’ve been working on it. I’m very happy to finally show it to people. Yeah. I want people to take a look at the betas. Try it in your app.

It’s, it’s a great way that we believe your users will be able to interact with your applications. You know, it’s your app, it’s your content. They have kind of a deep understanding of this. And to just be able to ask natural questions about it and do these follow ups is it’s just really awesome to see. Yeah. And I’ll add on that, you know, at the heart of it, you have a large language model that is processing all of the input.

So it gives users, you know, more natural capabilities for how to express and take actions with your app. Previously, the commands had to be very specific to form into a shape. Now with app schemas, you have a lot more flexibility into how your users can phrase requests and have our model sort of parse that information and pick the right action for your users.

Yeah, no great stuff. Great videos, by the way, from this dub dub on the on these topics. So definitely encourage you, of course, to, to watch those as well. Yeah, a lot of great, great sample code to go along with that. All right. Well, thank you for that question. Moving on.

I believe is the way we say that username. Is it okay to mix and match non schema app entities intents with some that match different schemas from different domains all in the same app. I’m loving all these. Schema questions. By. The way. The answer is absolutely yes. With schemas the. Our intention is that you pick and choose the schemas that work for your app, matching your app’s capabilities.

So if you have an app that can do, you know, calendar actions, but it can also do reminders actions, please pick schemas from any of these domains. The cool thing is that you can mix and match them, and they can work together to deliver full holistic experiences. Yeah. And it’s also this is not a binary choice where you have to use schemas or you have to use app shortcuts or search in app or something. They play nicely together. There are going to be some experiences.

You know, you’re, you’re the developer of your app. You understand your users and you should be. We expect people to have little bits of schema, little bits of app shortcuts. They. It really builds this holistic experience for users to use your app. Love it. Oh that’s great. All right. Moving on. Next question from a B high 27. I’m not sure how to say that one.

Is there a practical time work budget for foundation models during one background wake? If. No. If there. If no. Fixed number. How should apps safely chunk work? Yeah, I can take that. Yeah for sure. Yeah. So with the on device system language model with foundation models, if you’re running in the foreground, your app, like there’s never any like rate limiting or anything like that. Unlimited requests, right? If it’s running in the background, we do apply some throttling if the system is under heavy load. And so what you can do with the API is catch the rate limited error.

So if you’re getting the rate limited error with the on device model, that’s the way that the system is telling you that your request, even though it ran on device that the system was under heavy load. You should try again later. Right. Okay. Yeah, that makes sense. You know, I have to say, I was so excited when I learned we’re going to be making PCC available to developers. Yeah. Yeah. It’s an awesome model. Yeah. Anyway, that’s really great. So yeah. Talk more about that. Later. I bet. Yeah, yeah.

Awesome. All right. Moving on. Also from Abby. Hi 27. The question is what exact error type codes indicate foundation model throttling. Okay. I think you may have addressed that a little bit. Previous question system busy thermal pressure code exhaustion, which are retrievable versus terminal is nuanced air reporting available?

Yeah, I think, Matt, if you want to talk more about our air handling in general. Yeah, yeah, for sure. So this year we came out with the language model protocol. And as part of that and kind of encompassing common errors that a lot of models will see, we have the language model error. And so there’s a lot of those, you know, common use cases of like rate limiting and refusals, those sort of things, but specific with the PCC model, like you guest coming up, we also have quota usage, so you can get a signal through the property on that specific model saying, hey, are you getting close to that limit or limit, are you not? And we also have errors as part of that that will throw specifically for that model saying, hey, during this this prompt, you have a quota exhaustion and you can handle it from there.

Yeah. That’s great. Yes. You could find this in the documentation documentation. Yeah. He has a great session. Plug that real quick. Yeah. So yeah, definitely check those out. They’re really great. Yeah. It’s great to hear developers bringing those kinds of questions because, you know, it’s, it’s, it’s the rainy day situations you got to prepare for in your app too, right? Not just when everything goes well, right? So yeah. Great. Thank you for that question. All right. Next one from Thub THUB. Hopefully I got that right here.

How to avoid language model session dot generation error dot guardrail violation. In my tests I am getting quite often guardrail violation errors for prompts I cannot see any issues with. I’m thinking the evaluations framework. Yeah, yeah, yeah. We, we ran into this a bunch when we were building samples for ours.

And there’s a model judge evaluator in the evaluations type, and you can create safety rubrics inside of that where you just give it a scale like let’s say 1 to 4. How safe is this? Does it have some safety implications? But if you actually ask the model, you know, or even ask, you know, a different model, let’s say if you’re, you know, you’re getting this a lot in the on device model, you could use the PCC model to kind of check that at being a larger model. It could kind of evaluate those and let you know if there’s safety errors. But the guardrail violation is there to help you avoid unsafe responses. Yeah. In your app.

No. Great. Yeah. Great advice. And I want to put a plug in for the evaluations framework. Oh my gosh, it’s one of the biggest questions we would do workshops was about how do I test my intelligence based features? And I have to say what we’ve done between the model shows quantitative, qualitative metrics. It’s really, really, really great stuff I don’t know of anybody else doing. I’m a fan. But yeah, you might be.

A little biased there, but no, no, definitely check it out. So and thank you for that question. Really, really appreciate that one. Okay. Moving on. Tattered NTT in iOS 27, can the Foundation models framework reason over video or work with a video model, or is multimodal input limited to still images?

Matt. That might be a good question. I’ll take. That one. So back to that language model protocol. So as models and people come to the foundation model framework, we have them have capabilities that they support. And one of these is vision. So if you add that as a part of the model, the framework can handle to say, yes, this is approved for this specific model. And if so, it can continue on and do that sort of thing. Specific to reasoning in Apple’s models, the PCC model has that where you have the the capability to specify light, moderate and deep and reason over those things.

There’s another great session talking all about that, where we have a new part of the prompt where you can have an attachment and bring in those images that you want to look with. And, and we’ve shown that with my dog Sully, which has been great to. Test with all of us and.

See that the model performs very well. Yeah. So yeah, those are the sort of things that I can say. That’s sort of the key thing that like we’re making the framework, the foundation model framework extensible this year, right? Yeah. But also this year we added support for image input, both with the updated on device model we have. And then of course, with the new server model running on, on PCC.

Right. But in terms of like the, the extensibility with the language model protocol, letting you really bring any model you want to the foundation model framework. And the protocol itself is sort of open ended where if you have custom modalities, for example, like video that we have the custom segment API that you can really sort of build whatever things you want on top of it.

Yeah. The framework doesn’t, you know, kind of hold you into, you have to be within these certain things. If your model is very specific, like it can smell something or crazy, you can use a custom segment to still reach the users. And with it being open source, also announced, you have even more capabilities where you can run on the server with Swift and those sort of things.

If I may, I mean, this is totally I mean, that was a lot of great information on foundation models. But again, if you’re looking for a one shot approach, I think everything they said kind of makes sense. But if you’re willing to do processing ahead of time, or if you want to build a tool, you could build a tool that uses Apple’s other frameworks that could get a transcript of that video.

You know, maybe what it’s hearing audio describing it. There’s a lot of good frameworks in our API, or there’s a lot of good APIs in our frameworks that allow you to extract those, and then you can just feed that text description, right? And at that point, I mean, if you’re doing summarization or something like that, the on device model is great for that. Yeah. There is a great what’s new in image understanding with the vision framework. Oh yeah. That’s really awesome. Yeah. Yeah. And it’s like you build a tool and it’s like the tool does that processing and then feeds it into the model.

Yeah. No, that’s, that’s, that’s really great stuff. I mean, the open source, oh my gosh, that’s such a huge announcement. I’m like so excited to hear that. Absolutely. For the developer community, right. I mean, yeah, Apple’s behind that. I mean that’s huge. But the extensibility piece. Yeah, that’s really, really powerful as well. So cool. And hey, another quick plug for evaluations framework. I know I saw.

You mentioned. Tools, right? Like, you know, one of the problems I’ve always had is like, what tool got called when or, you know. Yeah. Now I’m able to see that. I mean, yeah, that. Oh my. God, you feed your, you feed your evaluation. The transcript from the language model session. And then we have a tool called evaluator that will allow you to build assertions against that transcript and see when it’s getting called, what values are being sent to it, etc.. And if you’re a visual, I like visuals. We have instruments now hooked up to. Also see.

As it goes, so you can really see everything that got called. It really helps with that. So you can get kind of like batch evaluation, and then you can get kind of real time what’s going on as I’m. Looking at it. That’s really cool. All right. Well moving on. Next question is from Kyle BSHR. Our app intent schema is something that give your intents an advantage when integrating with Siri. Or can any intent entity be discovered by Siri?

Siri AI the same way. Notably, there’s no finance schemas. That’s a very good question. We’ve heard this one. Yeah, I can take this. And Dan, I’m sure we’ll you’ll jump in here too. I think advantage is probably not the right way of thinking about this. You know, using schemas is the way that AI can reason about the actions and entities in your application.

It’s how. Advantage is really throwing me here. I, I think one similar question we get about this is how can you influence series, right? And for that, we have a ton of new APIs that can help you tell Siri about what actions are happening in your application. So we have new intent donation, like a new intent donation system that’s used for when a user is using your app, you know, they’re, they’re tapping around. If a user performs a similar action that has a corresponding app intent, you donate that intent and that can, it gets donated to a kind of temporary transcript that Siri can use to learn.

So if you’re kind of consistently messaging the same person and then you use Siri to like, hey, message Dan, it will know, Siri will learn that, oh, you might want to use message Dan in unicorn chat or, you know, or one of our many sample space theme sample. Apps this year. Yeah.

Yeah. I’ll also add on that, you know, we have the app shortcuts API that allows you developer to specify the phrases that can activate your app intents. But if you adopt one of our app schemas, the cool thing there is that Apple has done the heavy lifting of essentially providing those sample phrases of training the model and making sure that we can deliver a consistent experience across apps when you integrate with a schema. So if you’re messaging using the first Party Messages app, or you’re messaging using many other third party messaging apps that are available out there that do conform to our schema, you get a consistent experience no matter which app you use.

And that I think is, is really powerful about what we’re building here is that we’re building an ecosystem where users can use Siri AI to take actions across many apps in the system and tailor the experience to your users. You know, you may message a coworker using a very specific messaging app.

And so when you send a message to that coworker, the system will learn about these integrations like James was saying. And so, you know, it won’t even try to use iMessage to send that message. It’ll use that app that you always use to communicate with your coworker. Just one more small plug.

You know, we know a lot of a lot of developers listening and building app intents. It takes a little bit of work. You’re you have to define a lot of your properties. There’s a lot of code that you have to write, which we all write code by using app schemas.

It actually lets you write a little bit less code for these for this, because as Dan mentioned, Apple has done a lot of that heavy lifting for you. So some things that you have to provide in kind of custom intents, you no longer have to provide because we do it for you. So it’s another developer benefit as well of hopefully all the user benefits. Totally.

You know, I have to say too, like going to App Intents workshops, working with developers who are new to app intents and when they get it that moment and seeing how they’re like, oh, wow, it’s like, you know, you converted my device into my, you know, the app knows the device knows so much more about my app now.

And just seeing that light bulb turn on. Yeah, it’s really exciting. And then when you add all the new capabilities, it’s, it’s pretty, it’s fun. Yeah. And the cool thing too to see is when they add the schema to their app intents, they get to delete a bunch of code because that’s no longer needed. And developers love deleting code. I love deleting. Code. Especially when it’s not mine.

Awesome. All right. Well, thank you for that question. Next. Moving on interferon. Oh, that’s a cool one. Okay, cool username here. Can I use Image Playground to generate photorealistic images inside my app? I think I can take this one. Short answer is yes, you can. A lot of great new additions to the Image Playground API.

There’s a really great video this year at this dub dub, and basically what you do is you’d use the image playground style dot all as a a parameter there. So yeah, please check that out. Great video. And I’m sure there’s some great documentation on that as. Well because. That’s running on private cloud compute now, right? That’s right. Images. Yeah. Yeah.

That’s yeah. You get the. Advantage of the more powerful model and yeah. And the API itself. So great, great. Thanks for that question. All right. Moving on here. Lewis 42. Hi, Lewis 42. Thank you for your question. Where’s Lewis 41. Yeah, yeah, where’s. Lewis? 41. Lewis 41 in the House. Submit Mr.. Is he all right?

I’m confused by the difference between indexed entity and things with defined entities. Example app schema dot books entity. Should we use index entities for everything that isn’t a book, calendar, event, etc.? Or are these serving totally different purposes? Great question. That is a great question. So the schema defines the shape of the content. It defines the properties that it can have.

So if we’re talking about a book it can have the content of the book, the title of the book, the author and so on. Indexed entity is a separate API that essentially allows you to index that content into the system semantic index. So the content can essentially be retrieved by Siri AI when users are performing actions like I want to read a book by a certain author.

This is the API that allows the system to essentially look into that index and find the right entity that represents that specific content. So in a way, they actually complement one another. If you adopt an app schema and you also make it conform to the indexed entity protocol, and you then make that call at runtime to essentially index content from your app into the semantic index. You get the best Siri AI experience for your customers.

Yeah. And back to the building blocks. You can use index entity as well for your custom custom entities. And you can donate them to spotlight. The advantage there is they show up in spotlight and you can, you know, you can have a system dot open intent. So if you go from spotlight and you tap a result that’s in there, you can kind of deep link directly into your app.

Another kind of advantage of the schemas that I mentioned before, the deleting code is if you’ve worked with spotlight before, you have to you have these indexing keys. So you know, the title of a book is the display name for app Schemas. We’ve done all that work for you so you don’t have to worry about it. You use one API to donate and we handle everything kind of on you on for you in the back end.

Yeah, that piece actually is really nice because when you’re using the app schema, you don’t have to worry about trying to figure out what’s the right key value pair for a spotlight. Like Apple does the heavy lifting for you. As you developer, you have to basically focus on the minimum amount of adoption needed, which is, you know, you define your app entity, it conforms to schema, and then you simply provide values for those properties and we take care of the rest.

Yeah. No great points. And you know, to plus one, Dan’s point about we’ve done the heavy lifting. I mean, oh my gosh, the engineering that went into those domains, right? Those schema, I mean, it’s a lot. So you’re really getting advantage of all that engineering effort from Apple, right?

So yeah, really, really great stuff. Well, cool. Well thank you for that question. Moving on. Username. Let’s see. Easy. Franca 2015. Thank you for the question. I love the evaluation framework session. Rob plug for Rob here. Did you. Write this question. Ernie? Yeah, he paid me 20 bucks. And. A few. Bills.

It’s both a professional developer and an academic researcher. Oh, well, thank you for being an academic researcher. I’m curious whether Apple envisions the framework being used for academic research, such as collecting metrics, generating evidence for peer reviewed studies in addition to product evaluation. That’s really interesting. I think there’s a lot in, you know, what kind of academic research you could do.

You’re I think what it comes down to is what it is you’re measuring. The evaluations framework was designed not just for evaluating language models, but really any stochastic system. So, you know, if you have a machine learning model, a more traditional model like linear regression or classifier evaluations can absolutely be adopted for that.

It’s just a matter of implementing the framework. We split our metric collection into a couple phases. So you, you iterate over a series of samples and you feed that into kind of the function that is generating the model’s response. And then you use a series of evaluators that will calculate those measurements.

So if this is something that you know, the metrics you want to collect is something that can be measured in those evaluators. And it can be scenario based. So you’re feeding it in a collection of input samples. Then I would say yes, it doesn’t matter whether you’re doing product development or you’re doing research, you can use this.

But you know, we modeled this over, you know, a lot of model providers have benchmarks that they submit and they say you know, this model performs this well on this benchmark. This model performs this well on this benchmark. We wanted you to be able to create your own app specific scenario, specific benchmark specific to what you’re trying to do, whether that’s research or product development.

Yeah. No, totally. And I have to say, oh yeah, just to say really quick, the three sessions we have, right? You get the introduction, you get the more advanced features. I think the synthetic data capability we provide is freaking awesome. Yeah. And then the hill climbing, right. Like how do you incrementally keep improving things? So it really is kind of a nice three session set. Yeah.

I was going to say that Xcode integration too, of showing visuals also really helps maybe to provide that kind of academic side where you can kind of see graphically everything that’s happening. Yeah. And that’s actually great. And you can save those reports, you can export them. They’re all Json based data structures. So if you want to feed it into another workflow.

Yeah. But yeah, that’s yeah, it just, there’s a lot of work going on in that academic research. Yeah. To do specifically. So if it’s if it’s non-deterministic output or responses to a system, then evaluations is a good fit. Yeah. That’s great. Great question. Thank you for that. All right. Next question is from Li Shuang Quan.

Hopefully I’m saying your name username correctly here. This year’s Siri can understand screen context. Can Siri also understand context on CarPlay screen when connected? Also can Siri platform actions like play a song based on user voice input like Siri play song on row two. Column one. Does this also apply to CarPlay?

Well that’s interesting. We we haven’t really talked a lot about on screen context today, so I’m excited to to talk about that. Right. It’s one of the, you know, the three pillars of the new Siri AI is Siri can understand the content that’s visible on your screen and take action based on on that relevant context.

It’s really powerful. We have great APIs for that. We have integration with user activity. If you know of the old user activity API that’s been around for a while, we integrate directly with that, but we also have new view annotations API so that you can annotate content that is on screen with your entity conforming schemas.

I’m just really excited for people, for developers to see this work for users to see this work. It’s, it’s just cool. It’s like you build a SwiftUI view, you have your entities, you attach them, you start speaking and it does exactly what you want. Right? It’s just we have sessions. That’s why they’re in there. It’s just really, really powerful. It’s natural. It’s kind of natural way of interacting. Yeah, totally. And what about the CarPlay part of that question? Yeah, I was no, sorry.

If you have specific things you’re looking for, we always take. You can file bug reports for your enhancement requests. Yes. But I will also say that on the car, you know, we want to make sure that you’re focusing on the road, not on the content on your screen. Yes, totally. Yeah. No, that’s that’s a good point. But yeah, definitely, you know, for if you find a bug. Okay, we have bugs.

By all means a feedback. Request, but also enhancements, you know, we’d love to hear from you and we take that very seriously. Your inputs from the developer community and how we prioritize new features and work. So yeah, please take the time and file that about requesting that support for CarPlay with some safety features, of course, like.

Gotta. Keep those hands. To keep the hands. On the wheel, you know. And don’t file the feedback while you’re driving. Yeah, yeah. Not while you’re driving. Please. Yes. Yeah. All right. Next question from Chaz K. Are we going to be able to use app intents with HomePod since currently there is not a beta for HomePod.

The new Siri AI is available on iPhone, iPad, Mac, and visionOS. It is not available on HomePod. Not on HomePod. Okay. Okay. Yeah. But your existing your existing app shortcuts have worked on HomePod, you know, for a while. So we still have, you know, options for you to call into your app. Right. Right. No, that’s great to know. I love my HomePod. Okay. Thank you for that question. Next is a Harris crown. The message is can we expect watchOS responses to App Intents to match up 1 to 1 with iOS?

This is interesting. I think yes, the answer would be like, we want to make sure that you’re kind of testing these different experiences. Yeah. Dan, I don’t know if you have anything else to. Yeah. I’ll say, you know, the cool thing about the new Siri AI is that it’s available on many different platforms, and we want to encourage developers that when you’re adopting our new or old APIs, that you’re making sure that you’re testing your experiences across all the different devices that we support, including AirPods, you know, a lot of people forget about AirPods. And you know, when, when I’m on the run and I have my AirPods on, Siri is extremely helpful to perform quick actions.

And so we want to make sure that developers are leveraging the full capabilities of the Apple ecosystem. Yeah, I feel like it’s a somewhat less used API that we provide an app intent. So when you perform an app intent, you can optionally provide some dialog, but that dialog can be changed depending on what you call it full and supporting modes.

So you know, you might be in a, when you’re using your AirPods, you don’t have a screen in front of you. You might want to be a bit more verbose. Yeah. Where you’re on, you know, you’re using your iPad and we have a nice snippet and you might want a shorter version of the text. So those APIs are all there.

And, you. It makes sense. You know, you might need a little extra content. Yeah. And, you know, intense testing framework this year is out. So for everything we’ve been discussing, but. For some questions. About that. Yeah. Yeah. Great. I’ll also complement that. We have a great session this year that allows you to focus on the customizing the Siri experience like intent dialog, like James mentioned. Highly recommend to check out the sessions by our coworker Antonio. It’s a great session. Yeah. Yeah. Really let a really great, actionable, good, you know, toolbox kinds of tips and tricks in that that session. So really good stuff. Okay. Moving on. Username.

Apple jacks. That’s interesting. All right. Apple jacks. I want to say Apple geek. Anyway all right. Does Image Playground now require an internet connection for PCC or does it fall back to the previous model gracefully? Well I can take this and actually Image playground always requires an internet connection at this point to generate images because as we mentioned, it uses PCC, which is our server based model. Yeah. It will not fall back to an on device use of the model. So thank you for that question.

All right. Next one here is from MC Rich 23. What are each of you most excited about regarding the updates to Apple Intelligence? Wow. Well thank you. We could spend an hour on that one. Although a lot of us, you know, having worked on delivering those features, we’re probably like, oh my gosh, yeah, a lot of hard work here. We’re going to go down party lines. Yeah. Exactly. Right.

But yeah, any. Yeah. Anyone want to share what you’re most excited about with Apple Intelligence? The one that I was really excited about was the like intelligence features inside the phone. Like when you’re on a phone call, like I think the example I showed was the flight. Oh, right.

Right, right. I’m always struggling like, what’s your confirmation number? And you’re going through the mail and you’re like, I can’t, you know, and having that right there, I was like, that’s just natural, easy way to, to have that. I like those little features where it’s, it’s so powerful, but a small little thing. Yeah, yeah.

For me, I mean, two answers. Okay, don’t yell at me. I, I’ve always been doing a lot of opinions, but it’s so, it’s been so great to see kind of some of our partner teams, like all the foundation models work, the evaluations framework. It’s also cool. And I was a little less connected to that. So I have a long flight home and I’m going to be watching a lot of videos tonight. And you can flight your, you can find your number. I can, I can, but the one thing is app intense testing.

We haven’t talked about it today. App intense testing is a new way to kind of unit test your app. Intense. So traditionally you might build app intense. And we would say, you know, you want to test it with Siri, you want to test it with shortcuts. Yeah, but what was missing is a way to ensure that this functionality was working in kind of your CI pipelines and any type of automated way.

So Venkatesh has a great talk this year on it. But for testing your app intents, testing your entities, the queries integrating with spotlight. There’s a ton of features there. We’re really excited for people to to dig into that. And it’s not just unit tests, right? You can test UI integrations, you can test spotlight integrations. It’s a really valuable framework. Yeah.

I love Venkatesh’s session, how he’s he’s got the spacesuit. You got to watch the video. Yes. Really cool spacesuit spoilers. Anyway. You have to watch the whole thing. Don’t just go to. The yeah, just, you know, see the whole thing. A lot of good stuff there. So cool. Robert, what. Would you say? I’m going to borrow from James here. It was really interesting to see what a lot of teams did.

I foundation models with kind of, you know, bringing in different language providers, but then also like to see that integration go out to say core AI. And now I can just bring in any models from the web, but it’s like I could still use foundation models. Like it’s, it’s like this central API. That’s really neat. And then yeah, you know, I’ll plug our evaluations just a little bit.

And if you’re using it with foundation models, it’s really easy to bring in evaluations because, you know, like James was saying, testing for these things is, is really a nascent thing. Like, I don’t, you know, a lot of people talk about it. A lot of people don’t know how to do it. And I think we’re, we’ve taken a lot of steps this year to kind of say, all right, how can you think about this feature instead of just assuming the model is going to work like magic? Yeah, yeah.

That’s great. Louis, what would you say? Let me guess. Foundation models. Yeah. Well, I mean, you’ve mentioned some of the other things already, so yeah. But and we talked about like image input support with the on device model. But but the big one I think is private cloud compute, right.

That developers now get access to because we know people have been asking for it. You’ve been asking for it. Sure. Since last year. Yes. And so yeah, this year, new server model running on private cloud compute. You can easily access it through our API. And honestly, this is this is like the perfect time to talk about my new book. Actually.

If you don’t mind. Yeah. Don’t worry, I’ll explain. I’ll explain. It’s not a real book. But just to talk about the benefits here with private cloud compute, the real thing is that you don’t have to worry about API keys or anything else, or authentication, things like that, like with typical server models.

Right? Right. And so, you know, just, I mean, this is a joke, but like, it’s real where like, it’s just one line of code to connect to a server model now, right? And so by the way, in the video that we have, I did a little like. Yeah, you throw it here. Right?

It’s like drive the point home that like, you know, it’s, it’s easy. You don’t have to worry about API keys. And yeah, so I think. Everybody back house just freaked out. They were like, oh God. No books were harmed in this. No. Okay. So there you go. Yeah. That’s great. Dan. Let me guess. Oh, no. I was going to say that’s a tough act to follow. Did you also bring a book? Yeah, I did not bring a book. I missed a memo, apparently.

But yeah, no, I am really excited about the new Siri AI. I mean, we have done a lot of great advancements to Siri, and we have a lot of great APIs this year for developers to adopt and bring their Bring Siri to their app. I mean, I think it’s just really exciting and I can’t wait to see how we’ll eventually get to this ecosystem where you have all of these apps that are deeply integrated with Siri and Apple Intelligence providing content to the semantic index so that, you know, people can ask natural questions about content from all these different kinds of apps and get answers instantly.

And then when you integrate with intense schemas, Siri taking actions on a variety of apps across many different domains, I’m just really excited to see sort of like the bicycle for the mind. You get this ecosystem that is deeply integrated in Siri, taking action across many different apps. I want to change my answer to that, actually.

It’s cool to see also like I’m a little far removed, but how much work you guys have put into that and how natural it becomes when you’re the end user. It’s just like amazing to me. I think that is so. Powerful, large. The illusion of. Simplicity from a lot of folks.

Yeah, definitely. But I can’t wait for all my favorite apps to adopt our APIs and then just be able to have Siri, you know, perform all these actions. Absolutely. Very excited. Incredible. And I’d say for me, all of it is really exciting. You know, one of the great things about being an AI machine learning evangelist is you get to see all of these great things, you know, and get exposure.

It’s so I’d say kudos to everyone here at the table and all your teams and the people behind the scenes who made all these features happen. It took an incredible amount of work. So yeah, kudos to you all, but great stuff. All right. Well, thank you for that question. Next one is from Gerald A. Let’s see, Gerald Dash go to capital A like he’s an A student anyway. Sorry. They’re all students. Thank you. For. Your question. Names will be anonymous.

The question is why? Again, going for hard coded schemas instead of choosing a dynamic approach like GPT or Claude are taking. I’m thinking about markdown. Describe skills that can reference app entities. Wouldn’t this be much more flexible and solve a lot of the other questions mentioned? That’s a great question. I’m loving all the schema questions. Keep them coming, by the way.

The cool thing about what we’re building here and the ecosystem that we’re going for, is that we’re striving for consistency and privacy. Like our Apple’s take on Apple intelligence is that it’s personal intelligence and private to, to you and the people who use it. And so with schemas, we’re able to effectively guarantee an experience that is holistic but also standardized across the platform.

I think one of the things that is cool about it is, you know, if you’re using Siri with an app that has adopted our schemas in one of the domains, like I’ll just pick messaging, for instance, when people interact with Siri and the messages domain, they’ll get an experience that will naturally transition to other apps that adopt schemas from that domain as well.

This way, you get a consistent behavior and interaction with the system in a way that’s like once you learn how to interact with Siri in a specific domain, you will just feel natural integrating with all these other kinds of apps. And that’s what’s really cool about the experience.

And also, you know, schemas provide sort of like a security benefit as well, which is like, you know, in certain situations, if you’re sending money to somebody, for instance, you probably want to confirm before you’re actually just having, you know, Siri send money to someone. And so with schemas, like you get all these security features built right in so that you get a consistent, safe experience on our platforms. I think Antonio does a great job in his talk describing this around our kind of new.

We have a new kind of enter entity ownership API, you know, for certain, you know, items in your app, certain entities, they might be shared with other users, you know, a calendar might be your own calendar or event, but it might be shared with folks. And by using these schemas that we know about that we can reason about, you might get different behavior where if it’s just your event, maybe it gets deleted, but if it’s shared with other people, then maybe we want to add that confirmation. So it’s these kind of things that we really feel like, is this this powerful where the schemas give us this power?

So all those edge cases that we’ve talked about, right? Yeah. You know, we want to provide for a lot of folks, this will be their first time integrating with these types of systems, building this way. So we, we always want to be as safe as possible as we’re building these APIs. Yeah, that’s how you build trust and safety, right? That’s right. Key to. It. Neither of you mentioned localization, but I know a lot of the reasons we do this is to kind of integrate in support for localizing this in other languages.

I know we haven’t talked explicitly about, you know, plans, but I don’t think it will surprise anybody to know we’re trying to get this out to different. Yeah. That’s the power of the API, right? Like Apple has done the heavy lifting in terms of providing all of the model training the natural language strings.

And so when you’re talking about localization, Apple can scale all of these integrations across many different locales in a way that you develop, or you simply adopt our API and we take care of all that heavy lifting for you. Yeah, we will make, we will try to make this better. You know, we’re going to continue to make this better. Like that’s, that’s our that’s the promise there.

Yeah. No great stuff. Great stuff. All right. Well, Gerald, thank you for that question. Next question is from Nia Murphy. Does the evaluations framework have a way to see if my tools are being called? I know. That one. Yeah, I think we we talked about this a little bit earlier. You can supply the transcript from your language model session that you’ve configured, and then use a tool called evaluator in your evaluator process.

And actually then you just provide in your data samples the assertions you want. We have an expectation system where you could say, you know, I want my tools to be called where they called. Were they called in the right order? Yeah. Where they sent the right properties were the values that were sent to those properties. Like you could go pretty deep, right, with this.

It’s a pretty sophisticated assertion system. But, you know, really you supply that at the data sample, like the scenario level. And then our evaluator takes care of the rest. It looks at the transcript, it finds out all that information and it just gives you a report. Yeah, that’s that’s really awesome. And again, you know, having worked with developers at workshops where they’re implementing some of their first tools, you’re like, wait a minute, it should have been called this way and it wasn’t.

And you know, it was kind of a source of frustration, frankly. Yeah, authentication is a big one. Yeah. If you if you need to access something that requires an authentication, you want to know that, you know, and the model won’t always know that or the model won’t know, oh, I have to call this tool before this.

Totally. And, you know, the importance of really giving the LM good guidance on when that specific tool should be used, right? Dynamic profiles, dynamic profiles. We haven’t mentioned that yet. Yeah. Other great stuff. So yeah, that’s a great feature. And yeah, thank you, Nia Murphy for that question. All right. Moving on to the next one here proton stirrer. I like that proton stirrer. I don’t know. Yeah very cool. Superhero. There’s a person with a cape you know. Next year anonymous. Too much time is being spent on the names.

How does the Apple intelligence framework mitigate semantic entropy when passing context between on device models and larger server based models? That lives up to the username? Yeah. I it’s funny you just mentioned dynamic profiles. That’s a foundation models framework, I think is what it’s alluding here. But yeah, it’s a great question.

Like we, you know, we have the system language model that has a 4K context size and we have the PCC model that’s a 32 K. So when you’re using dynamic profiles, which quickly over view of that is you can with the same transcript switch between specific profiles. Maybe somebody is specific to accounting and they do all those things, but you want to have another profile that’s doing something health related. You can switch between those. It’s like agents, right?

Yeah, exactly. And in order to kind of keep the, the, the context sharing We have modifiers in this declarative style like history transform to say, hey, I switched to this different profile. Maybe I want to throw out some tool calls. Maybe I want to change something and keep those balanced.

And then another great thing that we added this year is the foundation models utilities package that has ways where you can do like summarization and maybe compact the transcript if you’re going to a smaller size. So there’s all these tools. Another great way to test that is with the evaluations framework. Yeah, we very well together and be able to see what works best with your use cases and tailor that.

Yeah. Awesome. Great. Thank you for that question. Okay. Moving on Scott is the username. Do I need to use the evaluations framework with foundation models? Oh it’s a great question. As I mentioned earlier, I think when we were talking about academic research, you can obviously use a language model. You don’t have to use the foundation models framework to do a lot of the evaluation.

We just talked about tool called evaluator, which is one example where if you want to use the tool called evaluator, you do need to use foundation models because we need that transcript. We’re actually working to kind of do a more generic transform from other models such that if they’re using model provider, they’ll get that transform. But by and large, no, you don’t. You can evaluate any stochastic system. Yeah. With that. Yeah. That’s great.

Great stuff. Thank you for that question. Okay, next one from drop in message. The question is I have multiple apps that will all donate entities via index entity to the Spotlight semantic index. Can one of my apps Foundation model sessions retrieve another of my apps donated content via spotlight search tool same developer, shared app group, or is retrieval strictly sandbox to the donor?

Yes, retrieval is limited to your app sandbox only, so you wouldn’t be able to. Since they do mention app groups like you could do interesting things, presumably where like like two apps from the same developer can be in the same app group and they can share files right on this can share some data, not like literally true that sandbox, but like you could probably like find ways to maybe share things. Yeah. So the way we think about that in app intents is with the transferable protocol.

So that is kind of the best way for, for doing this kind of interop between different file formats. Essentially, you know, you can transferable works by a kind of progressive from highest fidelity to lowest fidelity, you know, data format. So if you were both to share a similar data format, you can use transferable to go between apps.

And the cool thing about that API is that you’re essentially opting into the data transfer. So it’s done in a private and secure way because you’re essentially saying like, I want to share maybe a small projection of my app, but not all the contents of my app. So you, the developers, always in control of what gets shared with whom.

Yeah. That’s great. Awesome. Great question. Thank you for that. Okay, next question again from looks like John with a. That’s an easy username. Yeah, J. Here we go. Schemas our schema bound entities interchangeable. Can an app’s file entity be turned into a photo entity and vice versa? I suspect this is possible through transferable. You’re right on transfer. Plus one. That’s higher. John.

That’s the cool thing about transferable is that you get to specify all the different data transformations that you’re, that your app can can support. So like, for instance, in our sample app that’s available that you can download today, unicorn chat, we adopt transferable in the app in a way where you can have messages that could be, you know, text, but they can also contain attachments. And so the app supports transferability, exporting text, but they can also export files.

You can also export photos. And so depending on the content you can even provide transferability in a dynamic way based on sort of the payload type. And that allows you to export your content in a way that can integrate with different apps based on the capabilities that an app supports. So when you export a message as a photo, it means that this photo can now be imported into the photos app, but it can also be imported into any other third party photo app that adopts our photo schemas.

Yeah, there’s one more note there. There is a file entity protocol that works for all app entities. It’s designed for when you need these these file based formats. So read the documentation for more information there. Well that’s. Great. You know, I hear unicorn chats like moving up on the app top app list.

Here we go. Ain’t that right? Yeah. It combines our technologies. Yeah. No that’s. Great. Well, thank you for that question, John. Okay. Next question from. Zulfi Shah, I believe is how we say that in my app, I adopt app intents and app entities for various concepts in my app. Some of these might map to app schemas, though not perfectly. For example, I manage a list of contacts and a close match would be schema from dot messages dot messages person. Is that acceptable to participate in new Siri?

Yeah, absolutely. I mean, the answer is definitely yes. The cool thing about this API is that you can pick and choose the functionality that your app supports and its capabilities based on the app schemas that we support. So, you know, you, you mentioned some kind of contact integration.

We have schemas for message person, for instance, where you can export contacts from your app to support some messaging features. In fact, unicorn Chat uses this and you can check out the sample code to see how we model content in the app using the schema to expose this information to Siri and Apple Intelligence.

But you know, if if you can’t find any schema that sort of fits into your app’s functionality, we have the more generic system search capability so that you can integrate with Apple Intelligence and expose your app search capabilities. It’s great for apps like e-commerce, food ordering, and many other types.

But then you can always use app shortcuts as well. As another way to essentially integrate your app intents in a way that can be more generic. Now, of course, because app shortcuts are custom, it means that you have to provide sample phrases that match your app’s experiences and the scenarios that you’re thinking about.

And that’s the cool thing about app schemas when you integrate with a schema. Apple is doing the heavy lifting, so you don’t have to provide those phrases. That’s great. James, anything you want to add on? I think you covered it perfectly there. That was awesome. I’m glad people are kind of asking this question though. It’s come up a few times.

So, you know, I think it’s it’s really cool that people are thinking about this. Yeah. Well, this. Is good stuff. I can’t wait to see what people build with these integrations and these APIs. Very excited. Yeah. Thank you for that question. All right. Almost out of time here. So let’s see if we can get one more in here from our red Hakurei shrine.

Apologies for stepping all over your username there. We already know that with image processing addition to foundation models, the token limit is still 4096. Do we need to be aware of any constraints to our other inputs, like prompt or instructions? In the case for the model to function at its best.

I mean, with the on device model you have 4096, right? With the image input support, I mean doesn’t really change anything. If you put an image in there, you can measure this with instruments. With the support we added this year, but I think it comes down to like about 200 tokens or so for, for, for an image typically.

But in terms of like your instructions or anything else, you can take advantage of the full 4096 if you want to give it 4000 tokens input and generate a, you know, really small output that will work just as well as, you know, giving it a small input and generating large output. Right. And then with private cloud compute, computing. Just for clarity, you get 32 K.

Yeah. Context size. Right. Yeah. That’s of course much larger. Yeah. Much, much larger. I think too, in 26.4, we also shipped token counting APIs. So you can play around with that to, to see what’s working. But also, you know, depending on your use case, you kind of want a hill climb. Yeah. Right over here and over here.

You know, you might have a very specific use case and tailoring that prompt and making it brief, maybe with that smaller context size and pulling in evaluations to do that. Climbing can also help with that. Yeah. The last thing I’ll say with the on device model, we get a lot of questions about, oh, it’s, it’s not as capable because it’s smaller and it’s like, it’s actually, you just have to be more precise with your instructions. And sometimes it’s finding the right synonym to use. So you might try different variations of words.

And to your point, hill climbing is a perfect way to just kind of benchmark that and see which variation of the prompt works best, but you could get a lot out of the device model if you’re willing to put some time kind of massaging your prompt. I put examples in your prompt as well, right? Yeah. Oh yeah. Give it examples. You know, give it as give it as much information as you can. Yeah.

Well great stuff. Well, unfortunately that’s about all we have time for today for this group lab. We’re really thankful for you taking the time out of your busy schedules to join us. We really hope you found this to to be really helpful and beneficial. You know, we also want to say thanks to our panelists. Thank you all. I know it’s been a really busy week and the release and everything else you have going. So thank you.

And yeah, just want to say thanks again to all of you. As we mentioned earlier, if you go to developer.apple.com/forums, please post your questions, your code questions and so on. We’re, you know, we’re there to help you any way we can. And of course, feedback assistant, right? If you have ideas and if you find a bug. Okay, I’ll be a. First, but still.

Hear about it. But yeah, exactly. But by all means, take advantage of the resources. Some amazing videos this year. A lot of great technical documentation. And yeah, you know, we, we really appreciate all your feedback and insights. If you’d like to see more of these kind of group labs. You know. Or.

Write more. Books and other things, please let us know that as well. We really want to help you be successful. And thank you for being part of the Apple community, the Apple family. We really appreciate all your time and great work in seeing the innovative things, apps, and new technologies you come up with. So thank you. And yeah, with that, thanks again for joining us. Have a great rest of your Wwdc. And we can’t wait to see what you build. Yes, absolutely. Thank you all.