Hey everybody. Welcome to the machine learning and AI group lab. I’m Shashank. I’m part of the evangelism team here at Apple. And today I’m joined by a fantastic panel of experts from across our machine learning framework teams. This has been an amazing WWDC, a lot of exciting machine learning and AI announcements. We had the new core AI framework, new evaluations framework, a slew of exciting updates to the Foundation models framework, and distributed training and inference with ML, and so much more.

So to kick things off, we’ll start off with a quick round of introductions. And as each of you introduce yourself, please share one thing you are most excited about in your area and one thing you think the developers should be most excited about. How about we start with you, Tom? Hi, my name is Tao. I’m from the Korea team. I’m super excited to be here today.

One thing I’m super excited about this year is the new AI models Open source on GitHub. You can find out a curated set of ready to use model exporting recipes and reusable Python primitives to bring new models to core I, as well as Swift runtime utilities for you to integrate those models in your app. Last but not the least, there is a set of skills that you can leverage using your favorite coding agent. Exciting.

Awesome. Hi, I’m Marcus, I’m a manager on the Evaluations Framework team. I’m super excited about the being able to release this framework to you all and being able to let you guys get started with actually evaluating your ML powered feature. But the thing I think that I don’t think all of everybody knows about yet is we’ve also shipped an update in Xcode, which allows you to get a report back of your evaluation runs.

I’m personally really excited about this because it’s been really, really helpful in terms of getting people to actually increase the scores of their evaluations using really rich data and really rich feedback also is a little bit of a like selfish note. I wrote my first program in Xcode, and now it feels really, really cool to be able to actually contribute something to the thing that I learned to program on.

Hi, I’m Michael, I’m also from the core AI team. Tao stole one of my my exciting things, but I’ll repeat it. I think it’s really exciting now that we’re, along with our frameworks and open source tools, right? We also have skills to put your AI agent. I think that’s really transformed how a lot of people worked, especially me.

I’m not very familiar with PyTorch, but I can quickly go and ask and have a conversation about, hey, help me make a plan of how to export this model or what would be a good strategy for testing it and verifying things we’re working, but I’ll put a separate plug in for. We have a more advanced feature in AI this year about being able to compile your models ahead of time, which really helps with large, complex models, particularly on iOS. So we can sort of get the load or first cold start experience to be much quicker.

Awesome. Yeah. I’m Louis, I’m on the Foundation models framework team. And yeah, there’s so much new stuff this year, but I think the new image input support that we have with the on device model and also the new one on private cloud compute is definitely one of my favorite features. Yeah.

Cool. Great. Yeah. I’m Ronan from the team. And yeah, a couple of things which are exciting. You know, with this year, I would say is like the GPU neural accelerator support that we added, you know, for the M5. That’s really amazing in terms of performance. And, and then the Thunderbolt 5 Rdma support, which allows us to run like frontier models over a cluster of Macs. Right? It’s really, truly amazing. If you want to do advanced athletic tasks, you know, on a couple of Macs.

Really cool. I’m sure we’ll get to all of these through your questions in addition to this panel. Thank you for introducing yourself. We have a whole team of experts behind the scenes triaging all your questions, so please keep them coming. We’ll try to get to as many as we can today.

If you do have code specific questions or we can’t get your questions for some reason, I encourage you to head over to developer.apple.com/forums and ask your questions there. We’ll continue to answer them throughout this week and in the future as well. If you do have a bug report or a feature request feedback assistant.apple.com is your friend. Please go there and share the extra information that we would need to help you. And we can keep this discussion more applicable to a broader audience. All right, with that, let’s dive into your questions.

Here’s a question from Phanteks. How should developers decide whether a task is better handled by on device foundation models versus private cloud compute. Looks like we are starting off with you, Louis. All right. Yeah. I mean, yeah, when you’re deciding between any models, I would say doing that based on evaluations definitely is the way to go about it. But just to spell out like the, the concrete differences, right between the on device built in system that we have and the new server model this year that you have access to on private cloud compute.

One of the major differences is of course context size. So on device you have 4K. With the server model on private cloud you have 32 k. So depending on what kind of feature you’re building that can influence your decision. The other thing is of course that with the on device model, it’s available offline.

If that’s what you need for your use case, obviously that’s a benefit. And the other thing with server model is that it supports reasoning. It’s a new capability with server model on private cloud compute, where you can set the level of reasoning to let the model really like think to itself before it generates a response.

So again, depending on like what kind of feature you’re building, that might help a lot. Right. But in terms of making that decision based on data, like honestly, like evaluations with the evaluation framework. Yeah. Yeah. I think a lot of what I would consider in this situation is actually being able to do comparative evaluation.

So what you want to do is take your model feature. What’s really lovely about the foundation models framework stuff, right, is that you can sort of switch between on device and private cloud compute with a simple line of code, right? It could be an if statement. So therefore what you can do is basically like take your model feature. Maybe it’s a service class of some sort.

Instantiate that one with private cloud, compute one with the on device model, and then kind of run them against the same set of evaluators that will allow you to actually run them as a suite of tests. And then when that runs, you can go to the report and actually check. We’ve actually also built in some really nice comparison tools into Xcode that actually allow you to actually see side by side how those things stack up against one another based on your evaluations.

So this is actually a really great way to kind of figure that out. And then as you kind of chop and change and make small tweaks, you’ll be able to actually understand, like in what cases do these things work better for one another? And like, what you might also consider is that maybe if you find an instance where private cloud compute is the better option, it’s still knowing that, like if you don’t have a network connection, you have to fall back to on device.

Just also like how well your feature will perform in those scenarios. Yeah. Pretty cool. Thanks, Marcus. I’m curious actually, like in terms of latency, do you have a take? You know, if my app really requires low latency, should I go with like the local model or the PCC model?

Yeah. That’s interesting. I mean, yeah, it comes down to measuring it presumably, but with the it’s true that with the on device model, you don’t have the network latency. But if your network conditions are good, then I mean, typically the round trip to the server is not the expensive part necessarily. So in terms of like tokens per second that you’re getting out of it. Yeah, that might not really influence the decision too much.

Surprisingly. Yeah. That’s cool. I will plug in that. The instruments this year, Foundation Multi Instruments has been significantly improved with a whole slew of extra metrics, things like round trip latency, token budgets. There’s many more. Right. Yeah, exactly. All of this you can measure. And while using evaluations framework to figure out what works best for your application in terms of, you know, snappiness for the user and stuff like that.

Awesome. One small thing you mentioned, Marcus, which I thought was interesting, is you said you can use if else to switch between private and on device. Louis we have dynamic profiles now, right? Yeah. Very quickly. Do you want to just share? Yeah, actually because I mean the question here is asking like, how do I decide between the two models? And maybe you want to use both.

And you could use both in your app for different, you know, well within the same feature, there might be things where you start maybe with the on device model for some routing, and then you only use the server model for for certain more complex things. But in that feature and yeah, dynamic profile is this new API that we’re adding this year to let you more easily use models together while sharing context and doing handoffs, all, all sorts of stuff, right?

We have a really great video about that as well this. Year. To learn all about that. So yeah. Yeah. Thanks. There is a video called a Dynamic Agent experience with dynamic profiles. Something, something along those lines where there are examples and we talk about how to use these. So be sure to check those out.

Great. Next question. This is from Pichaya. I successfully set up agent mode in Xcode to local LM using open code ACP with config model to ML community Quinn 3.54 billion ML eight bit version using ML server. But the way Xcode communicates to model still acts weird long time and has the I am underscore end tag. What is the what to config more? Did I wrongly set the chat template?

I, I don’t know if you want to take it. We don’t have a Xcode representation in this panel today, but we can probably talk to the ML side of things. And you’re probably on the right direction with the chat template. Yeah. But I think we do see this kind of ending token when we’re bringing up the query language model, similar as ML language model, which is a foundation model, language model, protocol conformance with package.

So when we see such kind of things, generally we look into the tokenizer. Look at your configuration of your model, see if this token is actually correctly recognized. Maybe there is something that we could configure on the ML language model side. But that’s a very common issue when you bring in new language models.

Yeah. So I would say that, you know, we have actually a lot of good experience using open code on its own with this type of language models, so it should work in practice right now. Maybe it’s a configuration issue, I don’t know. I know there are several ways as well to communicate with language models from Xcode. So there you know, you know better than me.

I mean, I think you have like the chat mode, which is the kind of old way I believe, and the CP mode, which is, you know, the way to go. So we, you know, it should work in some way. Right, right. So maybe the best would be to post in forums and. Try. To debug that, you know.

Forums like debugging stuff that they should be able to like pull for that. That’s helpful in this case, or is just kind of saying like, hey, this is what I’m seeing is enough. I would say what I’m saying is enough because, you know, we know that it should work with. Right? Yeah. Yeah. I would think like. We have both like ML X language model and the language model, open source. So you could post your question there. And then someone should be able to help on the GitHub. Yeah, yeah. No thank you. And we have a session this year.

ML where we demonstrate the presenter demonstrates plugging using Xcode with open code. Specifically, we do use a model. I’m not sure if this this specific model, but it does seem like a configuration related to templates or GitHub issues is a good place to share this or forums to, and we can get. We can get you taken care of. Thank you.

Now this question is, is it possible to use local LM running on ML LM using downloaded model or the one we created ourselves trained ourselves using core AI to be to be coding agent that can manipulate the project more, not just a chat one on Xcode. How do I do that? I think there are a couple of different questions hidden in this question and maybe a few corrections. Maybe we start with core AI, which is purely an inference framework.

Sure. I mean, I think this question I’m getting in terms of is the question, if they’re trying to integrate it into Xcode. Xcode. Right. And so they’re, we’d probably refer to you as sort of on Xcode and ACP protocol in terms of integration. And the other one is, can you provide your own model? And you sort of pointed out in the question both for ML and core. Yes, you can provide your own model. Try and understand the aspect of the question of I see manipulate the project more.

I think that’s more of the capability of the model and what it has access to, which may, again, if you’re going through. Xcode. Is through the Xcode. I think it relates to the chat mode, you know, versus the agentic, you know, way. And, you know, I would say the Agentic way is the way to go. Yes, I think that’s what we’ve heard from the Xcode team in general. Yeah, exactly. Yeah, that’s my understanding as well.

Yeah, I think I misunderstood the question as saying training with core I, but the reference is it’s trained and we want to bring it in. Right? Yeah. And yeah, we’ve shown examples in sessions where if you run a ML server, Xcode can automatically detect all the models on device, right? And you can pick one. And this year we have chat mode and the ACP mode. We have some demos and examples in the session. And I believe those examples will be on GitHub. So yeah, do check them out when you get a chance.

Anything else from the panel? Okay. I’m curious actually in terms of like plug in models, you know, I mean, I guess it differs a bit from the. So I will provide the core inference just like ML. You know, there isn’t, you know, a server component on top of that, we have open like you have for the ML one. But you know, any, You know, you’d have to write that conformance to the to the protocol right now. Makes sense. Somebody has to contribute or have like a OpenAI compatible language server layer on top, which is. You’re interested in that please.

Yeah. Make an issue on GitHub. Yes. Or contribute. Or contribute. Right. Maybe you want to build a language server that other apps can use. And developers have options, right. Options are good. Sounds good. Next question. This is from Robin in Apple updates the on device and PCC model silently and tune prompts regress when that happens. Is there a model version identifier exposed at runtime? So my evaluation suite can detect a model change and gate releases on it. What’s the recommended way to know model changed?

Yeah. Well, yeah, I mean the, I mean, the best thing you can do right now is check based on the OS version, right? Because with the on device model, that’s obviously part of the OS, right? The on device model really only updates as part of an OS update. So it’s not like we push out an update, a model that silently would like, you know, affect your, your device without updating the OS, right?

So at least you have that control, right? But the server model is more complicated. Of course. Right now what I would also say is that again, honestly, like going back to evaluations where like assume that these models will update over time both on device and server, right? So whatever prompting you’re doing that, that yes, like over time, it’s going to change, you know, the behavior that you get and that with evaluations, you sort of gain back that control. Yeah, exactly.

So when a new OS update comes out, like if you can rerun your evaluation and see sort of how it affects. It should be part of your test suite. You have this like, again, this comparative situation that I was talking about before you could run it and then look at a, at a run that ran on the old version of the OS and the new version of the OS and see if there’s like a, a notable difference, right?

And then kind of from that, you can consider how you want to proceed. Maybe you, you know, prompt engineer for a different version of the OS or, you know, add a tool or something, whatever makes the most sense for you from, from there. But the idea is that like, yeah, you should be sort of able to detect it and kind of create a plan. Yeah.

And generally our prompting advice is to not get like two hyper specific with specific words. So you don’t really want to end up in a situation where you have any sort of if statements where if you’re running on, say, 26.0, that like it uses a slightly different sentence than on 26.4 than on 27.0. Ideally, you can just rephrase your prompt or just be more specific where it sort of gives you the quality you want across the versions.

Yeah. Yeah. I had a related question actually, like, do you plan to do updates more like on minor versions or like major versions, you know? Yeah, yeah. With the, with the on device model, like looking at the past year, for example, we had two updates. Oh, I see. And so typically and so like twice a year, typically we have a big update. Okay. Yeah. Okay.

I think that’s a good question because the updates happen in the beta releases, right? So you all have a chance to test it during the beta period. Yeah. And if you make sure that using evaluations it’s behaving as you want. So once the general availability happens, then you expect the bulk of the audience who actually use the apps for it to work.

Yeah. Like for example, now we’re like in beta period 27.0. So, you know, definitely like try it out for your use case or if you have new use cases, even with like the image input support we’re adding this year, we’re looking for that feedback, right? Like we evaluate the model, of course, before we release it in beta, but it’s hard to know sort of even how your developers like how you use the model, right? So please tell us if like you have a use case that’s maybe not working as you expected with the model during the beta period.

Yeah. I also want to emphasize the positive part of it is. So the question is in terms of regressing. Right. So the advantage of the model updating over time, right, is it can get better. Yeah. So so that also is a trade off between having it fixed. That’s why that’s also why you also discover evaluations positive things. So I just want to yeah. To be clear we update it to make it better. Yes.

Great. All right. Next question. This is from Eric’s questions. Can foundation models run from widgets app intents or background tasks? We want AI classification at capture time even when the app isn’t in the foreground. Are there rate limits or thermal rules we should design around? Yeah. With the on device model, it can run in the background and in widgets.

But keep in mind that based on the system conditions, because there’s so much else that can be going on on your phone, of course, that you may be rate limited for the on device model. And really that just means that you should try again later, right? Because if something else I mean terminals or whatever is going on in the device, right, that the request may fail and it will throw a specific error that you can catch. We have an API for that. So, you know, if if it was for that specific reason, and then you can just try again later, either in the background or in the foreground, right? So don’t show the error to the user, handle it in the app. If it’s in the background, maybe that’s difficult. Right? Right. So actually technically apps compete to, to access to the language model on device language model, right.

But what would the on device LM is sort of managed by the system where if two apps want to use the on device model, it’s only in memory once, right? Like the system is very smart and like managing those resources, but. Do you give priority to the foreground task or is there like.

Yeah, like if something else expensive is happening, like someone’s playing a game, for example, right? That’s like resource intensive. Yeah, that’s going to take precedence because obviously the, like the user has more control over what they’re doing in the foreground. They want to play that game. They want the optimal experience there.

If it just so happens that in the background, an app is trying to do something with the LLM that feels more like it could happen later, or it’s like deprioritized. Makes sense. Louis, is there a nuance there on what platform macOS versus iOS? Actually, yeah, a good point. Yeah. On macOS, that rate limiting does not apply at all. Foreground or background? Yes, exactly. It’s really just on iOS that if it’s running in the background.

I think it still respects the sort of quality of service setting in terms of your requests of where they’re going, the system overall will respect. Right. It’s going to prioritize user interactive threads or shows in terms of when you’re doing over, let’s say, background or utility. Definitely. Even I think on macOS. But on macOS, it’s not going to. I guess you’re saying it’s not going to rate limit you on that. Yeah, exactly. Thanks.

Great. Good discussion. This question is from Anne Scratching. How can developers create or adapt models to their own apps. Data and domain ideas I have are creating user profile image from existing images, but with a style theme like Cartoonized etc. or an app. How to guide that adapts as new app functionality is created. Best resources for exploring. This sounds like an opportunity to bring in a custom model or Micro. Yeah, I think I can take that.

It sounds like the the latter part of this question. You’re trying to bring an image generation model of some sort to generate this user profile images on the fly. I think that if I were to, you know, speculate what you’re trying to do, maybe the best way is to use the foundation model to basically bring the user data of your app into your prompt and generate some sort of like a per user profile capture from those app usage from that user, and then use this prompt to generate the image using your for your image.

User profile image. So you know if you really want to go one step beyond. Even like bringing other, more powerful image generation models, you can consider even fine tuning a open source model from like one of these open source hugging face or GitHub and then bring that model into your app.

Yeah. I think in this world of these powerful foundation models, like user adaptation does not necessarily require for you to fully fine tune or adapt. So I’d say that would be the extreme case. So the, the, the question is, can you achieve yours with sort of customizing or extracting information that will be useful to prompt, you know, either the on device model or a custom model in terms of bringing in that sort of customization? Yeah, yeah. It’s I agree with that. I would try, you know, to, to make it happen with the tools which are provided, you know, with us first.

And if that doesn’t work out, then go, Yeah. Cool thing with local models is that data stays in the device and you have access to with user consent. Other personal data across the Apple apps ecosystem. And then what I think some people refer to as In-context learning, you provide it to the model. But if there is a reason for you to fine tune and train your model, you can. ML is a great place to start. And yeah, obviously we, we, you know, we support like fine tuning and, and training recipes, you know, for all sorts of models.

I mean, many models are supported by the open source community actually. So if you know, that’s basically the end of, I mean, other end of the spectrum route, you know, compared to prompt engineering, I would say. Right, right. But yeah, it’s another way. And if you do that, then, you know, it’s actually easy to, to make the model available in Korea, then just run it, you know? Yeah, yeah.

So one of the other things that like, I also want to throw in here is that this is also a really great chance to do what we call hill climbing, which is this process of like using your evaluations to sort of like progressively increase the quality of your generations, right?

Because as you start this process, you’re going to be like, I wrote this prompt and this prompt kind of got me here, but it didn’t quite get me exactly what I was hoping for. And so maybe you’ll make a small change to your prompt or add additional tools or, you know, maybe swap out a different model and, or, you know, maybe you go to actually go down the fine tuning route and actually fine tune, fine tune a model, right?

And you want to see how those changes kind of like over time affect these things. And one of the great things is that like, even if you are doing this with image generation, the, the evaluations framework is, is built up to handle any of these sort of options, right? You can bring any model to it. And then from that, you could use like a model, like a model judge evaluator, sort of say like, hey, is this image that got generated in the style that I expect?

Right? Is it kind of vibrant and cartoony, if that’s kind of what you’re aiming for or kind of whatever else have you. So this is also a really great opportunity just to kind of like use those evaluation scores to kind of like guide you in the process of like making these small incremental changes in a way that like, you know, is getting you toward where you want to be. Yeah. Because one of the harder questions that I’ve heard people ask is more like knowing the size of the model that you need for your task. So let’s say you are going with a custom model that you want to run with core AI on device, right?

How do you know the size of the model that you need? For example, for this for generating profile pictures? Right? Yeah. So in that respect, actually there is maybe another alternative, which is like a training, a very small adapter per user, you know, and they are actually, I’m not super knowledgeable, you know, but I, I remember there is a feature to support adapters with like the adapters. Yeah, but that’s not for diffusion models, right? Oh, I see. So yeah. Okay. Yeah. Right. But but but it’s still, you know, you could, you could imagine it on top of like the text part of the foundational model.

If you extract some features beforehand, I don’t know how. Right. Yeah. I think with all of them, I guess I have a question for you in terms of evaluation. So does the model judge. But in terms of how do they simulate user data? Like is there a way for or a recommended way for them to generate kind of data that they would say, oh, let me, you know, can we use the foundation model framework to sort of generate the evaluation data to say, oh, pretend you’re a user who is into nature and other things like, you know, generate content so that I can test based on that content. Do I generate a prompt? Yeah.

So in the evaluations framework is a type that we call the sample generator. And so it is built for taking any text based input and producing kind of synthetic style output. So you can bring any model to that. And then you can go ahead and just say, I want this level of output, this many of them with this kind of variety, and it will go ahead and generate that for you. Now this is it is text based.

So the limitations kind of do kind of limit around like you can’t have Deep Fusion style output, but I think you could build a really interesting pipeline where you use the sample generator to generate these like incoming prompts to your diffusion model, and then kind of run the diffusion model to sort of generate a number of these sample images.

And then you sort of if you’re like, yeah, this is a good one and this prompt is good, then you maybe save that prompt and image pair as your input for your evaluation and then kind of continue to move. And you have this kind of like working pipeline of kind of like a large, diverse data set. That’s basically what I was wondering about. Is it possible to do that? That’s great. It’s very. Yeah. Yeah. Very interesting. So the evaluation framework supports not just language models.

So you can have diffusion models in there. Yeah. It’s built with the idea that any any we built it with the idea around any system, with any sort of randomness can be evaluated through this thing. Right? So if you want to go as like, you know, after diffusion models, you want to build like a linear regression model you want to do.

I think we were. I think you and I were talking the other day about like, upscaling, like if you wanted to do like GPU upscaling, you could do it with that. As long as you have a way to sort of like turn that result of your model generation into like a boolean or a scalar number that you can actually kind of evaluate against and do some statistical analysis over. We support it. That’s pretty cool. Very nice. Thanks. Thanks. Next question from Phanteks. What types of use cases are currently best suited for foundation models framework and which use cases should developers avoid? Louis.

Yeah, I mean, it’s a large language model, so it can do so much, right? Whether it’s content extraction or content generation, and especially with the image input that we added this year, there’s so much you can do with it. It’s I feel like people come up with new use cases that. We hadn’t by imagination.

Yeah, yeah. So in terms of I mean, the only limitations, I would say besides your imagination would be more literally like it cannot generate images as output, for example. That’s when you need it’s not a diffusion model you would like either, you know, bring your own or, or. Right, right. So there’s that. But other than that, like, yeah, it’s a large language model. So yeah.

Yeah, use it for, I’ll. Give you some examples. So I think one thing to consider is like, is your use case, right? If you’re processing real time video data, I’m guessing you’re not going to want to call the foundation model framework per frame, even if it’s analyzing the, the image.

If you need something that is like sub milliseconds because you’re ranking, you know, a thousand entries in a list view or something like that, right? You can probably go with simpler models and maybe the foundation model or that full power of that LLM is not really where you want to go first.

I say maybe you can prototype with it, but sort of like I think of latency real time. I think there are categories of use cases where you, you know, it’s, it is such a powerful model, but it takes a little bit of time. We’re not, you know, we’re not submillisecond. I’m assuming. That is. Yeah, definitely.

For a response, maybe for a token, but yeah. Exactly. Yeah. Also. Apple has this other domain specific API like speech recognition, for. Example. It could be used for maybe suited for your use case as well, in line with the real time processing. So yeah, definitely try out these models.

Yeah, yeah. I think the Vision ones are a great example. I think there’s a great session here about Vision updates there and how it interacts with the foundation model and the foundation model using Vision as a tool for very specific tasks. But again, like those specific tools are super highly optimized, task specific ones which can be used in different contexts in which maybe the foundation model is not the appropriate context. So it’s a real time video processing trajectory, tracking of a ball zooming across the screen, things like that. Yeah, yeah. I think the, a lot of people overlook the high level frameworks we have for high level APIs like vision, speech translation.

You can use foundation models framework to translate, but we have an entire suite of APIs there for translation, which has more language support than the foundation model. So depending on your use case, you might want to choose a very specialized API that’s available versus prompting a language model. The other minor thing is probably we saw app developers build amazing experiences, as you mentioned, like summarization or text extraction and maybe grammar correction, and then like transformation, tone style, all of those great stuff.

Maybe if you were hitting a context window limit in the past, PCC alleviates some of those use cases, right? So yeah. So yeah, I think the question here is more like when, when do you want to use an LLM. Right. Oh, in general, yeah. Some people would have you believe for everything. But.

Maybe not. Maybe in the future you can’t be running it real time on a camera feed. Yeah. So it’s just right now it’s. Just. Other machine learning technologies. Available. One model to rule them. All right. Yeah. Someday. Someday. Great discussion. Thank you. Next question is from Natasha Prabhu.

Where should a beginner start if they want to learn about AI? ML. Are there any particular resources you would recommend? Wow. I’m going to jump on this one. Yes, please. In a past life, I was an engineer on the Swift Playgrounds team. And so I’m going to put a plug in for Swift Playgrounds.

One of the pieces of content that we actually made was an intro to ML, which used core ML to build a image classifier so that you could play the game of rock, paper, scissors by actually throwing out your hand to do the hand motions. And then we talked about sort of like, how do you build a data set?

How do you build a diverse data set? And then actually like, how do you then go ahead and create that model and then use that model inside of your, your app, right? So starting with something small like that can kind of give you an idea of like, you know, if you want to go down the full model training route, there’s a way in there.

But the other great thing is that the foundation models framework is so easy to kind of pick up today that you can kind of just like get started right away by just sort of making calls to the language model and getting kind of into prompt engineering and like tool additions and sort of all that stuff to actually teach you the fundamentals of like how to work with these technology.

And then as you get further and further into it, you kind of like peel the onion back and then get into things like, you know, maybe you download a whole bunch of weights off of hugging face and get started with, you know, Cori or ML and do something much more, you know, involved and think about how to like break down those sorts of problems.

So that’s my shout out for my, my previous job. Playground. Playground. So I think this year in Xcode, we added a very easy way to start Xcode and directly launch a playground like with one click. Previously, you had to create a project and create a playground, and you click and it provides you all the scaffolding required to just, you know, language model session. Session dot respond and you’re off to a start.

Like a great I mean, I would definitely encourage people to like, go deeper, right? And like, yes, we have a great API that like abstracts and makes things super easy, but it can be super rewarding to like learn more about like, how do these large language models work under the hood? How do they generate that text like token by token things like constrained decoding, the decoding loop, how they’re trained? Yeah, yeah.

Yeah, exactly. I was wondering, you know, what the question is referring to? Is it like using AI, ML. Or is it. Actually, you know, learning how those things are actually working? Right? So for the latter, I mean, indeed. So you can start from, you know, there are many tutorials out there with PyTorch NN or whatnot, right?

And or you, there are many courses available online as well, right? But in general, I feel it’s easier to start with like some with some concrete examples, as you were mentioning, right. So start with a concrete example, something quite simple. There are many tutorials online actually. That’s a great segue. The sponsor of this podcast is I’m kidding. No, we don’t, we don’t, we don’t.

We could have had. Yeah, I think I might pass. I would have turned towards, you know, like great tutorials and learning, you know, but I do think like nowadays getting your hands dirty and things and also even with an AI agent assisting you and asking questions and being like telling it, look, I told you lost me. I don’t know about this model optimization or like, step back.

So I think, yeah, it’s, it’s amazing all the tools you have available to you today to find learning at your pace. Yeah. Like if you’re new, what’s going on? If you’re new to PyTorch? Yeah. Barrier has never been lower. Right? Exactly. Yeah. I mean, I totally agree with like the agent stuff, you know, to, to learn new things. It’s amazing. Like you can query it with, you know, code prototype or whatever, you know. So it’s very easy to build things from, from scratch. Basically, I think build an app. Yeah. Have an idea. Build an app and.

Build a model. Build a, build a model. Build an app. I think some people refer to as project based learning, but ultimately you don’t start with the math. You start with trying to achieve something and you slowly open the layers. You start with matrix multiplication, right? That’s not right. That’s not the exciting part. I mean, it’s powered by that. But yeah, yeah, yeah.

You hurt my heart. Yeah. Exactly. Great. Yeah. So yeah, a lot of resources, I think just get started. Yeah. Everyone has to start somewhere. So you start building something and your passion will drive you to the depths of machine learning theory, if that’s where you want to go. Awesome.

You know, with like an internship with us as well. Yeah, yeah. No, I know there are positions open if you want to consider. Yeah. All right. Next question is everything we can do in PyTorch be imported to core AI or Apple ecosystem. What’s possible and what’s not? Oh I think this straddle sort of both Cocoa AI with importing PyTorch stuff, but also familiarity with ML and PyTorch in being able to experiment. So maybe we’ll start with.

We’ll start with PyTorch. So so basically, in general, if you can express your model in PyTorch and it is exportable by the PyTorch APIs, it will convert straightforward to core AI. Now that’s not for everything. There’s always new operations being added and new techniques. So even if it can. So if it can’t be exported, then you have to look at the PyTorch APIs or why can’t be exported. Query I supports all the core A10 ops, which is an internal detail if you know about PyTorch. And so if it can be expressible in those core, I will be able to do it.

Now you may come across some new specific particular op, custom op written that they have that may not lower all the way down, but you can provide a custom lowering in that case, right? You can sort of just wrap that up in a little custom lowering and explain it to Corey. I in torch operations, which will lower.

But a common thing also is that if you also want to consider performance, you can also wrap that even in a custom OP or a kernel. So you can even write a custom metal kernel, which may take a bunch of ops you’ve wrapped together and you say look like, yes, it could convert, but maybe the performance is not there.

So you kind of have those level of of flexibility is that. But again, majority of ones you see out there and maybe you can ask how like in the Corey models one, I don’t know how many of those models, you know, I’m assuming the majority of them don’t require any custom. No I. Don’t yeah. You just straight out export it.

Yeah. I think it’s mainly on those frontier or people experimenting with, you know, lots of people love different attention mechanisms and things like that where even if they do convert, you know, there may be some performance implications. And I think, you know, for ML, a lot of experience there with writing custom kernels and optimizing those.

I’ll just so the question says or the Apple ecosystem. So I wonder if Apple ecosystem. One definition that is you want to put this in your app, deploy it. The other possible interpretation could be that, hey, you want to run this on your Mac and experiment, learn ML like I know ML API’s are relatively similar to PyTorch. So yeah.

I mean, if you know, I mean, by the way, I thought 20 years ago, right? So in that respect. One of the core core authors of torch. Yeah. Indeed, they are quite similar. So if you are used to PyTorch, you will feel quite at home, you know, when it. Comes. To the neural network package.

If ML is very, very much similar to the the PyTorch one. We have like different modern approaches for the core M4 Max, but let’s not talk about that there. So, you know, if you know PyTorch, it should be fine. I would add that many people have actually converted on Hugging Face PyTorch model into ML models, right? So it’s actually often straightforward to just experiment with models which are already available in PyTorch, because they have been already converted on Hugging Face and ML. LLM supports many models from like more than 10,000 models from hugging face out of the box.

If you want to just experiment, it’s very easy. Right? If you just want to run a model server and you could convert and maybe like the earlier question, how do you connect your open code? Yeah. You know, to model server to. Yeah, a lot of different ways to experiment across the Apple ecosystem from .Mac to running servers, to integrating into your apps. Yeah. Anything I miss.

Yeah. I was going to also just say in terms of these one representation and another, there’s another place I’ll make a pitch for a lot of these agents and skills, right? Because the APIs are very similar, right. You can even just directly port it right into and directly author it in either the core AI set of things or in ML, right? So a lot of that stuff is like translation and conversion is.

So much easier. Much easier. So what I often do myself, you know, just exporting the weights and then, you know, it’s quite easy to just reload them in whatever framework you want. So you can effectively. Yeah. Because, and this is where you can learn about how LLMs work and things like that. They have some regular structure for many of these models. Right. And so yeah, sometimes you just, you can basically write the description of the model, particularly for deploying on Apple’s platform, right?

Most of the time you’re most interested in running the inference aspect of it, right? And so, you know, a lot of the PyTorch models and things you’ll find also have all the training aspects in inside of it. And so in a sense, sometimes there are aspects to inference which are not part of the training graph. So things like key value caching and things like that that have to get sort of injected anyway, and sort of some of these AI tools.

I’m advocating for can help you do that. Translation. In addition to we have examples online in these open source repos. And many of the people in the community have already done these things and established those patterns. And in respect, the AI tools can help you writing custom kernels as well.

By the way, that’s true. Yeah. One of the harder parts of the the job is when you when something is not supported and you have to write a custom kernel, I think hasn’t. Of course, I think you have to understand some level of metal programing, but it’s a good start, I think. Right. To get stuff working. Yeah, exactly. Definitely helps. Good discussion. Thank you. This question is from Eric’s questions again what’s the best UX when foundation models are unavailable? Older device Apple intelligence is of low battery.

Is there I mean valid concerns, right? Is is there an official way to detect availability early and degrade gracefully? And we can pre-warm the model to cut first request latency. Well, there’s a couple of questions in there. But to start well, to start with the last one. People are trying to sneak in multiple questions in a single question. But all good. We are here to answer your questions.

Yeah. I mean, to start with the last question that was there like, yes, we have a pretty warm API where if you are using the on device system language model on an Apple Intelligence device, if you know ahead of time, sort of when the user is going to, maybe your view appears and you know that there’s a button there that often triggers the model, you can pre-warm the on device model, which really means loading it into memory by, by the system before the user even interacts with that button. Right. So that’s a good signal that your app can give to reduce that initial Latency. And for the prior questions there about if it’s.

Well, yes, we have an availability API that too. So you can check that. So you can know if the model is not supported on the device. It’s running on, like if it’s an older device right now, in those cases, it’s a good question. Like in terms of core AI, like what kind of alternatives or fallbacks you can offer?

Would you like bring your own model, maybe to give like some of that feature set on an older device or. Yeah. Or would you rely on a server model or like, I don’t know what the best fallback. I think it depends. On your use case, right? So we’d have to know more about their use case and their main concerns of this thing.

So like if, if it’s network connectivity and you need the PCC model, there’s a question of like, well, if you needed the, the private cloud compute model, you probably did an evaluation and found you need a pretty large model. And so maybe having another on device model or core ML may not satisfy your, your use case. Right. And so yeah, maybe there you have to find some. You can use the language model protocol and have other server side. Yeah.

Good point. Actually that. So even if you’re running on an on a device that doesn’t support Apple intelligence, you can still use the Foundation models framework API this year. Right? Because we added that protocol that you mentioned, the language model protocol to let you plug in any model you want. So cloud server or on device with core ML.

Right. Any model you want really on device or server. And that way you can still like take advantage of the great API we have, even if it’s not with the built in system model, if that’s not available for whatever reason. Right. I think there’s a, there’s a nuance here about UX, right?

And we’ve discussed this in sessions last year and also some of the things we’ve done wherein depending on the availability API’s response, you want to show different things to the user. Right. If, if the one of the things is, is if Apple intelligence is off, then rather than saying error, you can say, hey, go switch on Apple and or guide your users to consider turning on Apple intelligence. So that’s like gracefully handling those things. Older device. Then either you handle it, handle it under the hood, or advise the user. I think there are.

Yeah, there are ways to sort of not pass on the pain to the user, right. Stuff that you can handle from a. But if you don’t have a fallback and you know the button is not going to work, it’s better to like hide the button than to have the user tap the button and just get an error or. Provide a different experience in the app, where in the user doesn’t feel like something’s missing.

Yeah. Right. They feel like it’s the experience they are getting. So yeah, the availability API is awesome. I think we’ve discussed this in some of the sessions, and we may have expanded some of the options this year too, but yeah, do check out the documentation. There are a lot of options for you to handle this gracefully in your app experience.

Good good discussion. Next question. This is from Cry Wolf. How should we migrate from Coreml to core AI? We have to wait a year or two until most users have updated their devices to iOS 27. Or is there a way to do this quicker? It’s just a two letter difference, so it shouldn’t be. Yeah. Yeah.

I mean, I’ll first start off it’s like you don’t have to like, it’s not a call to migrate off of choreographing. Coreml Coreml works. It is supported. It will be there and support some models that core AI does not like decision trees and other if you really want sub millisecond microsecond pieces, those still models are still useful and so. But if you’re interested in the features of core I and I is where we’re investing for sort of modern AI generative models.

And you want to consider that transition. You can consider the kind of things you would do with any sort of new API or framework that’s only available in the latest OS, right? So you can decide what’s the time frame from you and which your user base has moved on and you’re willing to, you know, drop support or, you know, these are both asset based APIs in the sense that the core model asset.

Right? You know, you can bring down a different model asset and use a different API under the hood, depending on which OS you are on. The APIs are simple enough that, you know, it’s only like a few line difference or in you putting a wrapper on top of it wouldn’t be too difficult.

So that would be my recommendation is to consider, you know, the models that you’re actually doing. Again, it’s not a call to action to be like, stop using coreml. Like Coreml still great and is still supported. And so you can make that transition when it makes sense for you and when it makes sense for your users. The death of core ML is greatly exaggerated.

I would add one thing. If you are currently in your app, have some generative model experience, definitely consider using core AI. If you have that on your core ML stack, using query is the recommended way. Yes. Going going forward. Right. We’re going to recommend that for the for those those experiences. Yeah.

Awesome. Thank you. Hopefully that puts you at ease if you were worried or concerned. Next. Next question is from catastrophe zero. What’s the maximum on device local model size can we bring we can bring into our app before we should begin considering private cloud compute? Yeah. I think the real question there is sort of what’s what probably depends on the device. But like, what is the true limit, I guess for let’s say, I want to bring an LLM with core AI to my app.

Yeah. So, so a couple of things to consider, right? Is, is thinking about what you can actually fit in your app and you can look at your, I think there’s some OS process available memory, there’s some call. You can check how much available memory do I have in my app?

Right to, to use. Right. And you can also look at instruments and a bunch of other tools. We have to see how much memory a particular model uses. That said, that is you finding out and thinking of everything being around your app, which I think everyone has great apps and I think, you know, some of them, you feel like have the whole system, but you also want to be a good platform citizen and think about what the experience is in terms of that user, of switching away from your app or, you know, launching the camera and stuff, which we, we will preference over, over the app when people, when the user is indicating I want to use something, something else.

So we generally recommend on sort of iOS to try to keep your models kind of below two gigabytes if possible, right? Particularly on those, you’re starting to stress the system a lot. And again, it depends on your use case. And you can request in some places to have higher memory limits.

There are two gigabytes refers to like iOS, I guess. Yeah. iOS. So so on iOS, on macOS, yeah. It’s sort of just like any macOS, it’s available memory. You need to profile your app and see what your on iOS is a bit more constrained environment in terms of thinking about the whole system experience.

So yeah, you have to look at that collectively, I guess. Yeah. So yeah, there’s no hard and fast rule. That thing I just gave you is sort of a rule of thumb off the cuff in terms of if you look at that, how much available memory you have, right?

You know, generally you’ll see that’s how much is available for your entire app, right? So that includes also the models you’re, you’re loading into memory or the graphical assets you’re putting, or your game rendering engine independent of AI and ML considerations. And then you have, I mean, you have always the option of quantizing more your model.

But then I would compare, you know. Then then you’re going to go to. Evaluations. In. Terms of. Not quantized or bigger model quantized. You know, it’s, it depends on your application, I would say. Right. Yeah. We have a session this year, Cori session where we show quantization accuracy, loss, and then quantizing parts of the workflow to recover the accuracy.

And I think there’s a little bit of experimentation to decide, you know, how much can you quantize and how much accuracy loss is tolerable. And in some cases it’s almost minimal. Right? I think that combined with evaluation, sort of get you to a place. Because. I think if you quantize to two bit like and you want to stay under the two gigabyte limit, you could get to like 6 billion. Maybe even. I mean, it’s pushing it a little bit. Like you get around that like model size for an LLM, right?

But and depending on if it’s like a image generation or, you know, segmentation model or language model, I think the tolerance to quantization tends to. Be. Different. Yeah. And I get the point, right. We’re giving a lot of guidance on iOS like on macOS, there’s a large variety of like the .Mac platform in terms of configurations of that hardware.

And so you can also consider the size of your model being dependent on the device and available memory that is there, right. And so you can scale your workloads or scale your models for application, right. So you don’t need to always be using the same model. You are on your Mac mini as your Mac Studio, for example. And on a for Macs. And if you have multiple Mac Studio.

We have a session this year where we show how you can connect for Mac Studio using a Thunderbolt Fi, and then you can run a trillion parameter model. Exactly. Yeah. You can push it far. Yeah. So an app. 6 trillion right? 1.6. Some of the latest. But we can’t do that with iPhones. You have to get all your friends. Yes. How many iPhones do you need anyway? Yeah. I don’t think we support Thunderbolt 5. Until. Like Game Boy link, like you get with all your friends, you just link with. A. Cable. Yeah. Trillion parameter. Anyway.

Great. Moving on. This question is from Ahmed Zia. How do you decide when AI should help the user versus when it should stay out of the way? Ooh, a bit of a philosophical question. Here, Shashank. You can you can answer this question. I will I will summarize the discussion here. Based on all your individual inputs, but I yeah, I don’t know. I think broadly as an app developer, you have to go back to the fundamental experience you want to deliver to the user not doing AI for the sake of AI.

Just these tools exist, but so have other tools existed. You don’t use every framework in the ecosystem. So if there is a specific type of experience you want to deliver, whether it is, I don’t know, something that creative, then maybe a language model can be used if you want to do grammar correction. Yeah, maybe. Sure. You can use a model behind the scenes, but maybe your experience doesn’t need to have AI powered use cases. You don’t even need to advertise that you’re using AI, right? Ultimately, the delight of the user experience, I think that’s what matters.

You as the creator of the app and the experience you want to deliver should guide the tools that you use. If there’s a if there’s a gap in the functionality, then it turns out that the AI model can help you, then sure, you should use it anyway. That’s like one of the things I think we all do as developers is just try things and see what sticks, right?

There’s nothing more fun than kind of like a side quest activity that is like, I’m just gonna try this thing, try. This. Right. And you get through that process and you then can kind of make that decision once you’ve sort of seen it in action and say, yeah, this is good, and I like this, or this isn’t really quite going to be as like meaningful to my users.

I think it could be. Now, of course, there is like, if you want to take it the full way and go like full scientific on it, you could build an evaluation that could tell you that, right? Like, you know, one of my favorite things to do as a developer is like build user stories, right? I really enjoy thinking about like, who’s going to pick this up? Who’s going to use this and who’s going to do what with it? And then kind of building like a series of like data points that kind of align to those use cases.

And then from there, you kind of can work backwards and say like, okay, this is resulting in like quality scores of this number or like my, you know, maybe they’re really high and that’s great. And maybe like, that’s actually a really good sign that this is really doing something useful for the, like large complement of users that I think will use this.

Or maybe it’s the opposite where you’re just like, oh my God, these scores are so terrible. And this isn’t really worth a lot of the time. And so maybe this is just, you know, it was a really fun exercise. I learned a lot about how to use these technologies and these frameworks. And that’s really cool. And I maybe I’ll use it like for the next project that I do. But today it’s just, you know, it’s a fun little side side quest activity.

So there’s a lot of ways you can kind. Of. I mean, what we’re doing, of course, is we’re trying to integrate AI into our system, right? So it’s, it’s really sort of the little things like in the keyboard and throughout your normal use of everything in, you know, in iOS, on your iPhone or any, any device that, you know, try to just enrich that and just make people like get from point A to point B quicker, right?

With the help of AI and not necessarily like distract or like make. Them, you should get out of the way as much. As. Yeah. Is that the question? I guess it was saying, should it stay out of. I think if you’re like, I don’t think anything should be in the way of what the user is, is trying to do. So I think just even the wording of the question, I feel like the answer is kind of in there is that if it’s getting in the way of a user experience, then it’s probably not the right answer right now. Yeah. Exactly.

Yeah. The entry points to AI is a, I think, a relatively new discussion wherein with the advent of like chat bot experiences, right? The entry point tends to be AI, like the first question you ask, but a lot of experiences that are part of the OS have been there working under the hood, like you said, the dictation or a lot of things. I think ultimately sort of it shouldn’t be blocking you from doing something. It should be helping the experiences.

And when I talk to our design friends, that’s when I realize that we are all technologists and we we work bottom up like, hey, here’s all these cool APIs. I want to use them, right, right. And, but user experience matters so much more. What kind of experience are you delivering to your app?

You know, app users. So yeah, I mean, for me, AI is a tool, right? Right. Ultimately the right tool for the, for your task, right? Do I need a hammer or a jackhammer for nail? You know, it depends. Yeah. Yeah. So if you only have a hammer, you only have a hammer. Everything looks like a nail. Yeah yeah.

Yeah. Good discussion. Thank you. Thanks, everyone. This question is from triumph. We support older iPhones without Apple intelligence. We have a self-hosted LLM backend. Cool on cool was me saying that. Not in the question. Okay, on iOS 27 is wrapping the backend in a custom language model provider. The recommended path. So one language model session tools, general transcripts of all devices. Any pitfalls versus system models?

Yeah, exactly. So the, the new language model protocol that we introduced this year really lets you like use any model with the same unified Swift API for if you’re prompting or doing tool calling, getting general output. So if you have your own custom server model that you’re using right now, then I think the first question to ask would be like, does that conform to like the, the typical like open AI chat completions, web requests protocol, right?

And if so, we actually have an open source utilities package this year that has an example or not just an example, but like has an implementation of a type that conforms to that protocol, that conform, that that sort of matches that typical like web request format. So you could use that as is, I think for a lot of like server hosted models that you might be using already.

And yeah, I mean, we also have ML and core AI packages. So I think, yeah, I would say like, look into the protocol, right? It really lets you like use any model you want without having to necessarily rewrite the other parts of your code, but like how you’re prompting the model.

Right, right. And especially if in the same app you’re using maybe the system language model, but also a different model. Then again, I think we mentioned it before where it can just be an if statement. It’s just an if statement for deciding which model to use, perhaps depending on the platform availability, but then all of your other code stays the same. And that’s really nice, right?

We are at the end of the group lab session today, but I want to sort of summarize. I know. Having so much. Fun. Yeah. It’s been an exciting week for sure, with so many good announcements. I think sort of what you said at the end is, is one of my highlights, which is the new foundation models framework, sort of standardizing the API surface for developers.

It’s so much easier now to be able to try different models, whether it’s, you know, using core I conformance the package or the ML one and, you know, bringing your own models, you should be testing everything as we’ve been evaluating. It’s fun. Yeah, yeah. I mean. I have. A good time doing. It. You will too.

Yeah. You should be integrating this into your test suites and making sure. Ultimately, I think the end user people using their apps and their experiences, what should be guiding everything so awesome. Well, that brings us to the end of today’s group lab. Thank you all for joining us. We hope this was useful. I also want to say thanks to our awesome panelists here and all the folks working behind the scenes triaging your questions. So thank you, thank you.

A few things I’ll reiterate here. As I mentioned earlier, if we didn’t get your questions and I know there were so many here we didn’t get to. Please head over to the forums at developer.apple.com/forums. We can continue our discussions there and bug reports feature requests again, feedback assistant.apple.com is your friend.

We do really do appreciate all the things you file, so thank you. And speaking of feedback, you should also receive an email with a survey link to let us know about your experience at WWDC. We’d love to incorporate these feedback for future events. Thanks again for joining us, and we hope you had a wonderful WWDC this year. Thank you. Like and subscribe. Like and subscribe.