Welcome to the camera and Photos group lab. My name is Sergey and I’m part of the developer relations team here at Apple. Today I’m joined by a panel of experts from Camera and Photos engineering team. I let them introduce themselves. Matt, let’s start with you. Hi everyone. I’m Matt Dickoff and I work on the Photos Frameworks team. I’m Brad Ford. I work in camera software and have had the good fortune of working on every single iPhone in my 25 years of Apple.

Hi, my name is Iván Cavero Belaunde. I work in camera software as well as Brad and I specialize on the camera. The still capture pipeline. My name is Davide Concion. I work in a team where we do file format compression and raw and I had the. You know, I’m lucky enough that I’ve been working at Apple for 19 years now. I still love it. There is every day. Something new to learn.

Oh. It’s amazing. Oh, I’m Jake. I work on camera performance. I’ve not been here for 19 years. Just five years. Not too bad. So in addition to those on screen, there is a team behind the scenes helping us triage the questions. We are so excited to talk about building exceptional experience using camera and photos frameworks. And today we want to focus on questions that benefit everyone watching. So for specific questions, or if you if we somehow didn’t get to answer your questions today, go to the developer forums to continue the conversation.

Also, you can use Feedback assistant to file bugs or submit feedback requests. And to get started, I would like to ask a question to all our panelists. So today, camera and photos apps on the iPhone are packed with features. There are so many of them. And I’m just curious, what is your favorite feature on camera and photos app? up. Maybe. Maybe you can start.

Sure. Yeah, there are a lot. I honestly, the one I love the most is probably Live Photos. I think the, you know, especially with kids and nieces and nephews that live far away, getting this sort of like slice of time photo plus video and sound. It really brings those moments to life. So Live Photos is probably my favorite. Yeah. It’s great. All right. What about you? Yeah, Live Photos is kind of the OG, isn’t it?

It’s an Easter egg. My mom didn’t know for years that she was taking live photos. She says, how do I turn on Live photos? And I said, you already have live photos in your in. Your. Photo roll. Mom, I like a lot of us, spend a lot of time on my Mac in meetings. And so I love the video effects, the system wide video effects like portrait studio, light Center Stage the fact that you get them for free in any app as a user is very powerful.

Recently, we’ve added background replacement gestures. whimsical, delightful and add value to conferences. So I really love those. And all the new features on the iPhone 17 front facing camera, which I hope we get to answer some questions about. Yeah. For sure. Yeah. Thank you. Yvonne, what about you?

The one that’s near and dear to my heart is we refer to it as opportunistic depth capture or portrait in photo. And this is the feature that introduced a few years back where as long as there’s a person in the scene, you’ll see a little F in the camera app. And when you take a photo, we will get depth for you and you can go and add the portrait effect afterwards if you so desire, with all the controls that you that you want.

That release, we did a ton of work to actually make the depth processing a lot more efficient and, and available at on all kinds of zoom levels and be able to take advantage of deferred processing. So these are all things that is available for developers that are available for developers to, to adopt in their in their camera applications as well.

It’s great. Every day. Yeah. For me, it has to be pro raw for two reasons. Number one, I’ve, I’ve been working on that feature for so long you cannot even imagine. And the second reason is that is a vehicle for the rest of us to access features from the camera that usually are just for, you know, the pros, the people that know how to edit with the pro forma, which is available on camera, and even for third party, you can actually collect the full spectrum of what the iPhone can give us in terms of image quality.

And then on the on the other side, if you like to do so, you have the latitude to embellish these images or modify them the way you like. Because even though images coming out from the iPhone are definitely amazing, every one of us has a little bit of a different taste. So ProRes is actually allowing allowing you to do so.

Yeah. Makes sense. Thank you for sharing. Jake, what about you? Yeah. I think I’ve been having a lot of fun with the new tele on the iPhone 17 Pro. Like the zoom on it is insane. Like, get up to, like, 40 x. We were at Yosemite, like, a few months ago. Hiked up to the top of Glacier Point. And like I was zooming in all the way through the valley. It was like incredible. Were you able to see?

Yeah, you can actually see people. Yeah, they kind of look like ants from that far away. But it was pretty cool. Yeah. That’s amazing. A lot of fun. Thank you. Thank you for sharing your personal stories, everyone. And with that, I would just I want to jump straight into the questions because we have so many of them. And, you know, I want to give as much time to answer them. So the first question is from user with the name Florent in ENF.

And the question is when reframe. Our clean up is used to modify a photo. Does iOS 27 tag it with any metadata or content credentials so someone can tell the image was edited by AI. I’d like to take this one. If you don’t mind. Yeah, go for it.

It’s just because we do work with metadata every day. So yes, the answer is yes there is. The metadata in the file is being modified with iptc metadata together with with Exif, and the Iptc gets updated even based on which of the AI modification that you are doing.

Spatial reframe or even clean up. So yes, there is a way to to understand from the metadata what what has been done to the image. And just to add on a little bit that in the photos app, when you do one of these edits in info panel, so when you swipe, you know, on iOS, when you swipe up at the bottom of info panel, we display this information about which edit was used.

It’s great. Thank you for sharing. All right. So the second question is about the keywords in photos. So the question is keywords were mentioned as a feature common to photos. Will there be an API for third party apps to use or edit keywords? Yeah, I can take this one as well.

Yes. So the, the feature I think Ben’s referring to there is in on iOS now it’s a feature that’s existed on Mac for, for a long time. In the photos app, you can see and edit keywords again in the info panel. That sort of UI I was just mentioning.

And there’s also UI for sort of managing these keywords and searching by them. These will be reflected in if I take metadata when you export, but there’s no, there’s currently no sort of photo kit level API for fetching or querying based on these keywords. Gotcha. Okay. Thank you.

So the next question is from a user with the name j k underscore 27. When showing thumbnails of images in a lazy grid view that also have a matched geometry effect to a full image detail view when tapped. So what’s the optimal way to load the thumbnails images for performance? For example respective smaller size stored with og, use some Sy filter to scale them down or in it.

I don’t know who is. I can I can take this one. So on the core graphic side, there is a there is a property when opening images called open open Image with thumbnails. Internally, Core Graphics will understand if the image has already a thumbnail to be to be utilized, or it will try to decode and scale down as fast as possible. The. The main image on the. On the Core Image side.

We have ways to to request a scale factor. When we open images and depending on your UI, you can ask the scale factor to be as small as you as you wish. And this internally instructs Core Image to actually scale down the image as soon as possible during the process, so that everything that happens after then is is going to happen as fast as possible.

Okay. That’s great. Thank you. We have one more question from Ben on the first start. And maybe Jake, you can take that. Yeah, absolutely. So the question is, can the first start cause issues where the user may attempt to capture a photo before they capture photo output has attached to the session?

Yeah, it’s a good question. So actually talk about this exact point in my dub dub session this year. Build a responsive camera app that launches quickly. So yeah, with Deferred Start, basically what you’re doing is just pushing out initialization. So if you only do that to the Asker’s point is yeah, you may actually still miss the shot.

So the photo capture output actually has an is responsive capture enabled property. You set that to true while also deferring it. Well actually the system will add some buffering so you can launch quickly. Get the capture even if we haven’t fully initialized the output. So you don’t end up missing that moment. We still queue up that shot. This is great. And by the way, that session is awesome. You know, if you haven’t watched it, please go ahead and you know, find it in the developer app. It’s called let me find the name.

Build a responsive camera app that launches quickly. And we have a cool domino effect. Yeah. You can watch me play with dominoes. You can. Play. Yes. And just for background, for anyone who hasn’t watched it isn’t sure what we’re talking about. That deferred start API was introduced in iOS 26. It’s a way of telling your AVCaptureSession outputs which of them should start up deferred and which are most important to start right away. So usually you could say I want the preview to start first. Everything else can be deferred. Yeah, exactly. All about that fast launch experience. Perfect.

All right. So the next question from I hope I pronounce the name correct. What is the best way to get the depth map and the image for a live viewfinder and for the taken image to create a nice 3D effect with with both. Okay, I can take that. Go ahead.

So thanks for the question, Jillian. So there’s two parts to this part is the preview side. And part is the still capture side for the preview side. You want to use depth data output enabled on on your preview stream. And you want to use the AV capture synchronizer so that it’s synchronized with the RGB stream on the still capture side.

You you want to what you want to do is enable the enable depth on your on the AV capture photo output. And that will that will provide ensure that you get depth with the still capture so that you can use. So then so then you can use filters like in Core Image to to add the. Add the blur based on the depth that’s captured into the still.

Yeah, there are levels to it. If all you’re interested in is having a live depth effect in your preview, there’s a shortcut. You could just turn on the Cinematic Video Capture API, which we introduced last year, which lets you replicate exactly what we do in the cinematic mode in, in the camera app.

So if you just use a video preview layer and you set cinematic video capture enabled on your device input that will give it to you for free. So if you just want something cheap and easy, you got that. What Yvonne talked about is absolutely right. If you want access to the actual depth samples and you want to kind of do it yourself, then video data output plus depth data output, and then use a data output synchronizer to make sure that you get the depth and the video for the same timestamp in the same callback. And then you can render them together using a Core Image filter.

Makes sense. And the specific thanks, Brad. And specifically about the the specific API for the still capture side that you want to be looking for is in the. And the, in the AV capture photo settings for the, for the AV capture photo output, you want to go ahead and set a depth data delivery enabled to true. Okay, that’s good to know. Thank you so much. Useful info. Thanks.

Next question is on for the quality prioritization. From Eric. So the question is when for the quality prioritization is equal to quality. The AVCaptureDevice often overrides the manual exposure duration and ISO settings that it has been said. And when capturing a photo, is there a way ahead of time to find out what the final exposure setting will be before the photo is captured?

You want to take it? Sure. So from an API standpoint, actually it’s not just quality, but also balanced. The, the we choose because we’re, we’re doing, we’re doing processing algorithms, most of which include Fusion, which need to do things like capture, capture different exposures, like one underexposed image as well, as well as a normally exposed image.

We do not give, give control to, to, to any of the manual controls. The manual controls, in fact, only work are only supported when you capture with speed. If you’re. If it looks like balanced or quality are respecting your settings, that’s because you know that you’re getting lucky. But in general, there’s no guarantee that it’ll.

The capture stack will make that determination. Depending on the scene, the overall brightness, and so on to, to, to figure out how to expose whenever us, whenever you choose balanced or quality, you’re essentially handing off the, the decision making about how to expose this to us. And, and if you really need that level of control, you should use manual.

Yeah, yeah, we realize it’s a little opaque that you signal to the framework. I am willing to take this quality prioritization. So I’m, I’m willing to wait or I want to balance or I just want it as fast as possible. But we, we use that as a hint.

So depending on which format you’re using, if it’s the photo format, we are going to use our best algorithms, our best Fusion algorithms, and we’re going to ignore your manual settings for some of the video formats where we we choose not to use those Fusion formats because we don’t want to disrupt movies and preview. You may still get a speed capture. And so you might still have your your manual settings preserved.

So the rule is if you’re using photo mode, quality and balanced are always going to override your manual settings. If you really care about having exactly your shutter speed, ISO, and things preserved, then you should definitely use speed for your photo quality. Photo prioritization. That’s great. And I believe this was pretty well covered in the implement high resolution photo capture session this year. So yeah. Great. Session. Yeah, maybe. Check it out and you know, find more answers there. Great. The next question is on.

Asset framework API. So can you elaborate on pH acid? Regional resource choice and what it’s used for. There is not much documentation yet. Yes. So yeah, this is a new new API for an existing feature. So for a long time in the photos app this is related to Raw plus Jpeg. So you know, it’s very common for dSLR cameras to have a setting where you shoot Raw plus Jpeg or these days, heck, compressed in a raw image.

And then when they’re imported to the photos app, you’ll see badges that say raw plus J. And when you go into edit in the photos app, there’s a way to choose. AM I doing my edits on top of the raw image, or am I doing my edits on top of the Jpeg?

And users can kind of swap between them. So that’s what this original resource choice is, is it’s saying, is the compressed image the original, or is the raw image the original? And what am I using to do edits on top of? And also what are we using to make smaller image derivatives and thumbnails and such? Which which resource are we sourcing those from? So yeah, there’s a handful of APIs there around the FPS content editing, input source choosing which one. There’s a change request on assets for toggling which resource is the original. Yeah, I believe those are the main ones. Great.

Thank you. Next question is on barrel preview capture. Mac Pro. Capture. Preview. So will iOS 27 support the linear scene? Referred preview stream for barrel capture via capture video data output or AVCaptureVideoPreviewLayer without tonemapping or computational processing so it can match a linear CIFilter DNG conversation. And they actually they linked a related feedback assistant, which is great. Thank you for submitting that feedback assistant.

I want to take this question. I can take this. So today, the only way to actually get seen referred linear data, you know, in camera capture is actually through the the log format. We have two flavors of it. We’ve got log and log and log two introduced last year, which is an improved gamut of the of the log.

Now in terms of, you know, if I drill in deeper into the question, I guess I’m assuming that the developer would like to actually have raw data coming out from raw copper and being camera capture and being able to use the same filters that we use, say, for images.

Now that is not available today, raw frames coming out for ProRes raw, for instance, for capturing ProRes Raw from Mac App Store do come with with certain metadata, and that metadata is not compatible with the metadata that CIFilter would need for for rendering that image. But I kind of like the idea.

And thank you for the, the, the feedback request. I believe it’s something that is worth, worth exploring definitely for. Thank you for the future. Yeah. It’s great. Thank you. I really love the next question. It’s I think it’s more holistic, more high level. Maybe everyone can take a turn and give their own suggestion.

So the question is, for an app that lives and dies by its capture experience, what are your must have recommendations? I know Jake. Yeah. I mean, I’m going to be biased here, but I’m going to say you want to have that most performant app as possible for sure.

You know, I think like when you have a capture app that people are using it to capture life’s moments. And if you miss the shot, sometimes you can’t get that moment back for sure. So keeping things fluid, you launch quickly, you get that shot responsive capture that. I think we’ve had, you know, session in 2023 about.

And then in the most recent session, you know, talked about that and deferred processing. Like I think to me, those are your top three things. Get to have a fast launch, responsive capture and use deferred processing. Can’t go wrong with that. Sure. Yeah. Thank you David. I think make it pro.

Add any any capability that you know are available for, you know, for reaching the, you know, maybe the part of the world that would like to actually, you know, interact with those assets, you know, with more personal flair. Sure. So keying off of that, I’m going to say maybe not make it professional.

I’m going to. Say like in the Lives or dies by capture experience is one of these things that is, you know, it’s, it’s vague enough that we’re not really sure what you mean, like an optimal capture experience for something that’s intended to capture video for a social network is very, very different from, from a, from a, from a pro photography application.

You can have a photography application that’s actually like tries to lean into the Superfund, like remember Hipstamatic when it was first launched. And, and so, so, so it really depends what you mean by that. But by the, the, by the capture experience. That said, I will echo the performance stuff. People like users really don’t like to wait. The performance is critical. Reliability is critical.

There’s lots of things that, that, that, that, that you can do to, to make it more reliable and more stable. We have tons of tooling for you to analyze your performance and, and track the kind of issues that you should be addressing. But, but, but 100% like what I, the takeaway, I would say focus on the user experience based on who you think your users are going to be. Who is this for? Thank you man. Yeah.

We realized that AVFoundation is not an easy framework to get into. It’s huge. I think we’re the second largest framework in iOS after UIKit. It can be daunting. You know, you look at all of these properties, all of these classes, and you’re like, where do I start? My recommendation is to not start from scratch.

Use the sample code because there are best practices that you might miss out on if you just, you know, peruse the documentation and then start writing. A common problem we see is that people use their AVCaptureSession on the main thread, not realizing that it blocks to do certain lengthy operations.

And we, you know, you would know that if you looked at our sample code, that you need to have a dedicated serial queue in talking with AVCaptureSession. So right there, that’s going to go to the performance, to the responsiveness. Don’t lock up your, your UI thread doing stuff that’s meant to be done in a background thread.

Also, whether it’s pro or not, pro figure out what differentiates your app from others. I mean, I think camera apps are a dime a dozen. We give you so many tools, everything from from pro use cases like gen lock and lock frame duration, which we just gave you last year to very simple, you know, record a movie file. It’s all there for you.

But what are you going to do in your app that differentiates it, that draws people to it? Is it a social aspect? Is it, you know, a key feature, a key value add that you’ve got that, you know, we are providing the toolbox, but we don’t provide everything for you.

Make sense? Thank you. I’d be. I’d be remiss if I didn’t talk about the photos framework side of this. You know, I think it’s hard to talk about, you know, a camera app is great for taking photos. And then those photos need to go somewhere. And so, you know, the integration with the PhotoKit, both photos UI and photos framework, it starts with the user permissions, you know, like by default, your app does not have any access to save or read back photos from the photo library.

And you can sort of gradually increase that you can request to just save assets. And it’s a very simple request that most users are fine with. And then if you need to read stuff back, you can upgrade your permissions through prompts. And there’s sort of a very kind of tight coupling and performance considerations there around capture to review whether that’s in your app or someone’s jumping to the photos app.

Yeah, it’s a great advice and thank you everyone. I like the, you know, pro not pro discussion. I think we. Should give that better. Yeah. Oh, I love the props, too. Like I have some near dear to my heart. I was just, I thought it’d be interesting to, like, offer some contrast and look at the more fun apps as well. It’s super, super helpful. 100%. All right. The next question is on.

So the question is what is the optimal recommendation and recommended way for streaming video and audio simultaneously? My concern is that there will be a delay in transmission and the video and audio will not sync. Specifically, what is the most optimal way for streaming the audio, and is it safe to stream frame by frame of the video?

I can take this one. Sure. Audio and video synchronization is no joke. You should pay attention to it. Don’t. Don’t have incidental or coincidental sync. Our. Our video and audio on modern iPhones is synced from the same clock. So if you don’t handle your AV sync, you might think you’re okay and ship an app that is usually in sync, except when you run it on an iPad with an external camera, and then the audio and video suddenly become out of sync.

The the first recommendation I would have is use AVCaptureSession for both your audio and your video. In other words, attach a device input for a camera that you’re interested in and a device input for the mic that you’re interested in. And then if you get audio data output and video data output, the AVCaptureSession will already do the hard work of synchronizing those two sources so that what comes out of those outputs, the TTS, are already on the same timeline.

So whatever you want to do with them, stream them, write them to a movie, file, whatever they’re already in sync. If for some other some reason you need to use a different audio API, such as aux remote I o, then you’re going to need to do a little bit more work.

then you’re going to need to delve into clocks like C.M. clock. Every device on our system is backed by a time source or a clock, so the video will be on one clock. The audio will be on a different clock. And it’s important that once you get samples from those two sources, you synchronize them using these clocks.

And there is a low level API in core media called cm synchronization cm clock convert time, cm synchronization, convert time, something like that, which lets you say from this clock to this clock, give me the TZ and you give it the TZ and the source time frame and it’ll give you the output TZ.

So what you’re going to wind up doing usually is keeping your audio time because you don’t want to have to rate convert the audio, but you will synchronize your video to your audio clock. Get a different timestamp for that video buffer that came out. And now they’re both on the audio timeline.

So as far as meta concerns about things coming out. If you’re if you’re streaming them over a network, well, then you just have to rely on your the timestamps that you had them in sync before you sent them. So when you get them on the receiving side, the audio and video have a coherent timeline, then you’re going to need to take care of the playback synchronization on, on the playback side. And we have plenty of APIs in the rest of AVFoundation on the playback side, such as with AVPlayer AV, you know, sample buffer display layer that can take care of the synchronization for the playback portion. Sounds straightforward. Yeah, it’s very easy.

Is audio ramping easier to perceive than video ramping or something? I think the science tells us that people perceive changes in audio sample rate more readily than they do micro-adjustments in video time. So generally that’s the better thing to do is adjust the video to the audio than vice versa. Yeah. Unless you’re going to sample rate convert the audio. Audio is much more important for experience. You know. Just a tiny glitch in audio and people will hear it. Yeah, it’s super sensitive. One thing. What does DT stand for?

Oh. Presentation timestamp. Thank you. Thank you for asking the question. Yes. All right. We got. Some really, you know pro focused questions for it. So David your turn. So next question is from user with a username from number 16. And the question is why does Pro raw support 48 megapixel output from quad Bayer sensor. Whereas native Bayer raw output is limited to the binned resolution. Yes.

So to answer this question, maybe we need to differentiate what is a Bayer Raw from a Pro raw. So Bayer Raw contains Bayer data in the in the in the file while ProRAW has gone through a debating step and Photonic Engine Photonic Engine merge of multiple images to get that output. So. At that point, ProRAW does not care anymore. What is the format of the sensor? It may be Bayer, it may be Quadra.

It doesn’t really matter because the data that comes out is in RGB already linearized. Now to go to this question why ProRAW can do it. And Bayer can’t is because ProRAW is is linear. So we are able to do that. Now the the ability to do Bayer for Quadra sensor is not yet available.

And if if the developer would like to have actually this capability. I would highly suggest to to send a request because, you know, it’s something that, you know, if many developer were to be wanting could be something that we can look at as well. Yeah. So please, please submit that feedback assistant. We actually read them. Yeah.

And if I can add a little bit to that, the, the other aspect is that. So it’s like this raw output wouldn’t be Bayer, right? It’d be quad Bayer. That’s the thing. The Bayer output output. The raw Bayer output from a quad Bayer sensor is the binned sensor. We bin the quad sensor the the quad pixels into into Bayer.

And the other aspect of this is that this is an ecosystem question right. We can’t give you quad Bayer DNG raw output without giving you a way to decode them. Right. So it has to come. It has to come with support in Siri for filter. ET cetera. ET cetera.

So so so it’s it’s a it’s a heavier lift than what it sounds. Yes. And maybe to add even to that, the bearing data Bayer sensor is, is a technology and skills that have been developed for now 25 years. De Bayer in Quadra is a completely different beast. It’s way more complicated than 1st May may think.

Interesting. Thank you for sharing. All right. We have one more question for the pro camp. What is your recommended process to generate and write ISO, gain maps and gain maps, map metadata for hike and Jpeg on iOS? Yes. Wonderful question. Without going into the details of the API, I’m going to send the developer directly to a WWC talk that we had two years ago where exactly this question is being, you know, drilled into we have we have two major frameworks that can handle gain map and ISO, gain map data.

One is called Core Graphics and the other one is Core Image. For both these cases, David, one colleague of us is is giving exactly what are the APIs, how to do, how how input need to be prepared to then create outputs that are gain map compatible. And there is a third way that is possible. The specification for gain map have been added to both the spec and Jpeg.

We have been working. Apple has been working to add those those back And they are very, very clear. So that’s another way to even monitor understanding what this metadata does, why this image is divided into in terms of what is, what is your SDR or RGB content and what is the HDR addition to it.

So that’s a third way to, you know, if the developer likes to, to go about it. But again, for APIs on the system, that talk will give you all the information that you need and even power that you cannot even imagine the, the Core Image side can do, can, can control for the output, almost everything. What is the look, what is the headroom being utilized, how, how the data is, is merged together. So you will have, you will have fun to go through that that presentation. It’s great. Thank you for referring to that presentation.

Next question is about the first start. People are actually care about, you know, launching apps quickly. So the question is the first start says hold the photo output back. The new hires guidance says that says warm it early with set prepared for the settings array. A contract a decade older than deferral. Deferral. What’s the supported composition? Does prepare Q past deferral or forced start, and what does each reserve.

Yeah, I could I could start with that. Yeah. I think like the way I view deferred start is really it’s just all about getting that launch up and getting preview running. So you’re just basically moving the initialization from before preview to after preview. So I don’t think like in terms of it, like in terms of using the warm on the photo setting array.

Like you can basically still do that after, you know, previews running it. I don’t think it really makes much of a difference. With deferred start, I think deferred starts just moving that initialization out. Yeah. One way to think of it is imagine a graph of objects where they have branches going out for preview and for photos and for for movies or whatever you’re making.

The deferred start just says like, we don’t need to resolve all of the objects, buffer pools, etc. on all of the branches to start. We just have to get preview rolling as quickly as possible. These guys can get done when they get done. Whereas the the prepared settings array, which you’re right, has been there for a long time, is a way for you to tell the photo output up front.

Here’s the worst it’s going to be, you know, this is the, the most crazy thing that I might ask for quality plus, you know, whatever other features. And that lets us preallocate for the worst case in, in the still image pipeline. But you can do those at any time. You can. You can reprepare at any time.

It just it’s helpful if before you start your session, you you do call set prepare settings array. Tell us what the worst case is going to be. But I think these two APIs are kind of orthogonal to one another. Would you. Say so? Yeah, I would agree. And like, it is actually interesting though that the photo output quality will impact launch time if you don’t use deferred start, right?

Like with the speed capture, the launch is actually going to be faster versus quality. You know, to your point, we have to do all these heavy allocations. So yeah, I agree. Like, I don’t think they, you know, they collide. Yeah. And they can complement one another. Yeah, exactly. Yeah. Yeah. Makes sense. I like when people agree.

So let’s shift gears a little bit and talk about forest for a little while. So the first question I have is will adjusting pH assets new rating property require pH library change request. Yep. It’s a it’s a Excuse me. It’s a pH set change request. You’ll see the rating property rating property there that you can change per pH set. And there’s a there’s a new enum. It’s like a values of unset and one through five. So that’s how you can as a. That’s how you can modify pH set ratings via the a p I.

Okay. Thank you. The second question I have is, is there a native way to obtain metadata of OG file type PNG, JPG, etc. from the photo selected. And they clarified. I asked this because every time a photo gets imported from photo speaker, it seems to only show as PNG and follow up. And it does it mean it’s getting converted before saving to our app folder?

Gotcha. So there’s there’s a few things you should check. So when using the transferable to set up the data, be sure you’re setting the UTI type as opposed to it sounds like maybe you’re leaving it as a default image type. So specifying UTI type is one thing to look out for.

I believe there’s some sample code on on developer.apple.com for the photos picker from a session a few years ago. That should cover this. And then there’s also it sounds like you’re getting PNG out. So I wouldn’t necessarily expect this, but there can be some conversion that happens. And the photo picker, for example, if if the user has disabled other captions or location from being given to your app via the picker, we may convert from some formats like raw, but it sounds like the PNG is the output. So I would double check the. The UDT type for transferable.

Makes sense. Thank you. And the third question I have. Is there a way to learn about the adjustment list format that the Apple Photos app writes? And when exporting that file, is there a way to import those adjustments back into photos? So there’s. we do not have API for that. So if that’s something that’s just desirable for your app, I’d definitely file a feedback request. There’s no way to sort of decode that P list unfortunately.

Okay. Makes sense. Thank you. Let’s switch gears again. Let’s talk about let’s talk about some pro features again. What is your recommended process to generate and write ISO? Oh we already answered this. Sorry. My bad. Oh actually there is one really, you know, high level generic question. You can go to what are the biggest mistakes developers can make when building a camera heavy apps on iPhone. Quitting your day job.

A couple really common ones would be. Like I said earlier, the AVCaptureSession is meant to be called on a background thread on a dedicated serial queue. It by design blocks and waits. When it does long, you know, reconfiguration of the graph. This is clearly documented. Hopefully you’ve read the documentation. So you know, the most naive thing would be to just call AVCaptureSession, start adding things to it, call, start running on the main thread. It will block your UI, you will get little spinners and people will give you one star.

So don’t do that. The other one is when you’re reconfiguring your session, usually you’re going to change more than one thing at a time. Like if you just set one property on it, it doesn’t really matter if you call begin configuration and commit configuration because you’re just doing one thing. But imagine you’re changing from a photo mode to a video mode, or you’re changing from, you know, one high resolution active format to a lower one. Usually you’re going to do several steps there.

Implicitly, the AVCaptureSession will reconfigure its underlying graph. Every time you call a property that that causes a disruptive change. The way to prevent it from doing this, each and every time you set a property, like on the way to getting to what you really want to do is to call begin configuration.

Think of it like an ATM where you need to like, this is the start of the transaction. I’m going to do a bunch of stuff, but hold on. Like don’t do anything until I’m I’m done and I commit at the end. And if you do it that way, then you ensure that you do one operation or 20 operations. The graph is not going to reevaluate and reconfigure itself until you say commit.

So those are the two that that come to mind as you know, rookie mistakes if you’re not using those two features. And the other ones, guys. I was thinking about video data output. If you’re using that to render preview, I think sometimes it’s pretty easy to, you know, you get the frame data, you’re all excited to do some processing, but you could end up actually dropping frames if you’re doing too much of heavy lifting in there.

So, you know, using preview layer of AVCaptureVideoPreviewLayer to render preview, if that’s just your motivation is maybe a better option. Right, right. Yeah. The only real reason to use a video data output for preview is if you need to interact with the buffers in some way, if you need to either get metadata from them or draw on them or meter them for histograms or something like that. But yeah, our video preview layer is really, really efficient. I don’t think you’re going to do better than than the optimized path that we have. That’s great. Yeah. Thank you. Jake and Brett any any more.

Let’s move on to the next question. So the question is about zoom scroll and pan feature. Any suggestions on a close to native performant way to achieve Zoom scrolling pan feature for photo images, for example up to only its original max resolution without a pixel A18 and Davide, you want to take this? So I believe the best way is actually to use Core Image. Core Image is as a framework is actually built for for adding this ability of, you know, having UI movement, zoom pen.

And he does that by caching almost everything that happens throughout decoding decoding pipeline. So I do believe there is a, a talk or a document on the developer portal that actually does explain exactly this and shows how to set up CIFilter for opening images. That then would allow you to modify, you know, what you do. For instance, for a zoom decode only this rectangle.

Everything is cached and moving this rectangle around. So it’s really, it’s a really powerful framework for doing exactly this stuff. One thing to note is just because, you know, I come from the pro work workflow, the on the raw side, Core Image has also it’s a counterpart, a CIFilter for it.

It has the same capabilities. So even if you open a 100 megapixel images and then try to pan around all these features, all this capability of Core Image still are present. So you’re scrolling, your zooming will still stay very, very smooth. Because of this caching capability that Core Image has.

It’s great to know. Yeah. If I can echo something, Riffing off of that, the. The region of interest, the ROI management and Core Image is is really one of the places where it really shines. If you, for example, have an image that has a number of heavy operations that are applied to it, and then you’ve zoomed in a lot and you move around, it’s only going to go and recompute the area that you’re zoomed into, even if the image is many times larger. So it will do all kinds of optimizations like that to make sure that, that, that you’re not essentially wasting cycles on things that are not going to impact your, your application. Exactly.

Yeah. Thank you. Got some really highly upvoted questions. Let’s go back. Let’s go to those. So how many times can I use the intelligence feature? And also when using it, does it affect or change my picture quality. You can take this. This is about the Siri camera. Yeah I believe so.

I don’t know if you want to take it. Yeah, sure. Yeah. So on, on limit. You can use it as much as you want on the device. As far as the the quality, those captures don’t go to the user’s normal photo library. They get saved to the, the Siri app and the, as far as quality, their screen resolution aspect ratio. So you’re, you’re not getting, you know, the full blown image quality that you get in photo mode, for example.

Yeah. So don’t use it if you’re trying to get like the most beautiful photos in order to have a conversation with Siri about that photo. Yeah. I believe there is some limit on the number of conversational things you can say about a photo, or ask about a photo. Makes sense. Thank you for that. Next question. The new Spatial Reframe feature uses 3D modeling to reconstruct a scene from a different angle. As a developer, can we access that pipeline via an API or is it locked to the photo app?

So yeah, there’s no, there’s no developer API there for using that, that reframe capability. There are some parts of the ARKit framework that allow for 3D scene capture and reconstruction and manipulation, but it’s not a sort of out of the box, you know, photos reframe edit solution. Right? And we have, if you’re interested in live captures that make use of it’s not exactly spatial features, but we do have depth sensing cameras that we can use on the pro phones. We have the lidar sensing camera. So we do have an AVCaptureDevice that’s that uses the LiDAR depth camera, as well as the RGB camera and fuzes them together to give you depth.

And then if you’re using the front facing camera, the true depth camera, there’s infrared that you can use for capturing depth. But no, that’s, that’s a good feature to request as a feedback assistant. Yeah for sure. So files at feedback assistance request. And next question is about rotation handling.

When can you get the rotation issue? Just after. Handled by the camera capture. Since on iOS and macOS and iPadOS, it always seems to handle orientation differently. I spent so much time tweaking this for any new app and always get it wrong. Geometry is hard. Yes. I can’t tell you how many times I’ve taken a piece of paper and drawn a smiley face, and then drawn it in all the different orientations and then flipped it around, and it’s just reality.

You know, like you’re dealing with a an ID, a device that’s got a camera in it. The camera is physically mounted a certain way, a different camera in the same ID might be mounted a different way. One of them might be portrait, one might be landscape. There’s really no getting around the need for for dealing with orientation. It’s not something that we can authoritatively do correctly for you because we don’t know your intent.

We could say, well, gravity is here, so probably they want this to be up, but maybe not. Maybe. Maybe it’s an always portrait app and that’s the wrong thing to do. So this is why we don’t just automatically change, you know, pull the rug out from under you and you do need to deal with with rotation. But the good news is we have some good tools that we’ve introduced in the last maybe like three years.

AVCaptureDevice rotation coordinator is something that can help you either dial in your what the correct rotation degree should be for preview or to keep something horizon level upright, you know, so we have those, those two flavors that you can ask it. And based on that, you can know how you need to rotate the images afterwards. And then you can also just tell the framework to do it for you by talking to your video capture connection.

You can set the video rotation angle on it, and then we will automatically rotate it for you. The way that we rotate it may be different depending on the output. You know, if you’re dealing with video data output and you tell us to rotate the video, we will physically rotate the buffers.

If you’re using a photo output, we don’t have to physically rotate them. We’ll use Exif data to tell you to tell it, rotate it on playback. And same thing with movies. It would be great if for the asker of this question, who was it? Joshua or no Michael row one great question.

We have the new iPhone 17 for the first time, have a front facing camera that’s oriented differently than on previous iPhones, which has caused some people some consternation. We talked about that in a member of my team. Tracy talked about that in a session about the new Center Stage camera, and it’s called support the Center Stage front camera in your iOS app, and there is a portion of that session dedicated to the vagaries of rotation and how to do it correctly. You know, so that your defended against future changes to ID and rotation. You can write the code now and it’ll be correct in the future.

That’s great. Thank you. The next question is about 24 megapixel image capture from Joshua. I hope I pronounced it well. Is it possible to capture 24 megapixel images with depth data data on default photo capture? Yes. So we covered 24 megapixels in the session in one of the sessions this year regarding capturing high resolution images. And there’s a couple of details that you need to follow first.

24 megapixel processing is time consuming enough that we require that you opt into deferred processing. So and second, you should make sure that you opt in into the max folder dimensions. So it actually supports the device that you’ve selected, supports 24 megapixels. But but but if you do that, if you select quality quality prioritization as well as enable, enable depth data delivery, you would you will get captures with depth. Great.

Great. Let’s go to oh, we got so. Many good questions. Yeah, really good questions. Thank you so much for submitting all these questions. We have a really good one here. A group, labs shot an iPhone. Also, what APIs and technologies can you use to build such a multi-camera streaming apps? Nice. The answer is yes. I’m carrying an iPhone. There are many iPhones in tripods everywhere.

You know, last year we did so much with iPhone 17 front camera. We called it the year of the selfie internally, but it was also the year of other cool stuff. We it was probably the biggest year we’ve ever had for pro video. So we had a number of new features introduced for, you know, pro settings.

You can now use your iPhone and get amazing quality out of it, like the ProRes and ProRAW that we’ve talked about, but also just for a functionality, like if you want to do a multicam shoot. There were two new features that were specifically great for this. One of them is locked frame duration.

So if you’re working in a pro environment, you need to make sure that your frame rate is exactly 29.97. No deviation ever. It’s not enough to just try to set your min and max frame rate to the same thing. We introduced a locked frame duration API that ensures that the video is rock solid. 29.97 or 20 4 or 20 5 or 60, or whatever you need it to be, and that the audio is synchronized to that.

Furthermore, there is an extension to that, which is that you can ask for an external sync source to be the one that multiple iPhones synchronize to, which is called Genlock. And it’s something that’s, that’s used in pro environments all the time to make sure that all of the cameras are synced up. And when you go put them all in a timeline, there’s no tearing. Everything is exactly at the same start time.

And so using a third party Blackmagic Pro Dock, you can plug one of those into an iPhone. You can use an external genlock generator, you can put them all on that same time source, and then all the recordings will be perfectly synchronized when you bring them into final cut later.

And then lastly, we also introduced time code generation so you can get time code from an external source through your AVCaptureSession as an output now, and you can associate time code with video frames that you store to a movie. So in addition to having metadata, video, and audio, you’ve also got a timecode track, which makes it much easier in post to go and say go to timecode number, whatever. So yeah, we’re using it here.

There’s also a ton of API that you can use for that too. Yeah. Absolutely. Maybe just to plug the AV Pro video storage API that we just released this week. So that’s, you know, if you’re using ProRes for your like 4K 30 video captures, you know, we’re writing a ton of data to disk and AV pro video storage really helps to give you that deterministic file write speeds. So that way you’re not dropping any frames. So we talk a little bit about in the session that I had this week, but there’s a lot of documentation on it. It’s a, I think a pretty cool feature for people to start using.

Totally great. Thank you. And I think we have time for just one more question. Let’s just give it a brief answer. Is there a niche new API that may not be talked about so much? I think Jake just hit it there. That brand new one that we introduced is a great performance optimization. Yeah. Yeah. If you are interested in ProRes, if you want to capture to ProRes, you really should use this new pro video storage API.

It gives you a pre-allocated file on your disk that you can write to. Even if your phone is old and it’s got a fragmented disk. You can be sure that it’s not going to drop frames. So, you know, we’re, we’re retrofitting older phones with the same API. You can use it and we encourage you to use it. It may have worked before. Okay. But now you can ensure that it will work well for years to come.

It’s great for professionals to type like this, right? Yeah. Yeah. The settings UI is actually pretty nice for it to where you get to. Users can control exactly how much they actually want to dedicate to this. There’s just a little bit for your whole your whole disk space. Another great one, Davide, is the the ProRAW.

We have a new, a new raw nine engine that is on, on the 27th version of the OS that I think is going to blow everybody’s minds away, because this is an ML solution for problems like denoising. Yeah. Raw files from third party cameras and the outputs are outstanding.

Yeah. And unfortunately we won’t be able to double click on that one today. But there is a great session by David Hayward or, you know, on a camera team enhanced raw image processing with its core image. So I would, you know, anyone who is interested in that, please watch that. And it talks about the difference between row eight and row nine.

And that’s about all the time we have today for this group lab. Thank you everyone for joining us today. And big thanks to all our panelists for such an insightful conversation. I hope you enjoyed it as well. As I mentioned, as I mentioned earlier, if you have more questions, Developer Forums is a great place to get them answered. And don’t forget, you can file bugs or feature requests with feedback assistant. We actually do read them.

And actually speaking of feedback, we’ll send you a survey link via email. And your participation in this survey is so important for us. It will help us make this future events better in future. So please fill out that form. And with that, thank you again and hope you have a great WWDC.