Good morning. I'm Vince Utley, and I manage the QuickTime engineering team. And I wanted to thank you all for attending this session, especially, I know it's very early for some of you. I also want to thank you for your continued commitment to QuickTime technologies. Last year, we had an overview session. It wasn't this well attended. So we restructured it, and I'd be very interested in your feedback afterwards. We want to make sure that there's enough substance in it for you.

So part of what we're doing is introducing you to the other sessions we have. We want to tease you and motivate you to attend them, but we want to make sure there's enough substance in this presentation. But the fact that last year was at a much more reasonable hour, but only this is much more heavily attended, we appreciate that you're here.

So I'm going to give a short introduction to the sessions that are available to you at the conference, and talk briefly about what you'll be hearing this morning from the engineering team. So... I'll speak briefly to these points. We are advancing QuickTime, even though the code base is pretty old. And there are some challenges in doing so. And I'll speak briefly to that.

We'll talk about some of the new features we're adding in audio and video plumbing. And then we'll talk about some, what do our annual provision of best practices to guide the new developers to the right set of APIs and to enhance the skillset of the experienced developers. So I don't have any iPod giveaways, but can anyone tell me how old QuickTime is? QuickTime framework, not the logo.

End of '91, yeah, December '91 when we introduced it. So it's showing its age. In terms of Silicon Valley technologies, 15 years is an eternity. And thank you, Brad, for finding this image. So it is old, but part of what we're trying to do with this image is to debunk the notion that QuickTime is no longer usable or useful.

And thank you, Brad, for finding this image. So it is old, but part of what we're trying to do with this image is to debunk the notion that QuickTime is no longer usable or useful. because of our historical commitment to backward compatibility, We depend on technologies like QuickDraw that are appropriate to move into the 64-bit world and, in fact, even appropriate to retain in our long-term plans. So we're doing our best, but at certain areas we've reached our limit to how we can extend QuickTime.

However, having said that, the last four years the team has spent an incredible amount of effort redoing large portions of the AV plumbing. and we continue to make enhancements which we'll speak about today in detail. And in addition, we want to be able to help you integrate QuickTime into the more modern frameworks that are available on the platform.

And the APIs we've introduced in these renovations carry with them modern idioms that are based on modern programming practices. What we want to do is encourage you, one important message we want to give you today and the rest of the conference is to move forward with us. Please adopt these new APIs.

It's only through these APIs that you're going to be able to exploit the introduction of new technologies as they become available on the platform. So whether it be codec acceleration through GPU, if it's available, or the lower-level facilities, if you think really low, fragrant programming level, later on top of that is GL and core image. QuickTime rests on that to take advantage of more powerful media processing capabilities. And that's the only way you're going to be able to get to those resources.

And if you have to change your code, it's work to get there. We understand that. But it's work in the right direction. Even if the API is changed in the future, what we're quite encouraged to do is the design and the algorithms are going to be consistent with the programming model that we're endorsing on the platform.

It's consistent with all the frameworks. This is a picture that Tim Monroe is going to deconstruct for you in more detail later. But it's a picture you should become familiar with because it's one of the icons of our new programming model. And part of this is to ensure you that you're in safe hands with QtKit.

We illustrate, so the bottom portion of the diagram shows how we can modify the underlying pieces, underlying components of the technology, but have an abstraction layer on top of it to protect you from the API turmoil underneath it. And it helps us suss out and vet some of the issues in these low-level components before we give you exposure to the low-level C APIs.

QtKit is not new, so some of this information may not be new to you either. As I mentioned, it's a high-level programming interface. Currently, the only way you can get access to any kind of 64-bit support, and we've added a new functionality, AV Capture, that we're exposing first through QtKit, and you'll see some demos of that later.

People are adopting QtKit rapidly, and even internal to the company. Other applications within Apple are using this. Michael Jones has been one of our biggest evangelists. He spoke, I think, earlier this, maybe yesterday. He spoke in previous years, and not only is he using it, he's even using the new AV Capture capability. And they developed a very sophisticated animation production tool at Pixar. So we know what it's capable of doing.

quite a bit. In the area of audio, we did a complete renovation of sound media, re-hosting it on Core Audio, and that was a significant effort. And if you have adopted that, then you're familiar with audio context, and Brad's going to talk about some enhancements we've made to that.

Actively deprecating. So I think Apple is actually very generous about deprecating. We don't really deprecate. We don't throw things away. We allow you to keep building your apps. But we are encouraging you, if you're using Sound Manager, if you're using Sequence Grabber, please stop. We have newer, better things for you to use.

The same with video. When we, in QuickTime 7, we added B-frame support. And if you've, the ability to do that, and a code base that was never designed to accommodate modern codecs was huge. It almost killed us. But we can now deliver to you this very sophisticated codec, the 264 codec.

If modern compression and decompression techniques require a whole different kind of data flow model, and what we did was using, again, a new metaphor, a new construct called visual context, we can allow you to have buffers in flight. So moving off of the old shallow ICM buffer model was significant. I suspect that in changing your applications to adopt these new technologies, you're also going to have to build a more sophisticated memory management model. So again, this is consistent with modern practices. We're asking you to move forward with us.

Another significant factor of the modifications we made is to allow you to integrate into these other frameworks. And Ken's going to talk to you about what we did to allow you to integrate into core animation in the Quartz Composer frameworks. So in the Aperture mode, we've been littered with bugs about this.

And what we attempted to do here was be able to help users across the entire spectrum. The person just wants to open up their movie and have it look consistent no matter what application they use. And the video production expert who wants to be able to tag their content correctly and deal with dirty video.

So, feedback session. A couple things we'd like you to do. Do what you can with QtKit while you're here, especially the 64-bit piece of it. And come see us. Come to the labs and give us feedback as soon as you can so that we have time to make changes where appropriate before we ship Leopard.

and find a QuickTime engineer, ask all the questions you can, and give us good feedback. The last comment is about the session. I'm pointing you to the feedback session. In the past, it's been a place where people come to vent, and I understand your need for that relief. But what I want to encourage you to do is bring us some ideas.

We're trying to move forward. We want to know what is missing in media services on the platform for you. What is inhibiting you to innovate? If you come talk to us about a bug that someone filed 10 years ago, this radar bug, we're not going to fix it.

Took me a while to dig that up. And if you don't make it to any of the labs, but you do make it to the beer bus tomorrow, find somebody wearing this t-shirt and give him or her your feedback or ask questions. So I just wanted to thank the team, the QuickTime team, who are driving the sessions and did all the work for the code that we're delivering, the demos that we're doing here at the show. And I'll hand it off first to Tim.

Thank you, Vince. The number one feature request for QtKit from the very day it shipped was for us to give you a way to grab audio and video data from devices attached to your computer. So we went ahead and we've added some new AV capture classes to QtKit. The important thing for you to understand is that this is not just a wrapper on the sequence grabber.

On the diagram that Vince showed earlier, you saw there was a new capture engine sort of at the bottom. So that is a pro-grade level capture engine that we are now exposing through QtKit. There is no direct CAPI to that engine, so if you want to get the capabilities that we have implemented, you need to use QtKit. So what does it give us that, say, sequence grabber didn't? Well, maybe most important, it gives us accurate audio. Visual synchronization. This was sometimes very difficult to do with sequence grabber, and here we just got it right. Another feature that it gives us is frame accurate capture.

So you can say, I want to start capturing at this particular time code and I want exactly 245 frames and we'll write that into the file for you. Another thing it gives us is transport controls. So if you're hooked up to a camcorder that can fast forward or rewind, we can do that through our API.

And there are more features that I've added. We'll talk about in a second. So let's just get straight to a demo of these new classes. And could I have the demo machine, please? It looks like we've gone to sleep here. There we go. This is a bit tricky because of technical reasons. We're all having to look down at this monitor here to do our work.

So if I look like I'm sort of not paying attention to this monitor, well, that's because I'm not. So I'm just going to launch this application called QT Recorder. This is on your DVD, the complete source code for this. So you can build and run this yourself now with the seed you have. There's one restriction for the seed you have that won't work, which is that this little, let me see where my, there we go. This device selection pop-up will not work in the seed you have.

But if you come into the lab and you play with it there, it should work fine. And what do we have there? I can't read that. What does that top thing say? All right, so I can select various flavors of built-in audio and the eyesight with its audio. And let's just go ahead and do a capture here. So I'll come down and hit this button. Testing, testing, one, two, three. And I'll stop it. It'll ask me where I want to save that movie.

I'll save it. It'll take a second to write that out to disk. Did foo.move show up down there? Is that it there? And let's see.

Is that there? And it automatically switched, so we have on-the-fly device selection and device detection. And so I'm just going to go ahead, and the lighting here isn't that good, but I'll see what I can capture with HD. So let's capture maybe my monitor here. That's not going to look very much fun. Not much light there. How about this? All right. So I've stopped that. We'll save it.

Again, that'll take a second to write itself out to disk. And somewhere down there, is this the... And it's high-def, so it's big. So let me just try and fit that on the monitor here. And of course, I didn't take anything very interesting, but if I had focused on something wonderful, we'd have nice high-definition video capture there. Okay, so let's go back to slides, please.

And it's high-def, so it's big. So let me just try and fit that on the monitor here. And of course, I didn't take anything very interesting, but if I had focused on something wonderful, we'd have nice high-definition video capture there. Okay, so let's go back to slides, please.

So let me point you to the devices that you have, and actually, this list is incorrect. The core audio HAL devices are also in the seed that you have, so that should be yellowed in this slide. We will support DV devices, and again, if you want to do the ProDV formats, you need Final Cut Pro installed.

And finally, I think will be interesting to some of you, we're going to grandfather in sequence grabber video devices. So if you have existing devices that you work with, you'll be able to use those, too, but again, not in the seed that you have in your hands. So let me point you to... the main QT Kit session, which is tomorrow afternoon.

That's the Building Multimedia Applications with QT Kit. And then we have a couple of labs which are actually going on contemporaneously Friday morning in the Graphics and Media Lab. So with that, let me bring up Brad Ford to talk to us about modern QuickTime audio programming techniques. Thank you.

Thanks, Tim, and thank all of you for getting up as early as I had to get up. I'm going to give you an overview of the audio subsystem of QuickTime, and I'd like to start by approaching it from a question that we get very often on the QuickTime API list and on the Core Audio list. Which do I use? There is this set of Core Audio APIs, and there are the QuickTime APIs. I'm an audio programmer, or I'm interested in audio in my app.

Which set of APIs are appropriate for me to use? Let's talk about why you might want to choose QuickTime audio if we're doing your import, decode, export, et cetera. In the area of import and decode, it's all about the file formats. When you use QuickTime, you get a host of file formats, some of which fall into the purview of Core Audio and the Audio File API. Some of them do not. So what you find when you import using the QuickTime Audio APIs is a superset of what you would get if you use the Core Audio Audio File APIs alone.

For instance, the .mov file format. .mov. many other video/audio mixed formats, as well as third-party importers like for DivX and WMV. In the area of data formats, well, we can decode a lot of different formats with QuickTime, including all of the ones that you might get with the Core Audio decoders and sound decompressors. We also have some third-party decoders that only work with QuickTime.

In the area of playback, well, probably the biggest reason to use QuickTime for audio is because you have video that you want to play at the same time as the audio. If you want to do that, you can write an engine yourself that does synchronization, startup sync, drift sync, or you can use QuickTime. We tend to do that pretty well. Also, we take care of media time scaling for you. For instance, if you have some media that has an edit in it, so you have one section of the media that's supposed to play at a different rate than the rest.

For instance, you have one section that's supposed to be played at 1.5x. When you play that media using QuickTime, it will automatically take care of that scaled edit for you. We also give you these last three on the slide, pitch and rate control, volume, spectral level metering, sample accurate multi-track mixing. None of that is... uh... specific to QuickTime. You can get all of that using Core Audio APIs.

It's just that when you use QuickTime, typically there's an easier, higher level interface, so you write fewer lines of code. For instance, if you're going to do pitch and rate control using Core Audio, you would perhaps make an AU graph in which you inserted an AU very speed unit, and then do all the setup necessary to configure that. Whereas with the QuickTime audio interfaces, you just open a movie and play it at some rate that's not 1, and you automatically get the... the pitch and rate controls.

We introduced in QuickTime 7 movie audio extraction, which is a great set of APIs for getting mixed PCM audio out of a movie. This is something that people have been doing with various success by dipping down to the media sample level and decompressing samples themselves. We've had a lot of good feedback about this API, movie audio extraction, so we know that people are using it.

It gives you access to the decoded and mixed movie audio samples. So that means if you have some source media that has multiple tracks, what you get out is a summary mix of all of that. You don't have to dip down to each track individually and then perform the mix yourself. We do it for you.

And assuming that all of the audio decoder components involved in the audio playback chain are threads for the media that you care about, we support multiple threads, which means you can extract movie audio on a different thread than the main thread or the thread on which you open the movie. It also means that you don't have to work with the audio converter directly. That might not scare some of you, but some of you it might really scare.

In the area of encode and export, well, it should come as no surprise that we encode to a lot of formats, including all of the formats that Core Audio supports. There is a caveat that I want to bring up here because it comes up a lot. If you're interested in AAC and AMR, which is not listed here, and you want to encode to them on Windows, you do need to pay a licensing fee. We cover that blanket license for you on the Mac. If you're a developer on the Mac, you can encode to AAC for free. If you are doing so on Windows, you need to acquire the license and pay some money.

We also give you the standard audio compression component, or SC audio, to do compression cross-platform. I'll talk about that in a minute. This is a brand new feature in QuickTime 7.1, so you might not be familiar with it. We give you the file formats that Core Audio does, plus .mov, mp4, etc. So really what QuickTime Audio, using the QuickTime level of APIs, gives you is a uniform code path for opening, playback, import/export, and code.

Now, I've said a lot of nice things about QuickTime. I hopefully didn't detract from Core Audio because there are definitely times when Core Audio is the more appropriate API to use. So when would you want to use Core Audio? If you're an audio-only app, an app that does a lot of signal processing with audio, or you want to tap in directly to audio unit features, then QuickTime might just get in your way. It might be a level of ease that you don't want to pay for. You'd rather go down to the lower level and get the finer level of control.

So let's talk about, very briefly, QuickTime Audio API best practices. Vince already talked about, in general, what the best practices are to use and the strategies for going forward. And he showed you this slide. I just wanted to bring this back before you because this picture is a perception of QuickTime that will never go away unless we perform our due diligence and educate when necessary. And also, make the API level as modern as the internals are.

So we need to do our job and deprecate when the system changes enough that it no longer makes sense to have that set of APIs around. One of these is Sound Manager. So Sound Manager, if you came yesterday to the core audio sessions, you know that it is officially deprecated in Leopard. And let's talk about what that means to you. So Sound Manager is... Deprecated and leopard, let's have a moment of silence for sound manager.

If you are a codec writer, this means you should not be writing sound compressors or decompressors. Here's the biggie. Most people holding on to Sound Manager are doing so because they just want that sound converter. Well, that's deprecated too. If you're using a sound mixer, that's deprecated. In the area of capture, sound input components. In the area of playback, SDEVs.

And this one's a little bit more vacuous. The use of legacy QuickTime audio interfaces is discouraged. Let me tell you what that means. SG New Channel, for instance, if you're using Sequence Grabber, is an API that you use to create a new SG channel to capture from. This API, you can pass into it the kind of channel that you want to make.

So a video channel, an SG audio media type channel, or this sound media type one. That one, the sound media type one, happens to use a sound input component underneath, which is deprecated. We can't deprecate our API because it serves multiple purposes. But know that that particular code path, creating an SG channel of sound media, should be considered deprecated because we're no longer maintaining that code path.

I've told you a lot about what you should not use, but I'm not going to tell you what you should use instead. You'll have to come to session 223 later today at 5 p.m. to find out what to use instead. I hope you're really scared so that you'll come.

Now let's talk about SC Audio Compression APIs. New in QuickTime 7.1, 7.1 was released about four or five months ago. The SDK just became available this week. The 7.1 SDK for Windows is now live. The Mac OS X one is live, and it's also in the Xcode 2.4 installation. So you have these new SDKs available. SC Audio Compression is available on Mac and Windows. Look in QuickTimeComponents.h. And there's a sample code that you can Google for called SC Audio Compress.

What is it? It's the modern replacement for sound converter that's cross-platform. So when you say to us on the QuickTime API list, I need to know how to do such and such with sound converter. And we reply, oh, don't use sound converter. Use audio converter, you fool. And you say, well, I need it on Windows. And we have no answer for that. Well, finally we do have an answer. Use SC Audio compression instead.

It can do all that audio converter can do, plus mixing. So if you want to go from a 5.1 to a stereo mix, and additionally perform an encode. You can do all of that with a single API call. But enough talk, let's do a demo. Can we switch to demo machine please? I guess we went to sleep again.

And now we're back. OK, so I'm going to bring up a version of QuickTime Player in Leopard. SC Audio Compression is a QuickTime 7.1 feature, not a Leopard-only feature. It is shipping now. You can use it now. But what I'm about to show you is a Leopard feature.

So I'm going to close Tim's app. And up here I have an audio-only file. This is of a concert that I gave two weeks ago. Could we pump up the volume? So I'm going to close Tim's app. And up here I have an audio-only file. This is of a concert that I gave two weeks ago.

Could we pump up the volume? So I'm going to go into my sound settings dialog. Any of you that are familiar with QuickTime Pro probably have used this before. That's the sound settings dialog. It lets you configure an output and export to a new file format or just encode to a different format. And to those of you who have used this a lot, you'll notice that these two buttons down here are new.

We found that in feedback given to us, these advanced feature dialogues are great for those of you who know what you're doing with audio or video, but for those of you who just want it to sound right, this might be overwhelming. Because once you get into this dialogue, all of a sudden you've got eight or nine different choices for codecs and lots of sample rates and lots of channel choices.

And so you'll wind up iterating over this several times, perhaps exporting your 20-minute content and then listening to it and finding it's not exactly what you wanted. It's not the right quality setting. So using the SC Audio Compression APIs, we already know the format that you want to go to. We just take a 10-second segment of your source and loop it over and over and apply the settings that you currently have selected so you can hear it as you're making live updates to the settings.

And hear how that will affect the sound. So I'm going to go ahead and play this. So what if I take the sample rate down to eight? Maybe I'll choose AAC and listen to how it sounds if I select a different bitrate. And at any time I can compare that to the source by hitting play source.

So there you go. That's a nice... Back to slides, please. Hopefully this will bring the frustration level down a level. How did we do that? Well, like I just showed you, there's that dialogue up at the top. That's what you see. Underneath, there's that new call SC Audio Fill Buffer.

It knows about the source movie. It performs an extraction from that source movie and then creates a new destination movie using Add Media Sample 2 and plays it. For a better explanation of what we're doing, including what your code should look like, you'll have to come to audio session 223.

And another great feature that I'd like to spend a few minutes on is the Audio Context Insert API. Movie audio extraction got us... 75% of the way there to what people wanted. This should hopefully cover the rest of you. This gives your application access to our audio rendering path.

That is, you can insert your effects or visualizations or whatever into the audio signal path while QuickTime is playing. You don't have to take care of the synchronization you had to if you were using movie audio extraction. It does real-time rendering, so you get called back when you register an audio context insert. You get called back on the real-time IOPROC thread.

You can also use it in movie audio extraction if you want to insert some effects while extracting PCM audio out of a movie. It's designed for compatibility with audio units. That said, it is a callback interface, so you don't have to be an audio unit in order to plug yourself in. You just have to register a callback to do the processing or your own custom code. If you're on Windows, you can use direct show filters or your own custom code, whatever you would like.

Let's take a look at what the audio context is and how the audio data flow works in QuickTime's audio architecture. We have an abstraction called a device context. That is a wrapper for the audio device that you're playing to. You can get one of these device contexts by having the UID of an audio device and then creating a context for it. There's also a movie audio context, and that is the context in which all of the mixing takes place for the movie.

When you perform a movie audio extraction, there's also an extraction context to which you are playing when you extract PCM audio. You don't see this extraction context, it's just made for you underneath the covers automatically. Inside the movie audio context, you'll see that if there are multiple tracks, they are being mixed down before they're sent to the device context.

If you have a more complicated movie, for instance, here we've got one with three tracks with varying channel valences and varying sample rates, we mix them down to a movie summary mix, that is, we pick the highest sample rate of all of the sample rates, and all of the like channels mix to a common channel. So, for instance, there are two rights, there's one in track two and one in track one. Those mix into a single right in the movie summary mix. Thanks.

And if you register an audio context insert, you plug in right there after the movie audio summary mix. Now, you can ask for a different channel layout than we've provided in the movie summary mix. So, for instance, if you have some processing that must be done in stereo, like the audio unit that you're going to use only accepts stereo input, you can tell QuickTime, I want you to give me stereo, and what I'm going to give back to you is stereo or mono, what have you.

You'll see that that mix is performed, the samples are provided to your callback to the client application, you perform your signal processing, then you hand the samples back. The only caveat there is that you must not change the sample rate, because that would just be mean. Let's go back to the demo machine. I'm going to show you a sample app that was written by an engineer on the QuickTime audio team named Siley, who's great.

This is a Qt Audio Context Insert app. Not surprisingly, it plays QuickTime movies. Like, for instance, "For Brad's." So what can I do with this? Well, I can bring up the-- oh, whoops. That brought it up in QuickTime Player, didn't it? Let's try that again, bringing it up in the Audio Context Insert app.

And we have an audio insert panel over here. So here we can visualize what the context insert looks like. First we have to set a channel layout. I'll go ahead and select stereo. So I'm going to provide the insert with stereo. I'm going to get stereo back out. And then I can choose a Core Audio effects filter to apply to the audio as we're listening to it. So I'll go ahead and play around with it.

Make myself sound like I'm in a concert hall. Or bypass it.

The QuickTime application provides the ability to keep your application on the cutting edge by keeping it on the cutting edge. www.vincetot.com/quicktime-application/ www.vincetot.com/quicktime-application

and other experts at the University of Michigan.

www.vincetot.com/quicktime-application The QuickTime application provides a wide range of features that enable you to keep your application on the cutting edge by taking full advantage of these features. Learn how to keep your application on the cutting edge by taking full advantage of these features.

www.vincetot.com/quicktime-application The QuickTime application is a great way to keep your application on the cutting edge by taking full advantage of these features. Learn how to keep your application on the cutting edge by taking full advantage of these features.

The QuickTime application is a great way to keep your application on the cutting edge.

www.vincetot.com/quicktime-application

www.vincetot.com/quicktime-application The QuickTime application is a great way to keep your application on the cutting edge. Learn how to keep your application on the cutting edge by taking full advantage of these features.

www.vincetot.com/quicktime-academy/ www.vincetot.com/quicktime-application www.vincente.com/quicktime-application- application

www.vincetot.com/quicktime-application/

www.vincetot.com/quicktime-application The QuickTime application provides a complete and complete set of features that enable you to easily and effectively use your application.

www.vincetot.com/quicktime-application

The QuickTime application provides a complete and easy-to-use interface for your Mac OS X.

The QuickTime application is a great way to keep your application on the cutting edge by taking full advantage of these features. Learn how to keep your application on the cutting edge by taking full advantage of these features.

www.vincetot.com/quicktime-application/

www.vincetot.com/quicktime-application

The QuickTime application is a great way to keep your application on the cutting edge by taking full advantage of these features. Learn how to keep your application on the cutting edge by taking full advantage of these features.

www.vincetot.com/quicktime-academy/

www.vincetot.com/quicktime-application/ The QuickTime application provides a complete and easy-to-use interface for your Mac OS X. The QuickTime application provides a complete and easy-to-use interface for your Mac OS X. The QuickTime application is a free, free trial of the QuickTime application. The QuickTime application is a free trial of the QuickTime application. The QuickTime application provides a wide range of features, including the ability to control the volume of audio, sound, and sound effects.

The QuickTime application provides a wide range of features, including the ability to control the volume of audio, sound, and sound effects. www.vincetot.com/quicktime-application/ www.vincetot.com/quicktime-application/

www.vincetot.com/quicktime-application/ The QuickTime application provides the ability to integrate the Internet into your application. The QuickTime application provides the ability to integrate the Internet into your application. www.vincetot.com/quicktime-application/ www.vincetot.com/quicktime-application