Good afternoon.

Well my name is Sam Bushell and I work on Media Frameworks. And that's what we're going to be talking about in this session. We're going to talk about Media Frameworks, old and new. We're going to talk about QuickTime and QTKit and we're going to talk about AV Foundation and AV Kit.

In case you haven't heard, AV Foundation is a new media infrastructure we've been working on at Apple for the last few years. It's common on iOS and OS X. And it's focused on modern media formats like H.264 and AAC. But AV Foundation was built by engineers with deep media experience from building QuickTime over many years.

Let's give a bit of history about QuickTime. QuickTime was a pioneering framework for dealing with digital media. There were some early developer seeds but it first shipped to the public in December 1991. And over the years since 1991, there have been a lot of QuickTime updates delivering an enormous collection of features.

One of our biggest individual feature releases was QuickTime 7 which is shipped as part of Mac OS 10.4 Tiger. This introduced big deal features like H.264 and the QTKit Framework. The QTKit Framework introduced an objective C API that wrapped the QuickTime C API for easier integration into Cocoa Apps.

Four years later, we introduced Mac OS 10.6 Snow Leopard. And in that four years, we had built a new media playback pipeline as part of building the iPhone. We called this Core Media. Now we weren't ready to deliver a public API for Core Media at the time. But we delivered-- we added a mode to QTKit where you could use the Core Media playback pipeline to get optimized playback of H.264 and AAC. We called this QuickTime X.

And then in OS 10.7 Lion, we introduced AV Foundation as a public API. In our framework hierarchy AV Foundation sits lower than the UI frameworks of UIKit and AppKit. This means that we can deliver the same API across iOS and OS X. In the OS 10.8 Mountain Lion, we enhanced the AV Foundation API and we introduced the Video Toolbox as a public API. And this year in Mavericks, (I'm still getting used to saying that.) We introduced-- we're introducing AV Kit. AV Kit is the framework where we will put APIs that let you integrate AV Foundation with AppKit.

So, you can see in the last few years we have built in a new stack of media frameworks. We call this the AV Foundation family and this is the direction we're headed. This is our focus in the media systems group. We have not been adding new APIs to QuickTime or QTKit for-- for some time now. I said that with one small asterisk to come back to later.

So, as of Mavericks, the QuickTime C Framework and the QTKit Framework are deprecated. What this means is that we've marked these APIs as deprecated in the header files. And that means while your code will still compile you will get deprecation warnings like these. But your apps will still run.

Let's emphasize that. Your apps will still run. While were on the topic, the QuickTime Movie file format is still the primary file format for AV Foundation and for QuickTime Player. It's still supported. What we are deprecating is the QuickTime C API and the QuickTime, I'm sorry, the QTKit objective C API. The file format is still supported. And also on topic the movie file format's ISO cousin MPEG4 is also still supported.

So AV Foundation is the future of media applications on OS X. Let's talk a little bit about how we built AV Foundation. As I said, AV Foundation was built by engineers with deep media experience from building QuickTime over many years. We wanted to build a new platform for media that could do things that QuickTime couldn't do and go places that QuickTime couldn't do-- couldn't go.

So we built it on the same kinds of modern foundations that the rest of Apple has been moving towards, core foundation and foundation, GCD and Blocks and so forth. Now QuickTime was built on a modern foundation. It's just that it was a modern foundation in 1991 when System 7 was new. So a lot of the technologies that it was sitting on top of have or ones that we have moved apart, moved away from.

Where QuickTime's APIs-- QuickTime is often said to have a lot of APIs but if you count these up by numbers and if you look into the header files, a lot of these APIs are simply exposing the implementation of QuickTime. And sometimes that's a good fit for how you want to extend QuickTime and use QuickTime but sometimes its not.

With AV Foundation we have taken care to design our APIs to be a good fit for how clients will use them. And so QuickTime's APIs are monolithic and in some of these cases we have taken the opportunity to re-factor them in AV Foundation into multiple objects. In some cases these makes the AV Foundation APIs much more flexible.

Owing the QuickTime's pre-OS X heritage, many of its APIs were only designed to work on the main thread. Now we did some retrofitting. We did some refitting and make many of the APIs multi-thread savvy in later years. But that left the overall rules if you had coming to it as a first timer the overall rules were somewhat awkward and complicated. AV Foundation is designed to be deeply multithreaded. There are two goals for this. One is to take maximum advantage of multi-core hardware like we have today. And the other is to improve application responsiveness by letting you take slow, blocking operations and move them off the main thread.

AV Foundation is able to take advantage of hardware acceleration in ways that QuickTime could not such as video encoding. And we made major design decisions in AV Foundation with the goal of power efficiency. Remember this is a framework that was first delivered on the iPhone. It is a device that we tested and run on a battery that fits in your pocket and still delivered long hours of playback time. Some of these power efficiency design decisions could not have been made compatible with the QuickTime architecture.

And finally QuickTime's integration, deep tight integration with legacy frameworks, QuickDraw in particular, means that it cannot escape the 32-bit world. Now you may know that you are able to use QTKit a 64-bit application. But for playback and editing at least, what's happening when you do that is that it's running a 32-bit background process to use QuickTime.

AV Foundation has designed-- been designed to be 64-bit native from day one. So do you get the message here? There are lots of great reasons why AV Foundation is the direction that we're headed and why it should also be the direction that your apps are headed as well.

And AV Foundation supports the media types that matter, video, audio, closed captions and subtitles, chapters, and time code. Now as I mentioned QuickTime has a history, a vast history with many features added over the years. And some of these were breakthrough features when they were added in the mid 1990s. But the world has moved on.

If you look at this list you would think to yourself many of theses APIs-- many of these features would be superseded by basic things that we take for granted today like HTML 5. And HTTP live streaming has proved to be much more scalable for delivery than RTP Streaming ever was. That brings us to codecs.

So QuickTime has a vast history. It has collected many, many codecs over its long history, and many of these are rather old and have been superseded. The categories of codecs that are still relevant today basically are in three groups. Those that are used for delivery like H.264 and AAC and we use JPEG for chapter images. Codecs that are used for production workflows and editing, these are often called Mezzanine codecs. And Codecs that are used as-- for import from captures and device, standard formats that we need to be able to import from. These three categories of codecs are still supported by AV Foundation.

And that leaves a number that are left by the wayside. And if you have really good eyesight, you might be able to look at this and say well hang on some of those aren't really what I think it as video codecs anyhow. And the way that QuickTime's architecture was structured still image formats and video filters and effects also had to be registered as video decoders. Well nowadays, we have the image IO Framework which delivers still image format support and we have designed different ways of integrating video filters and effects into the playback pipeline.

But our media is personally important to us. Some of the media in these formats is irreplaceable. And just because the formats are out of date, it doesn't mean we want to orphan the content. So we've provided a mechanisms starting in Mavericks to help you-- to help you migrate the content in these legacy containers, legacy formats into AV Foundation-supported formats. It's called QTMovieModernizer. It's automatically run by QuickTime Player when it discovers a legacy codec in a movie file you're opening. It works with third-party QuickTime components.

And it's provided as a new API in Mavericks so you can do the same in your apps. The way it works is it produces a new copy of the movie in an AV Foundation supported format. Normally this will be H.264 in AAC. But there are some cases for production workflows where you'd want to use Apple ProRes and PCM instead. If the content has an alpha channel, then the alpha channel can be preserved in Apple ProRes 4444. Now, it's delivered as part of QTKit. And this is something we don't do very often. We're adding a new API to a deprecated framework.

But this is intentional because there's a very specific message we want to send here. QTMovieModernizer will be available to you exactly as long as QTKit and those legacy codecs are still available. So now is a good time to gather up your legacy media and bring it across the bridge.

So that's our story here about QuickTime and QTKit. QuickTime and QTKit are deprecated and we have been building a new media framework brick-by-brick, foundation by foundation as it were called the AV-- the AV Foundation family. In the rest of this talk, we're going to talk about how to migrate existing applications to the AV Foundation framework family. And first I'll introduce Stefan Hafeneger up to talk about AV kit and introduce it to you for the first time.

Thanks Sam. So I see most of you sitting in the audience here have some kind of media playback in your applications. Let's see. How many of you still use QT Movie View? Hands up. OK, so there's a few. And how many of you already use AV Foundation? OK, so a few more.

And who of you has ever written their own playback controls on top of AVPlayer layer? OK, a few. Yeah, I have written a few as well. Wouldn't it be nice if it didn't have to-- didn't have to deal with that? Here's something new for all of you. We want to make your life as developers easier. And that's why we are adding AV Kit to our stand. AV Kit is a new high level Cocoa Framework. Our goal is to provide you view-level services for media operations on top of AV Foundation and AppKit. AV Kit shows the AVPlayerView.

AVPlayerView is an NSView subclass for displaying audio-visual media. It comes with a set of standardized playback controls and behaviors. This means, you can get the same look and feel that we have been using in QuickTime Player since Snow Leopard, now in your applications. And as a bonus, it takes care of of localization, accessibility, high resolutions, state restoration and so on. And we made it really easy for you to adopt AVPlayerView in combination with the AV Foundation in your applications. So let me walk you through the necessary steps in the demo.

All right, so in Xcode you create a new project. And select OS X Cocoa Application and press Next. The name is going to be AVKitPlayer and we make sure that we created a document-based application. Press Next and save the document or the-- the project on the desktop. Let me make the Window a bit bigger here.

So the first thing that you have to do is to add the AV Foundation and AV Kit Frameworks to our project. So I'm here on the linked frameworks and libraries. I click on the Add button. And then in the New sheet, we set first search for AV Foundation. Oops. I select that, press Add and then we do the same for AV Kit.

Now go up with the mouse to the Info tab. And on the document types, first thing we do is we remove the MyDoc extension. Instead for identifier, we use public.audiovisual-content so that we can open all kinds of audiovisual media in our application. Finally, we switch the row from editor to viewer. So that's-- NS Application doesn't create a document for us automatically. Now on the left we select the documents xib file. We remove the label and then on the right, we search for AVPlayerView.

We're dragging an instance of AVPlayerView onto our document window and resize it so it fits the entire window. In the inspector, we change the control style from default to floating. And this would give us the nice floating HUD that we have in QuickTime player. Next, we show the assistant editor. And here, switch to the document header file. And now with the Control-click we can add an outlet to our document. We call it playerView and hit Connect. Xcode will now complain that AVPlayerView is not declared. So, what's still left to do is to add the header files.

All right, so what have you done so far? First, we create an OS X document application. Second, we aadd the AV Foundation, AV Kit frameworks to our project. Third, we'd set a-- sorry, modify the info plist so we can open all kinds of audiovisual media. And fourth, we drop an AVPlayerView into our document and create outlets so we can reference it from the documents. So let's see how much more we have to do to actually make it work.

So I close the assistant editor and select the document implementation and we scroll all the way down to the bottom. We're not going to use readFromData, instead we implement readFromURL. And here we just return yes. We want to use AV Foundation to open the document for us. So we scroll up to windowControllerDidLoadNib and implement [playerView setPlayer: [AVPlayer playerWithURL: [self fileURL]]] And now, press on Build and Run.

Go to File, Open. Select the movie, press Open. [Background Music] We have a fully functional document-based application for playing videos. OK and it's- [applause] thank you. Press the Play and Pause button to Play and Pause. I can use this Slider to scrub around and we use the Scan button to scan forward and backward. But you can also use the keyboards, for example space, just stops playback and stops playback. We can step with the arrow keys frame by frame, forward and backward. And for those of you who are familiar with the J/K/L navigation in other applications you have this as well.

All right. Wasn't that easy? So let's go one step further. Let's assume your users have some content where they want to cut off the beginning and/or the end of the movie, something we call trim. Let's see how much work we have to do to add these features to our application. So for that, I'm going to open another version of this project.

As you can see, I already set up a trim menu item in the Edit menu. So we now open the document implementation. The only thing we have to do is, in the trim IBAction to call our playerView, beginTrimming WithCompletionHandler and just NULL. But as a good Cocoa Application, we also want to enable and disable the trim menu item. So for that, in validateUserInterfaceItem for the trim IBAction, we implement for trim-- playerView canBeginTrimming.

If we now build and run, and we open the same document as before and go to Edit and trim, you get the same user interface in the QuickTime player. So we can direct the trim handlers to select the shorter portion of the video, hit the Trim button and now we're just playing [Background Music] that-- this portion of the video.

All right. So two lines of code-- sorry. Two lines of code was all we had to do to create a very basic video playback application, using AVPlayerView and AV Foundation. Two more lines of code and we have trimming working as well. Just imagine how long it would have taken us to create these controls from scratch. All this time, you can now invest instead making your applications even better.

So let's talk about a few details now. So how does AVPlayerView work with the AV Foundation class hierarchy? AVPlayerView has a strong reference to an AVPlayer object. This object provides the content of course. AVPlayer itself mentions AVPlayerItem which serves at mid-- which serves as the mutable data structure for an immutable AVAsset. This means in order to provide content for an AVPlayerView, you have to do the following steps. First, you create an AVAsset from an NSURL.

Once you have an AVAsset, you can create an AVPlayerItem. We have AVPlayerItem, you can then create an AVPlayer object. And finally, you associate this AVPlayer object with an AVPlayerView. However, if you really just want to play the content of the movie file on disc, you can do all four steps at once.

So as we showed you earlier in the demo, you can directly create an AVPlayer object from an NSURL and then pass this object to the AVPlayerView. Let's go back to the object graph from earlier for a second. I told you, the first step in order to provide content from AVPlayerView is to create an AVAsset from an NSURL. But what if you're using AVCompositions in your applications instead? All I'm showing today that AVPlayerView would just work fine for AVCompositions as well.

AVPlayerView let's you choose from four different standardized control styles. Which one you pick for your applications is up to you. It really depends on the type of application you have and what looks best in your user interface. Let me walk through the difference real quick. The first style doesn't show any controls, but instead it gives you all the gesture and keyboard events that AVPlayerView implements.

The second style has the controls at bottom of the view. This is the closest match that we provide to the QTMovieView. The third control style has controls in this-- in the floating HUD. This is exactly the same UI that we are using QuickTime Player today. The last style is just a Play/Pause button at the center of the view. That will also show a circular progress indicator during playback. All controls automatically show and hide upon user interaction. You can change the control style in interface-- builder or in code at anytime.

For the second and third control style, we also have Trim controls. The screenshot you can see on the left shows the TrimUI for the floating controlsStyle and the screenshot on the right shows the TrimUI for what we call inline Trim controlsStyle. As you can see, the only difference is the margin around the controls and the graying on the background.

AVPlayerView has dynamic controls. This means that if your content has chapters, additional language or subtitles, it will automatically display-- adds chapter and media selection pop-up buttons to the UI. Some of you might have streaming content in your applications. AVPlayerView will automatically switch to a different set of playback controls when the content type of the current AVPlayerItem changes to streaming. AVPlayerView has also API for customization.

For instance, if you want to allow your users to share their content from-- with an AVPlayerView, all you need to do is to set one property to yes. AVPlayerView add the share-- the OS X Share Button to the UI takes care of everything for you. This includes optimizing the content for the destination if needed.

So let's switch our focus to Trimming now. As I showed you earlier in the demo, it is really easy to add Trimming to your application via AVPlayerView. All you need to do is just call beginTrimmingWith CompletionHandler and NULL as the argument. But since that's not going to really tell you that if it would actually show the TrimUI, you should always call canBeginTrimming first though. This method not only returns no when AVPlayer already shows the UI, but it also makes sure that the current AVPlayerItem can actually be trimmed.

But what if you are really interested in what the user did in the TrimUI. We start off with exactly the same code as before. This time, we are going to implement a completion handler. The CompletionHandler has just one argument of type AVPlayerViewTrimResult. The result that it will return to you will either be AVPlayerViewTrimOKButton or AVPlayerViewTrimCancelButton.

In the former case we apply whatever the user chose in the UI, in the later case we will set the trim selection before dismissing the UI. So I just told you what AVPlayerView does when the user interacts with the TrimUI. I haven't told you yet what it does under the hood though. We're not replacing the current AVPlayerItem or creating AVComposition. Instead, we use existing AV Foundation API and functionality for setting the reverse and forwardPlaybackEndTime on the AVPlayerItem.

The player controls are constantly observing these two values and updating when calling me. One important note here, AVPlayerView does not provide state restoration for the AVPlayer property because we can't guarantee that we will be able to restore this object for you. This means it is your responsibility to restore the content when your users relaunch the applications.

And this, of course, includes the reversePlaybackEndTime. So since we are just setting some properties on the current AVPlayerItem when the user interacts with the TrimUI the underlying AVAsset of course it contains the entire movie, which means that if you use Trimming and the application supports export, you have to set the timeRange on the AV asset export session. So first, you get the reverse and forwardPlaybackEndTime from the AVPlayerItem and then you create a CMTimeRange and set this value on the ExportSession. And this is exactly what we do for sharing.

So wrapping up. AV Kit is a new UI level Cocoa framework for AV Foundation. It provides you view-level services-- sorry, it provides you standard playback and trim controls through the AVPlayerView. With AV Kit you're now able to use the power of AV Foundation without having to write your own custom playback controls with our AVPlayer layer.

So please consider adopting AVPlayerView in your applications. This will make sure that we can provide a consistent look and feel, not only here in Apple's but across all media playback applications in OS X. And if you have-- still not fully sold on AVPlayerView maybe this will convince you.

Starting on OS X Mavericks QuickTime Player uses AV Kit from media playback. The view you see in the middle is the same AVPlayerView that you can use in your applications. With that, let me call Sam back on stage to talk to you about moving from QuickTime and QTKit to AV foundation and AV Kit.

Thank you Stefan. All right. So for the rest of this talk, we're going to talk about what's involved in migrating an existing QuickTime or QTKit application over to AV Foundation and AV Kit. How will you get to AV Foundation? Well, it depends on how you use QuickTime and how you use QTKit. For some developers, it will be a pretty easy change, but for others it may require more refactoring and deeper thought. We're not providing an API for API swap. We've taken the opportunity to change the API model where we think it can make AV Foundation better.

That said, some of the more recently developed QTKit APIs, were actually developed around the same time as their AV Foundation counterparts and so they have a very similar API feel. So, we're going to go through a bunch of API areas and for each of these API areas I will show you-- I'll have some reminders of the kind of QuickTime and QTKit APIs you would have used and then focus on the AV Foundations version and some of the things that make that interesting. So let's start with the basics.

In QuickTime, every AV resource in use is represented by capital M, movie object and for every track, there is a track object and a media object. And QTKit has classes wrapping all of these. In AV Foundation, AV resources are represented by the AVAsset class and for every track there is an AVAssetTrack. What's different in AV Foundation is that AVAsset and AVAssetTrack are immutable. They provide a read-only view of the AV resource.

QuickTime and QTKit have a number of APIs for creating a movie or QTMovie object. In this simplest form they all operate synchronously which means that your main thread is blocked while QuickTime and QTKit go off and unload the file and parse it. With AV Foundation, creating an AVAsset from a URL always finishes very quickly and succeeds, that's because it hasn't actually done anything yet.

You can then use the asynchronous property loading API to request that AV Foundation load values-- load some properties asynchronously and provide a block that'll be called when data-- when enough data has been read and parsed to produce those values. So this lets that happen off on another background queue and lets you keep the main thread responsive.

In QuickTime and QTKit, the same movie object is the one that you would call to start playback or to pause or to set the time or to step. These are called transport controls. With AV Foundation, we have separated out the mutable state related to playback into playback objects. The principle playback object is called AVPlayer and that's where you send rate changes and that means play and pause as well.

Now, AVPlayer is designed to be able to play through a sequence of AVAssets, automatically advancing when each reaches its end time, in some cases with gapless transitions. So this design goal led us to take out the per item state related to playback into its own object called AVPlayerItem.

This is where you tell-- this is the object that you send messages to seek and to step. AVPlayer's subclass AVQueuePlayer can play through a sequence of AVPlayerItems and each will advance when it reaches its end time. So this is an example of how we've refactored the object in order to give us more flexibility in the API.

Another thing to know is that AVPlayerItems default when you seek is to snap to a keyframe. This is because keyframe is the most efficient display for random access and also the most efficient to start playback from. But there's a variant of seekToTime where you can specify a tolerance range. If there's a keyframe inside that tolerance range then we'll snap to it. If there's no keyframe inside that tolerance range then we'll snap-- it will seek exactly the time you specified. If you specified tolerance of 0 then we always seek exactly to the time you specified.

If you've used the QTMovieView to integrate into a view hierarchy then you should check out the new AVPlayerView class in AV Kit that Stefan demonstrated. One thing that's different if you used a QTMovieView is that the transport control methods are not re-exported by the view object. You make those calls directly on the underlying AVPlayer.

And AV Foundation provides the AVPlayerLayer class for integrating into a Core Animation layer hierarchy. It also provides the AVSynchronizedLayer class for synchronizing Core Animation layer animations to match the time of a playerItem as the playerItem's time and rate change, so will the time and rate of that synchronized layer. We've demonstrated in previous year's sessions how to use this class for editing applications. But there are other ways you could use it that aren't editing applications as well.

Another thing to know is that in applications linked on or after iOS 7 or on or after Mavericks by default, AVPlayerLayer and AVPlayerView will both honor the user's choices for media accessibility by-- automatically. This may not work in the current seed but it will work fine in the next seed. We have a session coming up tomorrow on preparing and presenting media for accessibility that goes into more details about what it means for the playback pipeline to honor the user's media accessibility options.

Let's move on to authoring and editing APIs. QuickTime has a number of high level movie transcoding APIs and these are all synchronous. And QTMovie has a writeToFile method which wraps these synchronous APIs. And in QuickTime-- sorry, in OS 10.7 we introduced an asynchronous QTExportSession API. Well, AV Foundation's, AVAssetExportSession API is also asynchronous and it's internal pipeline is deeply multithreaded.

It's also preset based which means that you can offer your users high level options like 1280 by 720, and let AV Foundation choose a well tuned bit-rate for the encoding. If you need more control or if you need access to the media frames-- the data that's going-- going through the pipeline then you can use the combination of the AVAssetReader and AVAssetWriter classes.

With QuickTime, the APIs that you'd use for reading media data from a file were different depending on whether you wanted the data be decoded or not decoded. With AV Foundation, AVAssetReader provides features for both of these. It depends on how you configure it. When you create the AVAssetReader output to decode data-- to retrieve data from a track, if you pass an output settings dictionary then we will decode data into the format you request. If you pass a nil outputSettings dictionary then we won't decode it. Now, I should note that the QuickTime APIs for reading video data are synchronous. They don't begin loading and decoding video data until you request it. By contrast, AVAssetReaders pipeline inside is deeply multithreaded.

The file reading, the video decoding, and the audio decoding all happen asynchronously and in parallel. This means that when you-- when you pull a new media sample out of the queues at the end the latency when you make that request is very, very low. It has probably already decoded it in the background while you were working on the previous frame.

The QuickTime APIs for writing media files are also synchronous and the same with QTKit. AV Foundation provides the AVAssetWriter API and its pipeline is, you guessed it, deeply multithreaded. And you can configure it to encode data, encode the video for you-- encode the audio for you or you can use it to write already prepared data to a movie file.

AVAssetWriter also provides you much more control over how you encode. For some developers they find that this control is actually too much control and they want preset access. They'd like to be able to use presets like AVAssetExportSession. Well, that's what we've done in Mavericks, we've added AVOutputSettingsAssistant which lets you use presets as a starting point for picking good output settings for AVAssetWriter. So, AVOutputSettingsAssistant is new in Mavericks and new in iOS 7.

Also in the session tomorrow on preparing and presenting media for accessibility we'll go into more detail about how to use AVAssetWriter to construct subtitle tracks and alternate audio tracks. But let me go into more detail about the internal pipeline of AVAssetWriter. Like I said it's deeply multithreaded which means that the video encoding and the audio encoding, and the file writing can all happen asynchronously and in parallel with each other.

But let's think about how we construct movies, in order to construct a movie for efficient playback we want to alternate, we want to interleave, audio and video and any of the track types. This means that the file writing queue, the file writing process inside AVAssetWriter will alternate between the different tracks, writing a portion of media data from each in turn.

Well, this means that sometimes, if that file writing code runs out of data on one track, then pretty soon it will stop consuming media data on the other tracks until there is more data on that track that it's waiting for. In fact, at some point it's going to need to be able to say to you at the API level, "Stop. I'm not ready for more.

I don't want more data on these tracks until I have more data on that track," or for you to say, "Actually, I'm completely done giving you data on that other track." So, there is a flow control API built into AVAssetWriter input where you can query to say, "Is this input ready for more media data?" And there's also an API utility where you can ask it to call a block over and over and over just as long as it's ready for more input.

If you wanted to get access to video frames during playback, for example, to integrate them into a custom OpenGL rendering, QuickTime and QTKit provided a way that you could set up a visual context to retrieve those video frames. AV Foundation's API for doing this is called AVPlayerItemVideoOutput. And similarly, there was also a way, that you could set up an audio context. And the audio context was a way that you could tap into the audio waveform as it went past, either for analysis or even to modify the waveform to provide some kind of audio effect.

AV Foundation in Mavericks provides this API as well. It's called an audio processing tap and it's an API that you install an object onto an AV audio mix and then you can install that AV audio mix on to an AVPlayerItem or on to an AVAssetExportSession, or onto an AVAssetReaderAudioMixOutput. (It's like Dr. Seuss.) Sometimes, you just want to get a still image out of a video file, QTKit provided a utility method for doing this. AV Foundation provides a whole class and this class has both a synchronous one-shot API and an asynchronous batch API.

Editing. So, all three of these, QuickTime, QTKit and AV Foundation - all provide APIs for inserting, deleting, and scaling segments expressed as time ranges and these are really high level, powerful APIs that you can use to deliver high level user focused editing experiences. With QuickTime and QTKit, these are methods on the movie and the QTMovie objects.

With AV Foundation, AVAsset is immutable. So it can't be there. Instead, we have subclass of AVAsset called AVComposition and it has immutable and mutable variants. Similarly, AVAssetTrack has AVCompositionTrack and there's an AVMutableCompositionTrack as well. But it is important to know that the API model is actually different in AV Foundation.

This is a bit of an advanced topic here. But in all three cases, QuickTime, QTKit, and AV Foundation, these editing APIs work by manipulating a data structure called an edit list. An edit list is part of a track inside the corresponding objects and it is a data structure that says play A for B seconds and then play C for D seconds, and so on. Well, with QuickTime and QTKit, to insert a segment of one movie into another requires a bunch of sample table copying. This is because in this API, a movie's edit list can only refer to that own movie's sample tables.

AV Foundation's AVComposition's edit lists do not require that the sample table be in the same asset. So, it isn't necessary for the sample table copying to have occurred and that makes it a bit more efficient when you're doing the editing. It also means that if you insert a bunch of-- insert a bunch of clips and then you delete them, you're not left with leftover sample table references that you don't need. Another way of thinking about this is that QuickTime and AV Foundation have different ways that they do their references. AV Foundation is using a segment-level file reference and QuickTime and QTKit are using a sample-level reference.

If you have been using the QTKit metadata API, then here's some good news. AV Foundation's AVMetadataItem API is very similar. In fact, if you look up there, you might be hard pressed to find a difference. Well, one little difference that's on the slide is that it's using Objective-C property, Objective-C 2 property, API's as a way of describing the methods. That's not really so interesting. The more interesting thing is that metadata is no longer loaded eagerly.

So, just like those other properties that you can use the asynchronous key value loading API, you use the same API to request that the metadata be loaded and it can be loaded on the background thread and keep your main thread free up for user responsiveness. Let's move on to capture APIs.

The QTKit and AV Foundation capture APIs are also broadly very similar. In both cases, you have a capture session object and for that capture session object you add inputs and outputs, and previews. Now, with QTKit and AV Foundation, in both of those, there is a simple mode of the API in which you automatically-- sorry, in which the session automatically helps you build the connections. When you add the inputs and the outputs, it automatically builds capture connections by matching media types. So, it links up all of the video inputs to outputs and it links up all of the audio inputs to audio outputs.

But with AV Foundation there is also an alternative more advanced form of the API, and in this more advanced form, you use a different method and you say, "No connections please. I don't want you to build the connections automatically. I will make explicit calls to set them up myself." And this is a better choice if you have multiple video inputs, multiple cameras, or if you have multiple previews, or if you have multiple file outputs because you can construct exactly the graph that you want by constructing those connections yourself.

So, going through the inputs and outputs, most of the inputs and outputs are the same but there are some extensions and some improvements. AV Foundation has a CaptureInput object for capturing from the screen and we've done some work in Mavericks to improve performance of screen capture when the cursor is visible in the captured frames.

In QTKit when you added a video preview layer, it would automatically create a video preview output and that output object was how you configured certain properties. Well, we no longer-- in AV Foundation, we don't have the video preview output object, we've folded all of those configuration APIs into the video preview layer. So, they're just one stop for you to access to get those configured.

And the video data output and the audio data output can produce both uncompressed or compressed data, and if available on the hardware you're using, the video encoding will be hardware accelerated. And there's also an audio file output to write to an audio file like a CAF file and there's an output for writing a still image, for example, when you take the photograph.

Another thing to know about the output object is that whenever there's a delegate, object that you can install, you can also specify what dispatch queue your delegate callback will be called on. The device discovery APIs you can see here is very similar. In both cases, there's an API to get all devices, there's an API to get all the devices with particular media type. The default device with the media type, and if you already know a unique ID, you can get that device as well.

What's different with AV Foundation is a piece of final deprecation with QTKit in a 32-bit app, the old ancient QuickTime sequence grabber video digitizer components were grandfathered into the device input list. AV Foundation does not grandfather those old sequence grabber components. It only supports the modern DAL and HAL devices.

And AV Foundation provides that video preview layer like we discussed and the interfaces that used to be configured through the video preview output object and now ones you access through the preview layer. There's no direct analog to the QTKit capture view but you can take that layer and put inside a view just like you would with any other core animation layer.

So we've gone through a bunch of high level things. Let's dig down into some low level stuff, beginning with time. We, in media systems, are of the opinion that the right way to represent time in a media system is as a rational number. Floating point numbers will almost always introduce some kind of rounding error because the numbers never have-- never happen to be expressed-- exactly expressible as floating point numbers.

And when you add this up by the millions, they can lead up to measurable drift. So, we use rational numbers. And in order to support really long media files, you need to use a 64-bit time value and a 32-bit time scale. The time value is our name for the numerator and the time scale is our name for the denominator. Now, QuickTime had a 64-bit numerator, 32-bit denominator time object called the TimeRecord. But, in the very early 1990s, doing 64-bit math was really awkward.

And so, there were shortcuts where you'd pass a time value that was a just a 32-bit time value. And these shortcuts looked like they were good conveniences 'cause, you know, most movies were that short, that's fine. But unfortunately, they were also shortcuts that were taken inside QuickTime and that meant that you couldn't really take advantage of that full 64-bit size. QTKit has a QTTime which is a struct for the 64-bit numerator and 32-bit denominator used in many of its APIs. It's great, but it's still using QuickTime for playback and editing. And so, it was kind of hamstrung by that limitation in the pipeline.

But with AV Foundation just like QTKit, we have a struct called CMTime which has a 64-bit numerator, a 32-bit denominator, and a bunch of other cool stuff that you can look in the header file. And just like QTKit, there's a time range a CMTime range expressed as two times-- as two CMTimes, a start and a duration.

And there's also a CMTimeMapping which is expressed as two time ranges, a source and a target, and that's used in some editing APIs. But the really good news is there's no 32-bit shortcut which means we didn't use a 32-bit shortcut either and you can use that full 64-bit size.

The APIs, the objects that represent time when it's moving are very similar. In both cases, there's a clock object that represents an external source of moving time which is not under program control and then there's a timebase that you can set the rate and set the time and so forth. And the timebase's time comes from its master which could be a clock or another timebase. That's a model that we like and we've continued.

QuickTime, in some places QTKit, would use a-- non-opaque data structure called the SampleDescriptionHandle to describe compressed data. Originally, it was read and written directly from movie files and that was great when everything was big-endian and the file is big-endian, and the Mac is big-endian, woo-hoo, but then we ported to Intel processors and we had to endian flip it and we had to have rather complicated rules for how you endian flip it and some pieces stay big-endian in memory and that's kind of weird to deal with as well. So, we've learned that lesson and now we have a rather nice FormatDescription object which has a clean API. And we also had a rather nice SampleBuffer object for holding individual samples.

It retains the format description. It carries timing information. The data that you've references may-- it doesn't have to contiguous and it can even just be a promise it's going to be delivered later. And it has a way to attach key value pairs of supplemental information for other descriptive purposes.

If you need to encode or decode video frames directly, the AV Foundation offers the Video Toolbox APIs like the enhanced Image Compression Manager APIs that were introduced in QuickTime 7, these use core video pixel buffers for the uncompressed images. They use core media sample buffers for the compressed data and there is an additional interface for transferring an image from one pixel buffer to another pixel buffer.

If you are still using QuickTime's graphics importers and graphics exporters for still image support, bless you. I worked really hard on those, but it is time to wake up and smell-- smell the 64-bit native uniformly thread safe coffee delivered by the Image I/O Framework and it's CGImageSource and CGImageDestination APIs.

So there's the tour. There's the travel guide. I've actually introduced you to quite a large number of APIs but we've scratched the surface. There are more AV Foundation APIs that you can find in Headers and Documentation. In summary, we are deprecating the QuickTime C framework and the QTKit Objective-C framework in Maverick. This is a reminder for you to make the transition. The deprecation warnings will appear on your screen but you'll still be able to compile and your apps will still run. Everybody, your apps will still run.

So, AV Foundation is the stack of media frameworks that we are building and working on and focusing on in the media systems group. And we're building frameworks with significant architectural advances over QuickTime across the board. They're better. That said, we're not deprecating the QuickTime movie file format. It is still supported as our primary file format in AV Foundation and in QuickTime player. We're introducing QTMovieModernizer in Mavericks to help bring your media into AV Foundation supported formats. We have introduced a great new view integration class in AVPlayerView as part of AV Kit.

Finally, we know that developers have had a long and rich history developing with QuickTime over its last 22 years. And we know there are probably some of you who are using QuickTime in ways that we haven't yet anticipated in AV Foundation's family. If you can't figure out how to bring your QuickTime API app forward to AV Foundation, we want to hear from you. We're interested in your feedback.

You can come and see us this week in the lab. You can write up enhancement requests using Apple's bug reporting mechanism, and you can send email to John Geleynse, Apple's Director of Media Evangelism. His email address is on the next slide. But we are not finished with AV Foundation. We-- we'll make it better with every release and we're interested in your feedback.

There's the email address for John Geleynse. There's two resources online that I want to draw your attention to, one is the AV Foundation Programming Guide and the other is called Tech Note 2300. It's all about moving QuickTime and-- oh, it's all about moving QTKit code to AV Foundation. There's also an AV Foundation zone inside the Apple Developer Forums. You can get help from other AV Foundation developers, and sometimes if you're lucky, from AV Foundation engineers.

And it's also searchable. Sometimes the question that you're asking has already been asked. We have three other sessions from media systems and AV Foundation folks this week. There's the preparing and presenting media for accessibility session which we'll go into more detail about what it means honor a user's accessibility preferences.

There is, "What's new in camera capture?" Which will focus principally on new features in AV Foundation capture on iOS 7. And there's also very exciting session on advanced editing with AV Foundation which will introduce a new way of integrating your code directly into our playback pipeline to do fancy video effects and filters limited only by your imagination. That's very exciting feature to me. Thanks for coming up. Bye, bye.