Welcome to session 314. This is Transitioning to QuickTime X. My name is Tim Monroe and I will be joined shortly by Kevin Calhoun. And we're going to talk to you today about QuickTime X and some of the other changes that we have made in QTKit for Snow Leopard.

So I'm just going to make a wild guess that most of you went to the graphics and media state of the unions. So you have a good understanding already of what QuickTime X is and where it fits in to the media architecture on Mac OS. You already know that it's a new technology for displaying time-based media.

It is one of the technologies that has actually migrated back to the desktop form our devices such as iPhone, Apple TV, iPod. It was implemented independent of QuickTime 7. And you'll see today that it provides some very interesting and significant improvements over the current media architecture. And you also learn that to get the playback advantages of QuickTime X, you must be using QTKit. So what are we going to talk about here today? We're going to spend most of our time talking about QuickTime X.

So we'll show how to open media so that you can play it back with QuickTime X, when it makes sense for you to do so, and also what you give up, what restrictions apply when you open a movie on the QuickTime X path. We'll also talk about some issues that are unrelated to QuickTime X and the one that is worth mentioning is asynchronous loading.

We've made some very interesting improvements in QTKit for opening movies in such a way that you don't block your main thread. And again, we have some very nice demos that will show you why you'd be interested in adopting that technology. Now there are some other API changes in QTKit that I don't want to spend a lot of time talking about. So I'll mention them upfront here.

One thing it's possible to do now is to take a single QTMovie object and display it in more than one view or more than one layer or some layers and some views. So the tight connection between a QTMovie object and its rendering destination has been broken you could now have a single QTMovie object displayed in more than one view.

Now because of that, it no longer makes sense to ask for the current size of a movie. Why? Because the single movie could be in different size views. So we have deprecated the attribute, QTMovieCurrentSizeAttribute. What you want to move to is either querying the size of your rendering destination, the view, or the layer, or you want to ask for the natural size of the movie. That is the size at which the movie was authored. So if you're using current size attribute, move to natural size attribute or ask the view what it size is. Now because of that change, we have also deprecated the QTMovieSizeDidChangeNotification.

And we have replaced that with a new notification, QTMovieNaturalSizeDidChangeNotification. Now, you may be wondering how could be the natural size change. And one way this could happen is if you have a streaming movie coming in through the net. And halfway through it, a new track appears that is larger than existing tracks or smaller or in anyway different. And if that happens, you will get this QTMovieNaturalSizeDidChangeNotification and you could react to it accordingly. So let's talk about QuickTime X. This is what the documentation says about QTMovie prior to Snow Leopard. A QTMovie object represents both a QuickTime movie and a movie controller.

Now you already know that this is no longer true because you already know that you can play back movies using QTKit and not accessing the existing QuickTime technologies. That is to say you'll be accessing QuickTime X. So if you look at the new documentation for Snow Leopard, we say something more general, namely a QTMovie object represents a playable collection of media data. This is the pre-Snow Leopard architecture. You've seen these kind of diagrams before. QTKit as we just saw, sits right on top of QuickTime or QuickTime 7 as we now say. QuickTime 7 consists of a whole bunch of APIs that are divided into tool boxes and components.

So you have the Movie Toolbox, you have Movie Controller components. A little bit lower down, you have Media Handlers that handle the media in a particular track, you have some Data Handlers who's responsibility is to load data from disk or from the net or wherever. And those QuickTime services, themselves that upon lower level system services such as the Sound Manager or more recently we have moved to Core Audio. It may sit on QuickDraw. Or again, more recently we're doing things with Core Video. And originally we use a lot of Carbon.

And now we have moved to Core Foundation. But they're still in the QuickTime 7 stack, a mixture of these lower level technologies. Now when we move to QuickTime X, we have QTKit seating either on top of QuickTime 7 or on top of QuickTime X. Now we're not telling you much more about QuickTime X other than it is a new set of media services and that those media services are sitting solely on top of this more modern foundation such as Core Foundation, Core Video, Core Audio, Core Animation.

So it's a completely modern stack underneath QuickTime X. So, new set of media services, it was designed for extremely efficient playback which I think you can understand by remembering that it's developed for the iPhone, Apple TV, and iPod. It is aggressively multithreaded. So it will try to fork off threads whenever it sees that there is something that deserves happening on a background thread It is completely 64-bit native when running in 64-bits.stydl7 It is based again, on core Foundation and it has ColorSync matching throughout the stack.

Now the current set of formats that it supports are what we call modern linear media formats, that's stuff like H264 video, AAC audio. We're not going to give you an exact list because of course that list will change over time as QuickTime X learns how to handle more and more media formats.

And there's a very, very, very simple API for you to access QuickTime X in Snow Leopard that I'll talk about in just a minute. Now there are also some other benefits. And again, you got a taste of this in the graphics and media state of the union where there were four components of QuickTime X.

And QTKit actually automatically takes advantage of some of those components. So you don't have to do anything for instance if your using the existing QTKit capture APIs. We now use QuickTime X services to process movies that are captured. And we also use QuickTime X to do the color tagging of those captured movies. Similarly, if you use existing QTKit APIs to transcode a file, to export it in a new format, you will automatically take advantage of new QuickTime X technology if you're exporting to the various device formats that it supports.

So, how do you get to use QuickTime X? Well, you have to ask very nicely when you open a file. So let's talk for a second about opening a movie file. Prior to Snow Leopard, the designated initializer of the QTMovie class was this long winded method initWithQuickTimeMovie:disposeWhenDone:error:.

Now again, if you realize that we have sort of bifurcated QTKit so that it stands upon QuickTime 7 or QuickTime X, you know that this is now a bad choice as a designated initializer. Why? For the simple reason that on the QuickTime X path, you don't have a QuickTime movie.

So there is not one of these capital M movies that you can pass into this designated initializer. This was actually already a problem in Leopard when you are running in 64-bits because there also the QuickTime framework is not available in 64-bits.stydlG So there you couldn't have had a capital M movie to pass into your designated initializer. So there was a little bit of a disconnect there already in Leopard. So in Snow Leopard, we now we have a new designated initializer for QTMovie and it is initWithAttributes:error:. That's an existing method which we have now promoted to be the designated initializer.

And here I have the instance method and the class method listed. Now if you have code that already subclasses QTMovie by calling initWithMmovie:error:, initWithMovie:disposeWhenDone:error:, don't worry about it. We have taken care of it so that your initializer gets called at the proper time. But if you are writing new code and you want to subclass QTMovie, you should move over to subclassing initWithAttributes:error:. So what's the idea with initWithAttributes:error:?

It's very simple. You pass in a dictionary of key value pairs which specify attributes that you want this movie to have. Now there are three general classes of attributes that you can put in to this dictionary. The fist class I called data locator attributes. You got to tell us where the movie data is. So one of these, is essential.

Typically, you'll use the first two, QTMovieFileNameAttribute or QTMovieURLAttribute. There are also some other data locator attributes for more specialized purposes. There are also movie configuration attributes. So, if you want the movie to open, say set to be looping, you could put in a key value pair that says, QTMovieLoopsAttribute, YES.

You could also open it with a nonstandard volume if you wanted to. Or you could make it editable at the time it's opened. Or you could assign a delegate to it. Now, what kinds of things can you put in as movie configuration attributes? Basically, you can put in any settable attribute in the QTMovie class. Now sometimes people send us code and I see that they put in attributes like QTMovieDurationAttribute.

And this has always puzzled me. For the simple reason that duration is not a settable attribute of a QTMovie object. It's like you went down to your Toyota dealer and sell I want a Prius with a 120 inch wheel ba se. Well they don't make them. The factory makes them at 106 inches. So that's not a settable configurable option when you're buying a car. Similarly when you're opening a QTMovie, you cannot set the duration to some nonstandard value.

Well how do you know this? Partly, this is our fault because we haven't been good about telling you which attributes are settable and which are not. So we've taken some trouble and we've gone through and we've header-docked QTMovie.h. And in particular we've made it very clear whether a given attribute can be read and written or read but not written, and so forth. So here for instance, if you look in QTMovie.h and you look at the QTMovieDurationAttribute, it says very clearly, this attribute can be read but it cannot be written.

So that's a clue to you that you don't want to put it into the dictionary that you pass to initWithAttributes. The third class of attributes that you could put in to this dictionary, I call special initialization attributes. And I won't discus these in detail because we have all that great header dock which now explains to you what these do. The one I want to highlight is the last one which I will come back and talk about later namely, the OpenAsyncOKAttribute. Now, how do you get QuickTime X playback capability?

As I said, you ask us nicely and you ask nicely by passing in a new special initialization attribute and this is it, QTMovieOpenForPlaybackAttribute. So you say, and the value of course, you would pass them would be YES. So you'll say I want this MovieOpenedForPlayback. And that is a clue to us, as QTKit, that you want to go down the QuickTime X path. So what does that look like in code? Very simple, you setup an NSDictionary with the key value pairs that you want. And in this case, we have specified open for playback, YES, and this is a clue that we want the QuickTime X path.

Now, again you heard yesterday that not all media files can be handled by QuickTime X. So what happens when you ask us nicely to play it with QuickTime X and we can't, we silently fall back to the existing QuickTime 7 path. You cannot tell whether we are taking the QuickTime X path or the QuickTime 7 path. It's not something really that you need to know.

Because you have requested to open a movie for playback, you are therefore restricted to certain APIs and certain attributes that apply to playback only. Again, how do you know which APIs those are? We signal this in the header dock comments and QTMovie.h and in particular you get something like this. This method cannot be called when the movie has been initialized with QTMovieOpenForPlaybackAttribute set to YES.

And in this case, this is the QuickTime movie method. So if you say, "well what's the QuickTime movie associated with this QTMovie?" If you open this for playback, you might be going down the QuickTime X path and so there is no QuickTime movie associated with it. Similarly, some attributes which you could otherwise have set are not settable when you open them on the OpenForPlayback. And a good example is QTMovieEditableAttribute.

And again, we mark that here by saying, "This attribute can be read and written but it can be read and not written when the movie has bee initialized with QTMovieOpenForPlayback set to YES." So when you request playback, you are restricted to certain methods and you're restricted to not being able to set certain attributes. So now, let's get a demo of using QuickTime X to open and playback movies.

Alright Tim, we've got an app here, did you say it was beautiful? Let's just say, it's conceptually beautiful. And the concept is to demonstrate the efficiency of playing back media with QuickTime X. This application is capable of opening initializing instances of QTMovie with or without QTMovieOpenForPlaybackAttributes set to YES.

So let me open a media resource without setting it to YES. So I will get in this case, the QuickTime 7 behavior. I've got 720P content here and I've also got activity monitor running. [Background music] And when I play this movie which is not only beautiful conceptually, but beautiful in itself, you can see at this particular video and audio requires, well I don't know, between 40 and 50 percent of the CPU when it is being decoded and played. Note also, this 64-bit application requires the services of a 32-bit daemon ap2plication, the QuickTime kit-- the QTKit server application, I'm sorry, process in order to perform playback on the QuickTime 7 path.

Let's see what the same video looks like and how much CPU it requires when I, this time initialize the QTMovie-- setting QTMovieOpenForPlayback to YES. So open the same media resource, different instance of QTMovie, this one open for playback. Also one of the things that I want to note is that when I open the movie the movie for playback, this 64-bit application no longer requires the services of QTKit server, so it goes away. [Background music] And I can play the very same media resource on the QuickTime X path, who's doing the calculation with your iPhone even now, to determine the improvement in the number of CPU cycles we require? So that's it, playback time. We have a more efficient media stack.

It's available to you and you're applications when you opt in, its better. But it's not only better and more efficient at playback time. It's also more efficient at other times as well, for reasons that we're going to be exploring in quite a bit more depth later in this session. Other periods at which QuickTime 7 has required to do some work are not required for QuickTime X. Let me give you an example of that.

In this case, I'm going to open an MP3 file, quite a long one actually, as you can see it's about half a gigabyte of music. And there's work going on. I am not playing the content. I merely just opened it. But something is going on. You can see the controller bar filling in saying, "Now, I've got more media available." You can see an activity monitor, some amount of CPU is required even though I haven't begun a media operation on it yet.

And of course, that amount of CPU will increase [background music] when I actually play it. Well what work is going on? Is it necessary? I'll say more in a minute. All I want to mention now is that if I set QTMovieOpenForPlaybackAttribute to YES, and the media resource can be opened by QuickTime X, that kind of work is not necessary.

You can see first of all that the app that I'm running, the QuickTime X playback demos, requiring no CPU cycles since I'm not actually doing anything and you also see QTKit server has quit. Only when I play back [background music] are any cycles actually required and fewer. So there you go, quick idea of the increased efficiency of the new playback step. Tim.

Always put the seat down when you're done

So there, certain movies open much faster with less CPU. And virtually all movies, when they take the QuickTime X path take less CPU to play back. So that should be enticing enough as a reason to be interested in QuickTime X.

So let's talk about asynchronous loading now. We have documented fairly clearly that when you open a QTMovie object you want to do it on the main thread. This is really a limitation that derives from QuickTime but it's something that we've documented and talked about here at WWDC. Now it turns out that lots of things need to go on to open some movies and you don't want then blocking the main thread.

Well QuickTime already has some capabilities of being able to background some of that work. For instance, some data handlers are smart enough to fork off a thread and load their data on a background thread. So even without the new capabilities we've added in Snow Leopard, there is some asynchronous work being done in certain circumstances.

What we have done QuickTime or in Snow Leopard is add new services that allow us to be even more aggressive at moving things off of the main thread onto background threads. So it's a new service provided by QTKit in Snow Leopard. And basically, it is your way of telling us, do everything possible to avoid blocking the main thread. So that means that we should try to load movie data on a background thread if possible.

And if that isn't possible, I want you to fail. I want you to return an error letting me know as the developer that my request was not honored. How do we do this? Again, there's a special initialization attribute that you put in to the dictionary when you call initWithAttributes:error. And this particular attribute is called QTMovieOpenAsyncRequiredAttribute. So you're saying, "I require you to open this asynchronously." And again, the code that you would write in order to use that is identical to what we showed before. It's just a different attribute that we're passing in.

Now when you open a movie asynchronously, you will get back a non-nil QTMovie instance, virtually immediately. However, there could be a lot of work that needs to happen in order to load that movie to the point that you can ask about its properties or start playing it back or export it or whatever you want to do with it. So you must monitor what we call movie load states.

These are movie load states. They're just integers that indicate that a certain level in the loading of movie data has occurred. And the very top one is interesting, it says, LoadStateError. So that way you know something went wrong in loading the movie. It couldn't be loaded. You should just toss the movie away.

And the other ones we'll talk about in more detail later, but they indicate various levels in the loading of movie data. How do you monitor a movie load state? The absolute easiest way is to register for the QTMovieLoadStateDidChangeNotification. And in this case, I'm using normal NSNotificationCenter method to install a notification handler.

In this case, it's called "LoadStateChanged." So when QTKit signals my application at the LoadStateChanged, it will call my LoadStateChanged method. And in this case, I have a very simple LoadStateChanged method. All I'm doing is grabbing the movie from the notification object and passing that movie to another method that my application has defined.

And the reason I'm dong this will be clear in just a second. And here's what the real guts of my monitoring movie load states might look like. At the very top, I call the attribute for key to find the QTMovieLoadstateAttribute to sort of find out what the current load state is.

And then I just want to go through the various possibilities that are of interest to my application. So here, I'm handling the LoadStateError, actually I just have some dots there. We'll fill that in a second. And the second clause is more interesting. If I get to LoadStateLoaded, I know that it's now safe to query movie properties and to do things like attach the movie to a view so that it can be displayed and played back.

Now it turns out to that when you call initWithAttributes:error:, it's possible that you never receive a LoadStatechanged notification. That's because the movie could immediately go to the top load state that it's ever going to achieve. So immediately after I call initWithAttributes:error:, I probably want to make a call to my own method with that movie just so that any work is done when I've achieved a certain load state and then I can still register to listen for the notification if I want to. So what about Asynchronous Loading? Not all media files can be loaded asynchronously. There could be media handlers involved in the movie that are not marked as thread-safe. And they cannot therefore, be used on a background thread.

Now you can determine if async loading has failed. And if it fails, what do you do? Well it really depends on the needs of your application. For instance, if you were just opening the movie to grab a frame of it, to make a thumbnail image, it may not be that important to you that you want to tie up your main thread to do that.

So you may just put in some icon that you have stored away. Or it really might be important for you to open that movie. In which case, what you want to do is retry initWithAttributes:error but without the async required flag set. So when async loading fails, you will receive this QTMovieLoadStateError as the load state in your notification handler. And what do you do with that? Well, we've added a new API in Snow Leopard, the QTMovieLoadStateErrorAttribute, that allows you to ask the movie what went wrong.

Previously, prior to Snow Leopard, you couldn't do that. It was just a mystery why it failed. Now you can actually ask the movie what caused you to fail. And then you can look at that NSError, get the domain, get the code, and perhaps react accordingly. Or again, you may just want to retry it without setting the open async required to YES. So let's have a demo of this. And again, I'll have Kevin come back up and demonstrate why async loading is interesting.

This demo, I claim is the most conceptually beautiful demo of all. Because it's main feature is a call to sleep for 5 seconds. I'm doing that in this demo in order to simulate very sluggish loading of a movie. For example, if you have a media resource in the file system but it resides on a file server, the access to which is very slow-- it takes a while just to get a few bytes-- those are the cases on the QuickTime 7 path when loading can be so sluggish that you risk blocking the main thread.

And it's specifically for those cases that you want to use the attribute that Tim has just been describing QTMovieOpenAsyncRequiredAttribute, set that to YES that cause all the work that needs to be done even if it's extremely sluggish, they pushed off into the background and allows your UI to main responses. So let us demonstrate what a non-responsive UI looks like. I'll click the button and I can't pull down my menus and the beach ball spins, this sends a message to our server in Cupertino to register beach ball spinning.

No, it doesn't do that. Imagine that it did though. If you are a programmer, you want to avoid that in your application. How do you avoid the beach ball spinning? Well you set async open required when you initialize the QTMovie. All the work will be pushed off to the background and even if that work comprises a call to sleep for 5 seconds, the UI will be able to responsive, the user pulled down menus, progress indicators will run, and then after 5 seconds you will still get the movie to load. So that's a good thing. Remember though, when you use OpenAsyncRequired, for some legacy cases, it may fail for reasons that we'll go in to a more depth later. So have a fallback ready, thanks.

Thank you. He learns. Alright, so we've said that you must monitor load states when you open a movie. There is actually one case, to be honest, where you don't really need to. And that is when you don't care that the movie is loaded asynchronously. And a good example of this would be if you have a command line tool, where there is no UI.

So there's no UI to block, you just don't care. What you really want is a movie in its fullest load state before you go on and work on it. And this is what you would do. You would set the OpenAsyncOKAttribute to NO. Telling QTKit, "No, it isn't OK to load this movie asynchronously in any way.

I just want to wait until it's loaded completely." So now let's start of compare and contrast this two similar sounding attributes. To say OpenAsyncOK, that says "I want you to avoid blocking the main thread, the thread I'm initializing this movie on." By default, the value is YES. That's the default behavior of QTMovie movie. Now, work may or may not be actually done in a background thread, if the media is not thread-safe, then it will have to happen on the foreground-- the main thread.

And it will not result in a failure to load the movie if backgrounding is not possible. It will just tie up your main thread more. The new attribute that we're adding in Snow Leopard will avoid blocking the initialization thread. It's NO by default. You have to opt in to this new behavior. Work will be done in a background thread if it all possible.

And if it's not possible, we will fail and return the error to you in the ways that I've already described. So now, I'd like to have Kevin come back up and actually drill down into some of the suffix a little bit more to perhaps clear up some issues.

Or to create issues if I may. I didn't know what to call this part of a talk.

I'm going to drill down on a lot of things that Tim said, so I just use the word usage notes but disregard that.

Let's instead use this fancier title, "A taxonomy of Media Operations", that's the way to start your talk. This in fact is not a complete taxonomy of media operations. It's really just those media operations that we support in QTKit, and make available to you as QTKit clients.

Obviously, you can playback media resources with QTKit. We've talked mostly about that. You can export media resources. You can re-encode their audio and video and write them to container formats using the export features. You can edit them in place in QTKit, cut, copy, and paste editing, insertions, deletions all supported. You can also capture media from external devices, cameras and what have you. Now, Tim's already mentioned that some of these operations are enhanced using QuickTime X technology in Snow Leopard.

Let me highlight those. Playback is the one we've talked about most. For enhanced playback in Snow Leopard, you want to opt in via QTMovieOpenAsyncRequiredAttribute as we've already discussed. For enhanced export functionality in QTKit, there is nothing that your application needs to do beyond what it is already doing. If you are calling QTMovieWriteToFile:attributes and the attributes dictionary that you provide specifies can export operation to an Apple device format, that export will be enhanced with QuickTime X technology and the primary benefit there is greater efficiency, but there are others as well.

The media will be as often as possible and virtually universally for these devices color tag appropriately so that your color fidelity and accuracy regardless of the destination will be the best possible. Tim also mentioned that the capture functionality of QTKit known cleverly as QTKitCapture has also been enhanced with QuickTime X. There is nothing that you need to do in your application if you're capturing via QTKit.

To take advantage of this, it just comes for free on Snow Leopard. One of the advantages is that for 64-bit applications that you're deploying on Snow Leopard, you will get 64-bit native capture services. Also, as often possible, the media that you capture from devices will be appropriately tagged with color profiling information, so again, you get the best fidelity and accuracy we can offer. The editing box is not enhanced with QuickTime X technology in Snow Leopard. So why did we do it the way we do it?

And why did we present these enhancements to you in the form that we did? Well, in order to explain that, let me give you a little background. Let me give you a little conceptual history. Going back to the beginning of QuickTime, what does QuickTime actually do with media resources?

You may know that all media resources would open with QuickTime from 1.0 up through QuickTime 7 are represented as a movie, a uniform representation of all media resources. You might be aware of some of the constituent parts of a movie, the tracks, the media of the tracks, the sample descriptions, and the sample tables of the media.

For QuickTime 7 and the earlier releases, to operate on media resources regardless of the operation that it performs, any of those that I mentioned on my previous slide, this movie representation must be created. It can either be loaded from a file if it's a QuickTime movie file or it must be created on the fly when importing media from a different format such as an MP3 file.

And in fact, when I was showing you earlier the demo of opening an MP3 file with QuickTime 7, all that work that was going on before any operation was performed was the work required to create the movie representation of the MP3 file. MP3 was chosen for that demo because in fact that container format contains just about the least information possible in order for us to create a movie representation from it.

There's no information there about, well there's seldom information there about the number, the extent of the audio packets. It's usually just a stream of audio packets in an MP3 file. So we've gotta do some parsing to create that representation. That work is what we do implicitly in QuickTime 7 if you call the movie toolbox API, NewMovieFromFile or NewMovieFromProperties or the QTKit method, QTMovie initWithFile, QTMovie initWithURL. By default, by creating that movie representation, we are implicitly preparing that media resource for all of the operations that you can perform on it. You can play it back.

You can edit it in place. You can export it because we have this representation with all of the powerful things that we can do with it implicitly. However as you've seen, it's expensive in some cases to prepare for all that. So all these years when you've been opening MP3 files with QuickTime, we have been laboriously preparing for you to export those MP3 files to other formats. Then you turn around and you just play them.

And we're left there thinking why did we do all that work? We could have waited to parse those audio packets until it's actually time to play them. Well the good news is, with QuickTime X, we have altered the design, we've introduced into the model the declaration of intent.

What purpose are you opening this media resource for? If you, the client, tell us the framework why you're opening this thing, we can tailor the work that we do to prepare it for the purpose that you declare. And that's why, as Tim explained to you earlier, QTMovieOpenForPlaybackAttribute. You're declaring your intent.

You're leaving us off the hook for preparing that media resource for operations beyond the scope of playback. Now for backward compatibility, if you say nothing about your intention, about the purpose for which you're initializing a QTMovie instance, we will do what we've done in the past and implicitly prepare it for everything that we can do.

All the things that I mentioned at the top of this portion of the talk. But regardless, whether you declare your intention or not, recall that time-based media has fundamental operation that's progressive. It's incremental. These pieces of time-based media are authored to be consumed overtime. The video frame we're displaying now is going to be replaced soon.

This is not the type of object as would be the case with many other objects in Cocoa for which we internalize all the data at initialization time. Operate on the data in memory and then if something's changed, write everything out, externalize it when you're done. That's not what time-based media does. We read incrementally.

We load incrementally, and that's why as Tim described earlier, these load states are fundamental to the operation of time-based media. And why? We introduced them with QTKit when initially shipped on Tiger, and we're still relying on the same loading model now. So, for QuickTime X, something new at initialization time. You declare your intent.

Subsequently, the incremental operation of the QTMovie instance is the same that you remember from before. The load states are exactly the same as we've presented to you on Tiger and Leopard. So I don't need to review all these since Tim has done a good job of explaining what these are. QTMovieLoadStatePlaythroughOK deserves special mention because that particular state is unique for 2 reasons. One, you can enter and also regress from that state.

What it essentially means is that when we reach QTMovieLoadStatePlaythroughOK that means that we're projecting through our finely tuned algorithms that you can start playback and have it proceed through the duration of the media resource without having to wait, having to pause for more data to arrive before proceeding. However, if network conditions change or it's just a bad day, there-- it might happen that data starts loading more slowly than it had been up to that point and the QTMovie instance may regress to QTMovieLoadStatePlayable.

If that happens and if the playback has to stall because more media needs to load before we can proceed, that will just happen automatically and playback will resume automatically at the same rate without any intervention on your part. You can derail that process if you like by setting the rate of that instance of QTMovie to 0 and then automatic playback won't resume. But the behavior for resumption is automatic, if you don't otherwise change things.

QTMovieLoadStateComplete also has very specific, very special meaning that's directly relevant to our QuickTime X story, so we'll get back to that in a minute. But suppose you're an application that just wants to play media. Tim talked about load states, and if that didn't go on long enough, Kevin talked about load states. Why do I need to know these load states, I just fill out tax forms.

I was going to say, I just showed how to use the track pad but I don't know. Anyway, there is a method on QTMovie that you can use that doesn't require that you monitor all these progress. You can get QTKit to do that monitoring for you. The method is called autoplay. So you would set up your code like this if you just want to play when it-- the movie reaches the playthrough OK state.

Initialize the movie, you can immediately put it into a view regardless of its state if you like and then say autoplay. And the result of that is that the movie will play as soon as it's ready and that will work on the QuickTime 7 path or the QuickTime X path equally well.

I should note, however, that if you do choose to put a movie into a view before it has reached the loaded state, that QTMovie view will draw nothing. So if you want to avoid big holes in your UI such as what I demonstrated when I first came up here with my first demo application, you can wait until the movie reaches the loaded state before showing your view, or alternately before setting the movie on to the view.

Alright, QTMovieLoadStateComplete, it's special. What does it mean? And what happens to it when I opt in for QuickTime X playback by setting QTMovieOpenForPlaybackAttribute.

Let's say what-- let's just describe what this thing is. Essentially, when a movie reaches QTMovieLoadStateComplete, we have what I described earlier as the uniform representation that QuickTime 7 uses for all media resources, the movie. The MP3 file that I opened earlier when that, the controller bar filled in all the way to the right and we reached load state complete, well we've got the compete movie that will locate all of the audio packets within that half a gigabyte of MP3 data. So at that stage, it's safe for your application to call movie format representation. The method on QTMovie that allows you to fetch the externalizable form of this uniform movie representation, then you can write back to storage.

But QTMovieLoadStateCompete also implies that all of the media data required by that movie is either immediately available because it's in the file system or some other immediately accessible storage. Or it has been transferred via a network protocol and completely cached, so it's all available at that point as well. So since all the media data is completely available as well, at that point you can call method such as writeToFile in order to perform an export operation.

All the media data is there for you to re-encode or whatever you're trying to do via your right to file operation. Now, note that both the use of the movie representation and the caching behavior are unique to QuickTime 7. In other words, QTMovieLoadStateComplete implies behavior that is very QuickTime 7 like and does not apply to QuickTime X for which we don't always have a movie representation and which does not sign a contract that requires it to cache all of the media data and retain it in it's cache throughout the duration of that instance of QTMovie. So, what happens to QTMovieLoadStateComplete when you say QTMovieOpenForPlaybackAttribute is YES?

Essentially what we're going to do is not signal that we've reached that state And there's no need for us to signal it because you've already declared my intention is only to play this instance of QTMovie, we will tell you when it's playable, we will tell you when playthrough is OK. We'll even do that work for you if you say autoplay. There's no need for it to enter any state beyond that because your intention is only playback. Well, that's a restriction, that's a change of behavior.

I opted in to this new-- this new feature, QuickTime X playback, and something is unavailable, QTMovieLoadStateComplete. I would be remiss if I didn't list all of those things that are not available when you opt in for this feature that enables the more efficient playback of QuickTime X. So let's go through all of the usage restrictions. When you say playback, we help you to enforce the contract that you have signed. The instance of QTMovie that's intended for playback is allowed only to play back.

No editing will be available, nor is it required since you declared your intention of playback. Saving is not available, write to file. Archiving is not available. Instead of archiving in the QTMovie instance, we recommend that you archive either the dictionary attributes that you initialize a QTMovie with or selected attributes of the QTMovie instance at the time of archiving that you want to have on archive when you want to create another QTMovie instance from the same media resource. Export is not available when you opt in for playback, and there is no access to the QuickTime 7 primitive such as capital M Movie or capital T Track.

How do we enforce these restrictions? Very simply, if you attempt to call any of the methods that are not allowed, when you opt in for playback or if you try to access the attributes that are not available, in that case, we raise a very particular NSException, with a name that's so long, you can't miss it. QTDisallowedForInitializationPurposeException, you declared your intent it's for playback.

Then you try to do something beyond the scope of playback, we raise the exception. And we do that in order to help you to find those places in your application where your usage might move beyond the contract that you've entered. So in order to demonstrate a case like this, I took an application, a sample application that we have on the Apple developer connection site called QTKit Player. And this is an application that we've been providing since 2004 that demonstrates basic usage of QTKit for playing back media resources. So I've started with this and only made some very small changes here.

For one thing, I took Tim's recommendation from earlier in this talk. I actually heard this talk sooner than you did, so I was able to apply what Tim said. And I have moved from using movieWithFile which is what the source, if you download it from the web, will use and I'm using movieWithAttributes instead.

And in the attributes dictionary, I am setting the path of the file that I want open using the key QTMovieFilenameAttribute and I'm also setting open for playback to be YES, because I want my playback to be enhanced with QuickTime X. There's one other thing I did to this application as well. I took the advice of an O'Reilly article that demonstrates the methodology for catching uncaught NSExceptions.

And why did I do this? Because I want to break in the debugger whenever anything is attempted by this application that goes beyond the scope of the APIs that are available when you say that the movies open for playback. So I wrote this little, following the O'Reilly advice, I wrote a little function called catch some exceptions and I'm looking for my lengthily named QTDisallowedInitializationPurposeException because I want to know why that happened.

And I'm also going to log to the console the reason that this happened because we provide, when we raised these exceptions, some very particular information that will be useful to you for debugging why did this exception get raised. So, let me build and debug this application, QTKit player with these modest improvements, and let me attempt to open a media resource for playback in that application. Well, I hit my breakpoint. I had the QTMovieDisallowedForInitializationPurposeException. Let me show the debugger and the console.

I'll clear the console. Let me step over this line that will log what happen and it says to me, well, I tried to query the QTMovie attribute, QTMovieCurrentSizeAttribute which is not allowed when I set QTMovieOpenForPlayback to YES. Tim mentioned earlier that this particular attribute, QTMovieCurrentSizeAttribute is deprecated in Snow Leopard, and that's true. If your application is using it, it will still function as defined.

But when you opt in via the QTMovieOpenForPlaybackAttribute for the new playback set, we don't allow this attribute at all. We're going to encourage you as you make the modifications to your application to adopt the different attribute that we're recommending that you use, QTMovieNaturalSizeAttribute. So, I will look for preview of coming attraction, QTMovieCurrentSizeAttribute, and I will simply change that to QTMovieNaturalSizeAttribute where it occurs, alright.

Is it possible to edit source code live on stage in San Francisco? I will rebuild and relaunch the application now with that modification. I'll try to open a media resource again, see what trouble I run into, and I hit the breakpoint another time. Let me display the console and the debugger, step over that line that logs the reason to the console. This time it says "oh, I tried to set the editable attribute." Well the person writing this application actually was very ambitious.

He not only wanted to display the use-- to provide an exemplar, the use of QTKit for playback, he moved beyond it and added some editing features as well. He wanted to set the QTMovies to be editable. Well, that's not allowed when I say QTMovieOpenForPlayback is yes. So I'll look for where that occurs, and just for the purpose of demos, I will simply comment that out, build a debug again, relaunch the application and with those two changes I'm now able to open media resources [applause] and take advantage of QuickTime X.

Now I realize that hitting breakpoints in debugger is not the most interesting demo in the world. But since you have code, you have code that might be as long as 5 years old now. That might have been worked on by other developers in the past. You might be interested in the methodology like this for finding those places in your code that you might need to modify as you opt in for QuickTime X.

So, let's summarize your transition. If you have code like that and you want to take advantage of QuickTime X on Snow Leopard, what do you do? Well first thing, tell us what you want to do and we'll help you help us to help you.

Tell us that you want to open something else for something for playback and we will tell you the work we do for your intention. It will be more efficient in a lot of cases. And you do that by means of the attribute that Tim described, QTMovieOpenForPlaybackAttribute.

If the media resource that you're opening is openable by QuickTime X, you will get QuickTime X in support of that instance of QTMovie for playback. Then, another thing that you must do before you ship your application that you're modifying for this purpose is to uncover any possible usage in that app that goes beyond playback, that enters into this restricted areas when you declare open for playback.

You can find those by reviewing the documentation, the Snow Leopard documentation that we provided to you along with the version of Snow Leopard you have calls out which methods and attributes are restricted when you open for playback. As Tim showed you earlier, the QTKit headers, QTMovie.h and the others will also call out the restrictions.

So take a look at those, you can watch where the-- this particular exception which gives you some diagnostic information when it's raised about what it is the app was trying to do when this exception was raised. And by the way, one of my colleagues asked me for the demo, why didn't I just break on Objective-C exception throw? Well, I didn't think of it.

No, actually what I wanted to do is to demonstrate a methodology that will allow you to run your own custom code when this particular exception is raised so that you can log a bug automatically or send yourself emails, and you better look at this particular part of the app.

But of course, just breaking on that other thing will work too. Now, if you happen to uncover any usage of this sort that goes beyond playback, you have a choice to make. You can decide to modify your application not to require the additional behavior that goes beyond playback.

That instance of QTMovie, well it's fine if it only plays back whatever it happened to be doing that goes beyond playback or perhaps that feature isn't actually needed, or perhaps what I can do in my application is to create a different instance of QTMovie from the same media resource and perform that nonplayback operation on that other instance, your choice. However, you may have an application which simply requires a broader set of features then just playback for an instance of QTMovie.

In that case, you simply wouldn't declare that that instance is open for playback and you would enjoy the same support that we've given you in prior releases of Mac OS X. Now, we've told you some of the benefits, now we've outlined the entire cost, work that you will need to do to transition your application. Let's close this whole section of the talk with additional benefits.

What do you get when you do this work to opt in using QTMovieOpenForPlaybackAttribute? Well, to review some of the things Tim mentioned earlier, the QuickTime X services are 64-bit native. So when you deploy your 64-bit application that plays media you want 64-bit native playback services, that's how you get them in QTKit file, this attribute.

Further, QuickTime X takes advantage of multicore so you want things to make, take advantage of the cores that are available on the hardware. That happens too when you opt in. Color fidelity is improved when you opt in for QuickTime X. More details about color management on Snow Leopard in general are in Session 301 which occurs on Friday. The particulars of color management for video will be covered there.

So go to that session for more details about color fidelity. You will also get, in your application, the services of the hardware code that Tim Bens [phonetic] mentioned yesterday in the Graphics and Media State of the Union. You want hardware accelerated H.264 decode, tell me how to get it, we already have. You opt in using QTKit via QTMovieOpenForPlayback. Now some notes about the hardware decoder.

This is available in various recent models of hardware that Apple has shipped. The hardware decoder has a limited number of sessions, video session that can support simultaneously. The number is not always one but it's small. If you want to guarantee or if you want to improve your chances of getting the services of the hardware decoder for the H.264 video content that you are playing your applications, you can use the QTMovie methods setActive.

If you set a movie to be inactive via setActive: NO what you've essentially done is to say, I don't need this to be prepared for rendering right now, I wanted to initialize it because it will eventually be used by my document or whatever it is that you're doing.

But I don't need to render it now, it's not on screen, it's not on the current slide, for example, in keynote, I'll set it to be active later and only when I need to render it will I do that by having just a small number of QTMovie instances active at a time, only those that you happen to be rendering, you increase the chances of getting the services of the hardware decoder.

Also, you may have gone to the Graphics and Media State of the Union and seen some iPhone apps for iPhone OS 3.0 that are displaying HTTP Live Streams. They look great and this whole story for setting up these streams is really good. More details about that in Session 313 and that's tomorrow morning.

But suppose you have a desktop application and you want it to be able to play HTTP Live Streams, that's available to you also in Snow Leopard via the same means that we've already described. You present the URL for one of these HTTP Live Streams to QTMovie in the attributes dictionary.

You say QTMovie URL attribute, and you give it the URL for an HTTP Live Stream, and you also say QTMovieOpenForPlayback attribute is YES and the Live Stream will play in your application. Alright. So that goes over in detail enhancements of QTKit in Snow Leopard. But suppose your application is already a media-rich application and you're performing a number of operations, for example you're performing exports on background threads, you're doing a lot of stuff, and you want to know how these new features and these new recommendations pertain to you.

Well, the answer is that all of the existing recommendations for use of QTKit in rich-media applications that perform operations such as export on background threads, those are still pertinent. Everything that we've told you still applies and it's not changed in anyway by these new features. The tech note 2125, I'll wait !%while you look that up and read the previous 2124, but probably not, I've done it, just a few minutes left. Thread-safe Programming in QuickTime, that tech note has a section that talks about QTKit usage for multithreaded applications.

Those recommendations include something that Tim mentioned earlier, QTMovie should always be initialized on the main thread, allocate and initialize QTMovie on the main thread. In Snow Leopard, you can also improve the responsiveness of your application via the attribute Tim mentioned earlier OpenAsyncRequired. Then if you need to perform an operation on that instance of QTMovie on the background thread, you can migrate that movie on a background thread which essentially means that you intend to message that movie and you expect notifications from that movie to be posted on the background thread. There are several steps that need to be performed in order to do that. On the background thread on which you want to operate on the QTMovie, the first thing you must do is to invoke the class method enterQTKitOnThread. What does this do?

This tells the underlying media stack that you intend to operate on media on this preemptive background thread and you're signaling the media stack to maintain independent state for that thread and not to interact with its state for the main thread or other threads. In other words, you're telling the media stack to operate in a thread-safe manner. This is essential.

Some of the components that are required for certain media resources, image decoders, audio decoders are not marked as thread-safe. Many of third party ones over the years have been marked as thread-safe, some are still not. You must do this in order to ensure that only those that are prepared to operate on background preemptive threads are allowed to operate.

Then once you've prepared your thread for media operation via the class method I've mentioned, detach the instance of QTMoive from its initialization thread by calling -detachFromCurrentThread on that thread, then with that same instances of QTMovie, call -attachToCurrentThread on the thread on which you intend to operate on it. Now note that some components are not marked as thread-safe especially some legacy ones, so you need to be prepared for -detachFromCurrentThread to fail.

This is similar in many ways to the cases that Tim mentioned earlier when QTMovieOpenAsyncRequired may fail because the components required for the movie are not prepared to operate on a background thread. So if that happens, you need to have a fallback ready. Either you need to perform the operation on the main thread or you need to signal the user that this operation is not available for that particular instance of QTMovie. The tech note I mentioned covers all this in detail, so review that for more information.

A particular point that is not brought up by the tech notes is this about multithreaded operations. Well what about if I just want to play the movie back? Something that you may not know that we'll call it explicitly here is that only QTMovies that are initialized on the main thread and remain attached to the main thread are eligible for playback. So you would not perform any of these thread migration if you wish just to play.

Now this restriction that only QTMovies attached to the main thread can play is not new, it's been true since day 1.

And it's appropriate as well because the playback of the QTMovie involves user presentation. And often occurs together with UI state changes. So it's appropriate for you to message the movie about things you want to do, change its rate, for example, and for it to notify you of state changes on the main thread where your code is going to interact with other UI elements. This restriction is true regardless of whether QuickTime 7 or QuickTime X is used for playback.

Now several times we've mentioned multithreaded applications and components that are not thread-safe, we want to make this better. We want to make the experience of media types that are supported by third parties as rich and safe as possible on Mac OS X. So let me go through some notes for component developers for those of you out there who implement support for media formats on QuickTime X that it doesn't support natively.

You say I want to be part of the media system, and many of you already know that QuickTime 7 has published APIs and has described component types that allow you to add support for new types of media to the system, these come in various flavors. If you want to add support for new video types, you can write an image decoder, an image encoder if you want to encode that video format on the system. Similarly for audio, there are encoding components and decoding components you can provide.

Also if you have a media container format that you want to add support for to Mac OS X, you would write what's known as a movie import component. And the job of that movie import component, by the way, to harken back to an earlier part of our talk, is to create a movie representation of the media resource in that container format that you support.

You can, beyond that, add additional support for new container format by allowing applications to write media to that container format, in order to do that, you would write what's known as a movie export component. But the question is, in our history, how quickly, how immediately do we provide support for all of this extensibility to QuickTime 7?

And the answer is, and this surprised actually someone a long time member of the QuickTime team just the other day, only image encoders and decoders were actually available to the public, to third party developers, when QuickTime 1.0 shipped in 1991, and the reason for this is that extensibility is something that needs to be worked out in great detail. How do we support add-on modules, how do we make them work well when they're added to the system.

What do we want to do in the future now that we're facing this same set of requirements for QuickTime X? We do want that system to be extensible. And further since 1991 the first time that we did this, we've gained a tremendous amount of respect for the need of security when code modules are added to the system.

But how do we do that for QuickTime X and make that extensible? We're working on it. QuickTime X and Snow Leopard is the start of this new media stack, this start of this new set of service with greater efficiency. For Snow Leopard, we, we're not saying anything now about plug-in modules, it's something to look forward to in the future.

In the mean time, we are very interested in your feedback for what kinds of things you want to do in QuickTime X. So for now, the extensibility API remains the QuickTime 7 API and the component types that we've mentioned earlier. But that's the way to extend the media system with new types for Snow Leopard as well as for earlier releases of Mac OS X, let's make them as good as possible.

Let's make them as thread-safe as they can possibly be. The thread-safety of these components is now going to be exercised as never before on Snow Leopard as QTKit clients opt in to use openAsyncRequired. Because what's going on under the hood when an application such as your says I want to-- I want to require asynchronous loading for this instance of QTMovie, is that QTKit when it's on the QuickTime 7 path is going to call NewMovieFromProperties on a background thread which will require that all of the components needed for that movie be marked as thread-safe.

Therefore if you're supplying components for QuickTime 7, media import components and so forth, their thread-safety is going to be tested as never before in Snow Leopard. Now if you wish to make your components thread-safe, the tech note that I mentioned earlier, 2125, it's kind of long, it's got a lot of information in it, also has tips for QuickTime 7 component developers, some general information about how to make those add-on modules thread-safe, how to mark them as such, and how to ensure that they are in fact safe.

But there are particular guidelines for movie import components that I want to highlight that are not called out in the tech note. Over the years and in fact recently since we've implemented QTMovieOpenAsyncRequiredAttribute, we have found that third parties have marked some components as thread-safe but are doing some things that are actually dangerous.

A very strong recommendation, I can't make it more strongly, is that if your component is operating on a background thread, please do not display user interface. Some movie import components want to display a loading progress or something like that. Any display of UI when attempted from a background thread can crash and that's a poor user experience.

So don't do that. When your movie import component is open, note the thread on which it's running, if it's Cocoa-based you can use NSThread or you can use the pthreads package and call pthreadMain to find out if you're running on the main thread, and if that's the case, it's safe for you to display UI, otherwise no.

Further if you're writing a movie import component, you are responsible for ensuring that all of the components required for the movie that you're building, the movie representation that we discuss earlier that the movie import component builds, all the components, the media handlers, the video decoders, the audio decoders, they must be thread-safe also.

If not, you must fail in a particular way in order to signal the client to retry the operation without requiring operation on a background thread. Note that as you're doing a survey of the components that are required for the media formats that are contained in the media resource that you're attempting to load, that Sound Manager components are never thread-safe.

So if the only decoder available for sound format is a Sound Manager sdec component, it's not thread-safe. And you must fail if you're trying to create this on the background thread. Instead in order to provide thread-safety for audio decoding, you must provide a core audio compliant audio decoder and that can be thread-safe. And in fact, the ones supplied by Apple all are.

Now I mentioned earlier that you must fail in a particular way to signal the client of what went wrong. What you want to do when trying to import a media resource, return the result from say NewMovieFromDataRef, you want to set the error to the OS status to componentNotThreadSafeError and this will signal that the whole thing ought to be retried without reliance on the thread-safety which is not available for this media resource.

OK. So that covers the enhancements to QTKit. It also covers some of the things that you can do in your code to gain the benefit of these enhancements and even some things that third parties can do to make experience of media even better on QuickTime X. Tim has some summary of all this for you, even now.

Well there really isn't much of a summary If you want to opt in to QuickTime X path, use special initialization attributes to request these new features. There are really just 2 of them, there's the OpenForPlaybackAttribute and the OpenAsyncRequiredAttribute. Let me point you to some other sessions that should be of interest to you, Kevin has already pointed to several of these.

For Color Management, Friday morning in this room, they will discuss color management in Snow Leopard. The HTTP Live Streaming will be tomorrow in the morning. And also the Sophisticated Media Playback through Open Web Standards should be interesting to you and that will also be tomorrow in the afternoon.

We will also be available in the Labs. The QTKit Lab will be this afternoon and there should be a good number of QTKit and QuickTime engineers there who can answer your questions. And if you have particular questions about HTTP Live Streaming, then that Lab will be Wednesday afternoon also at 3:30. So thanks for coming. If you have particular questions, you contact our Evangelist, Allan Schaffer, and I certainly want you to look at the documentation and sample code. So thanks for coming and I hoped you enjoyed it.