This is going to talk about how you can integrate QuickTime support into your application by using QTKit. My name is Tim Monroe and I will soon be replaced by David Underwood to talk about the capture APIs we've added. So let's just give a little bit of an introduction to QTKit, because I know a lot of you are first-time attendees. And even those who have been here before may be looking at QTKit seriously for the first time.

You all know, I hope, that it's a Cocoa framework for working with QuickTime content. That means that it exposes a set of Objective C classes, and we use all the normal Cocoa idioms to communicate between objects in the framework and your objects. For instance, the objects in the framework will issue notifications when their state changes, you can modify the behavior of various objects by implementing delegate methods, and so on and so forth.

In Leopard, QTKit has really just put the pedal to the metal. Anywhere in the user interface that you see a QuickTime movie or a thumbnail from a QuickTime movie, chances are very good that QTKit is behind the scenes there doing the work. Which was not true in earlier operating systems.

In Leopard, there are two additional reasons that you may want to look seriously at QTKit. The first one is as I already mentioned, we've added some new capture capabilities to QuickTime. But currently in Leopard, the only way to get access to those new capture capabilities is through QTKit. There's no C-level interface to that private framework. And the second reason to be interested in QTKit is that if you're working with a 64-bit applicdation, once again, QTKit is the only way to get access to those capabilities, to QuickTime capabilities.

So what are we going to talk about here today. I am going to start with a real quick overview of QTKit, and I'll do the basic build an application from the ground up sort of thing, so that you see how easy it is to get QuickTime playback and even QuickTime editing in an application.

I'll talk then about the new APIs we've added to the existing classes; that is to say those that existed in Tiger. And then we will talk about some easy integration with other parts of the imaging system. In particular, I'll show a nice example of using Core Image to modify what comes out in a -- in the movie that you see on the screen. I'll talk a little bit, but not much, about Core Animation, and then I'll say a little bit about 64-bit operation. And after that, the second half of the talk will be largely about audio and video capture.

So let's get started. As I said, QTKit is a Cocoa framework for working with QuickTime content. In Tiger we had five classes in QTKit. If you're familiar with QuickTime you know that the data is organized into movies. A movie is a collection of tracks. And a track can refer to media data. So the first three classes that I've listed here are object-oriented wrappers around those basic QuickTime notions.

You may also know that the basic QuickTime way of referring to the location of a movie or to media data is what's -- with what's called a data reference. So the fourth class that I've listed there, QTDataReference, again, is an object-oriented wrapper around the notion of a data reference.

And the final one should be obvious. QTMovieView is a subclass of NSView which you could use to display a QuickTime movie in a window or in a view onscreen, and also to control the movie in various ways. Now I'm only going to talk about the first and the last class here today. The middle three are for more specialized uses that we just don't have time to get into today.

So let's look at QTMovie. QTMovie is by far the richest class that we have. There are dozens and dozens of methods in QTmovie. So I'm going to touch on just a few. The first thing you're going to want to do with the movie is open it up from a file. And you'll use initWithFile error to do that, probably. So you pass in a file name and we will give you back a QTMovie that is associated with the QuickTime movie data in that file.

The standard way to change attributes of a QuickTime movie is with the setAttribute forKey method. And we'll see an example of that in just a minute. If the user has opened a movie and done some edits on it, and you want to save the edited movie back into the file, you'll call updateMovieFile.

Now the last two methods that I've listed here are very interesting. It's important that you understand that QTKit is a subset of QuickTime functionality. It's a proper subset. So the entire CAPI is not reflected in QTKit. There are some things that we just have not had the time to implement.

And so what you need sometimes is a way to get down and use the CAPI in areas that we are not offering an object-oriented wrapper. So the final two methods here, QuickTime Movie and QuickTime movie Controller, are ways -- what I like to call escape hatches. They let you get down into the CAPI. You can get the Movie or the Movie Controller associated with the QTMovie, and then call the C Language functions on that identifier.

I said QTMovieView is a subclass of NSView. So you will instantiate a QTMovieView by calling the standard initWithFrame method. Or more likely, you'll just put a QTMovieView in your nib file and it will be automatically instantiated when you open that nib file. To set a movie into a view, very simple. You call setMovie.

If you want to get the movie that's currently associated with a QTMovieView, you can call the Movie method. And the last one here is just an example of the sort of method you can use to modify the user interface shown by QTMovieView. This one in particular hides or exposes the resize indicator in the controller bar.

Now in addition to those five classes we also have several data structures that you will use in the QTKit API. In particular, these two; QTTime, and QTTimeRange you will use to indicate a particular time in a movie, or a duration, and then a time range. So the QTTime -- let's look at timeScale.

The timeScale field is the number of units per second that you are to interpret the timeValue field in. So for instance, if I wanted to indicate a point that's 2 seconds into a movie, I could set timeScale to 1, and timeValue to 2. Or I could set timeScale to 600, and timeValue to 1200. And so forth.

And as you can see a QTTimeRange is just two QTTime structures. One of which indicates the beginning of the range and the other of which indicates the duration of the range. And we also have a whole bunch of functions that you can use to create and operate on QTTime structures or QTTimeRange structures. And here I've listed just four of them. The standard way to get a QTTime structure is to pass in the timeValue and the timeScale, and QTKit will give you back a QTTime structure.

If you have two QTTime structures and you want to increment them, add them together, you can call QTTimeIncrement. If you want to scale a given QTTime structure with a different timeScale you can call QTMakeTimeScaled. And the last one, again, fairly obvious. If you want to create a QTTimeRange, pass in the two QTTimes for the start time and the duration and you'll get back a TimeRange.

All right. That's enough of the overview that I want to give. What I want to do now is jump right in and actually build a QTKit application from scratch. So we're over here on the demo machine. And I've got Xcode launched already. And I'm just going to create a New Project. And what I'd like to create is a project that can open one or more QuickTime movies in windows on the screen, and allow you to edit those movies. So I'm going to select the Cocoa Document-based Application right there. And let's call this wwdcdemo.

And this is the default New Project window for a Cocoa Document-based Application. Now the first thing you want to do is make sure that you link in the QTKit framework. So I'll come down here to Linked Frameworks, and Control-click, and then I can Add an Existing Framework. This automatically takes me to System Library Frameworks. And I will just search down here for QTKit Framework. There it is. And I will add it to my project.

The next thing you want to do is specify what sorts of files your application should be able to open. So I will select the target here and get its information. And then under Properties, let's make that a little bigger. I can -- set the extension. In this case, it will be .mov -- and the file type will be MOOV. Now, at this point we can add other file types too. We could add MP3, we could add Swift files.

But for the moment we will just stick with .mov files. So we're done with that. Now let's come up here and define our class interface. So I'm opening MyDocument.h. And I'll change Cocoa/Cocoa.h to QTKit/QTKit.h. And I'm just going to have one thing in my document window. That's doing to be a QTMovieView. And let's call it movieView.

There. It looks good. Okay. Now let's set up our user interface. And this is going to be very simple. Let's open the Document.nib. This will launch Interface Builder Three. And this is the default look of the default document window. We'll just get rid of that text there. And then I will scroll down in my Object Library, until I find this icon which is a QTMovieView. So I'm going to position it so that it completely fills the window.

And then I want to set the resizing attributes so that when I resize the window, the view will resize with it. And the final thing I need to do here is in the Files Owner add a New Instance variable -- sorry, a New Outlet. And of course we want to call that movieView. And then we want to connect that -- why isn't it letting me get -- let me try that again.

I thought I said movieView. Okay. Now let's try connecting this to movieView. Bingo. Okay. So we're done with that. I also need to make a quick change to the MainMenu.nib. It turns out that the default project does not connect to the open dial -- uh menu item -- to anything. So let's connect it to the first responder. And of course we want to connect it to the open document action. Oops. This machine is way too fast.

Okay. I think we're done there. Now the final thing we need to do is add some code to the document class. And the -- got to love these mice. The method we want to change now is windowControllerDidLoadNib, so this will be executed whenever we load a nib file.

And the code we want to add there is very simple. We'll just open the file -- so here you can see if the user has selected a file in the open dialogue box, I'm going to call movieWithFile to create a QTMovie object from that file name -- from the data in that file.

Then I'm going set the movie to be editable. By default a movie is not editable. So I want to allow editing. So I will set it to be editable. And finally, I will just set the movie onto the view like that. So let's save that, and if everything has gone right we should be able to build that. This will take just a second.

Okay. Build succeeded. Let's try running it and see what we get. There's the default window, so let's actually go out and open a QuickTime movie. Here is the stupid movie. And it opened. And -- it should play fine. There we go. We have no audio out on this machine, but that's okay. Notice that I can resize it.

But there's one problem here that you can see. The Window Resize box overlaps with one of the controls in the QuickTime movie controller bar. So I can step back by clicking the Step Back button, but I can't step forward. All I can do is use that to resize. So let's fix that problem really quickly.

And that's really easy to fix using the setShowsResizeIndicator method that I talked about earlier. So what we want to do is tell them -- window associated with the movieView not to draw its resize indicator, but we want the movieView to draw its resize indicator. So let's save that. Build it. And run it.

And in this case, when we open that same movie -- now you can see that we have the Movie Controllers resize indicator there, and I can indeed step forward with the Step Forward button. But of course we went black there so, as you can see I can step forward with that button.

Okay. So, let's see. Is that all I want to do there? No. Let me run that again and show you that I have a fully editable movie. So I can make a selection range here and just use the Edit Menu to cut. I could say come to the end here and paste it in.

I can also use the keyboard short cuts. So let's select some other range and do Command-X, and that cuts that. And maybe paste it in the beginning here with Command-V. And one nice feature, I can Undo that paste. I can Undo the cut. I can Undo the first paste.

I can Undo the first cut. So I have essentially unlimited Undo with all of my edit operations. Nice frame there. And I got all of that editing capability. All the tie-in with the Edit Menu, all the Undo with that one line of code that said make the movie editable. Okay. If I can go back two slides, please.

So just to recap, this is the little bit of code that we used to open a movie and to make it editable and to set it into the view. One thing I did not show you was if you select the Save menu item you need to call the updateMovieFile method to save the edited movie back into the file it came from. And then as you can see I'm calling an NS document method updateChangeCount to let the window know that it's been cleared. It's no longer dirty.

So it's really just that simple to get an application up and running with QTKit. So let's talk a little bit about new APIs in those five existing classes. Again, I am just going to give you the -- what are the highlights in my mind. We can't go through every new API in every existing class or we'd be here all night.

So we've added three interesting things to QTMovie. The ability to create a writable movie container, and I'll explain what that means in a minute We've added a method that allows you to remove a track from a movie, and we've also really done some nice improvements on the methods that allow you to get an image of a frame in a movie.

So in QTKit as it existed in Tiger, the basic idea was that you could go out and open existing movies, edit them, and save the edited movies back on to disc. Also maybe do things like export the movie in a different form and other things like that. One thing that sort of slipped through the cracks was the ability to create an empty movie, add data to it, and then save that out.

That's a case where you needed those escape door methods to get down into the CAPI to do that sort of thing. So we thought that's not good. Let's add some new API that makes this easy. So we added the API initToWritableFile. And what that does is you pass in a file name, this method will create a file at that location, and pass you back a QTMovie object associated with that file.

You can then add data into that new movie, which is empty. Say, by pasting in from some other movie. Or like I'm doing here. Reading an image out of my bundle and then calling the addImage forDuration withAttributes method. And then I just call our favorite little method, updateMovieFile, and it will write that data out into what formerly was an empty file, and which is now a completely formed QuickTime movie. So that's a very nice addition to the API.

Another nice addition is the ability to remove a track. Now this is a case where even if you've got the QuickTime movie identifier associated with the QTMovie, and went down into the CAPI, and called the CAPIs to delete a track, you essentially would shoot yourself in the foot. You would quickly crash.

It simply was not possible to use a CAPI to remove a track and not cause problems with QTKit. So since people wanted to do that, we added this method -- removeTrack -- which is very straight forward. In this example I am just getting the array of tracks in a movie using the Tracks method.

Then I'm grabbing the first item in that array and simply removing it from the movie. If I would then call up that movie file, the movie file on disc would have one fewer track than it had before. Getting a movie frame image. We've done some very nice improvements in this area. On the top I've listed the API as it exists in Tiger. frameImageAtTime -- you give us a time in the movie, and we'll give you back an NSImage that is the frame at that time.

Now there wasn't a whole lot of functionality here in the sense that you always got back an NSImage, whether that's what you wanted or not. Right? And you always got back an image which was the current size of the movie, whether that's what you wanted or not. So people ended up doing a lot of post-processing on the images they would get back from this method. And we got a lot of enhancement requests.

So we added this new method -- frameImageAtTime withAttributes error. The idea being, you can now pass in a dictionary of attributes describing exactly the kind of image you want, exactly the size you want. And we'll give you back that instead of just a plain old NSImage of the current movie size.

So let's look at some of the attributes you can specify. You can tell us what size you want the returned image to be. Very straight forward. You can tell us what kind of image you want back. So we'll still give you back an NSImage, if that's what you want. Or you can also ask for other types of image. Say, a CIImage, if that's what your application would find most useful.

The final three here are boolean values that you can use to say I want you to deinterlace the fields of this image, I want you to give me the highest quality, and I want you to give me a single field. And we'll get a real good example of how useful that is in a second. This lists the available image types in Leopard. You can get an NSImage, you can get a CGImageRef, you can get a CIImage, you can get a CVPixelBuffer, or if you're working in OpenGL, you can actually get an OpenGLTexture back. And work with that.

If you ask for an NSImage right now using the frameImageAtTime method, you will get back an NSImage with an NS picked ImageRep. Well, QuickTime -- QuickDraw has been deprecated for a little while. So the picked ImageRep isn't going to do you a whole lot of good. And people found themselves writing code to convert that image into something more useful. Now you can tell us exactly what kind of image representation you want in the image we pass you back. By default you're going to get back an NSBitmapImageRep. But you could ask for any of the other available ImageReps.

And if you want an OpenGLTexture you've got to give us two additional pieces of information. You need to give us the OpenGLContext and the PixelFormat. And we'll hand you back a texture that you can then draw into that context. So here's a simple example of how you would use rameImageAtTime withAttributes error to get a CGImageRef.

You'll make the dictionary with one key value pair, saying that you want a CGImageRef. You'll then say what time you want. Say, the current time in the movie. Then you'll call the new method and get back your CGImageRef. So let's look at this a little bit more closely in a demo.

So let's hide that. And I've got a precanned application here just to save us some time. And what I want to demonstrate is the quality improvements that we have made in this API. So I'll just open a random movie. And you can see I've got two buttons there; one labeled Tiger and one labeled Leopard. This is what you would get if you call frameImageAtTime in Tiger, or indeed, in Leopard.

This movie is a DV movie. And it was made with a camera that interlaces the frames. So you can see, especially if you look, say, at the right-hand here, it really looks pretty bad. You're getting a lot of interlace artifacts there, right? That was what you got with frameImageAtTime called on an interlaced DV movie in Tiger.

Now if you call frameImageAtTime withAttributes error, and pass it in nil set of attributes, which essentially says give me all the default values, this is what you'll get. A much nicer image. Higher quality. The interlacing is gone. So let's see that again. There's the old bad way. The wonderful new way.

Just to really drive this home, let's go to the first frame. This looks almost like it's had a CI filter applied to it; it looks so bad. But then in Leopard you'll get a much nicer image. Okay? So that's what I wanted to show there. Can I go back to slides?

Thank you.

So more capability, better quality, it's got to be slower, right? No. In Tiger, or again in Leopard, because the old API is still there if you want to use it, this is what you would get if you stepped through a movie and got a frame image at every frame of the movie.

For a given movie it might take you 131 seconds. Now using the API that I just showed, that is to say frameImageAtTime withAttributes error, and a nil set of attributes asking for the default values, that will happen in 98 seconds. So we've chopped almost a third off that.

Now, mind you, we have done zero work to optimize this new method. All we've done so far is implement it to give all the capabilities that we could. So I was playing around the other day and I thought, you know, if I'm getting the image from the same movie there's a lot of build up and tear down that I don't need to do. So I defined a new private selector or key that puts the method into what I call session mode. Telling the method that I'm going to keep calling this on the same movie.

And then when I run this loop through that movie, using this private selector, I'm down to 53 seconds to get all of the images in that movie. So it's a good chance that I can persuade people to make this a public API, and you could use that, if you're going to step through the same movie.

Let's talk a little bit about Core Image. We added a very nice delegate method to QTMovieView. It's called view willDisplayImage. The idea being before QTMovieView draws the movie frame in its view, it gives you a chance to modify the image. And as you can see we will pass you a CIImage, and then you can do whatever you want with that CIImage.

You're going to pass us back a CIImage. And that's what gets drawn in the frame of the window. So let's look at an example of how easy it is to apply a CIFilter to a movie. If I can go to the demo machine -- I am just going to bring forward the same project that I did earlier.

And this time I'm going add in a Delegate method -- view willDisplayImage -- and it's going to be very simple. If you've worked with Core Image and core filters, this will be perfectly straight forward to you. I am just going to use a glass distortion filter. I'll set the defaults. I'm going to read from my bundle an image that defines the distortion texture.

And then I'll say, well, the input image for this filter is the image that I'm being passed in by QTMovie. And the result that I want to pass back is just the output image of the filter. And one other thing I need to do, of course, is set myself or the movieView as the delegate -- sorry -- movieView -- setDelegate -- and that will just be me; this particular class.

Now I need to do a couple more things. Let's save that, and I can actually close this, but I'll just move it out of the way. Since I'm working with Core Image filters I need to add in the QuartzCore.framework. And I'll do that in just the way I did before -- Add an Existing Framework -- and we'll come down here and look for QuartzCore. There it is. Now, in theory this should build. If I haven't forgotten anything. Build has succeeded. Now let's run it. And let's open that same movie that we opened before. Ah. Something's wrong.

Let me think about this for a minute. Ah. (Inaudible) -- what? Any suggestions? Let me think about this. I added the framework. I set -- oh. Yeah. Very good. Very, very good. Thank you. I forget to add the texture to my project. So I'm going to come over here. And I happen to have it sitting on my desktop. I knew I wasn't crazy. So we'll just add it to our resources.

And now let's build it. And now let's run it. Yeah. And now let's open that previous file. All right. And we'll move to a more interesting frame. And now we see that the same movie plays, but I have added the CIFilter which is the glass distortion. Great. Ok. So back to --

Again, it's really just this simple to add that kind of effect to a movieView. Core animation integration. There isn't a whole lot to say there, other than we have added the obvious classes to support core animation. In particular, we've added QTMovieLayer and QTCaptureLayer. MovieLayer lets you play a movie in a layer. CaptureLayer lets you take data being captured live from devices attached to the computer and put them in a layer.

And I don't have a whizzy demo because you've already seen one. When Steve did the Keynote he had that big wall of layers. The movies playing in there were using QTMovieLayer. Okay. So it's really -- and the performance was excellent. So we integrate very well with core animation.

64-bit. What's to be said. Not a whole lot. I've already told you that if you need to move into 64-bits and you want to get QuickTime capabilities, you need to use QTKit. QuickTime framework is not available in 64-bits. The API is identical to everything we've talked about. All of the new methods that we've added into Leopard are available to you in 64-bit. There's one principled limitation, which is these escape door methods, the ability to get the QuickTime movie, or the QuickTime movie controller, or the QuickTime track, or the QuickTime media is simply not there in 64-bits. Because there's no QuickTime framework. You couldn't do anything useful with those identifiers.

In the three that you've got, there are two limitations that will be eventually removed. We haven't had time to do the drag and drop editing, and there are some drawing glitches. So you should know that when you start building 64-bit applications. So now I'd like to bring David up to talk to you about the new capture capabilities that we have in QTKit.

Thanks Tim. So today I am going show you just a handful of the things you're going be able to do with the new capture API. Again, if we showed you every class and method you would never be able to leave this building. So when we talk about capture, what we're referring to really is you're taking some sort of external real-time media source, and you want to take the data off that and get it onto your computer and use that data somehow.

So in this case -- our API supports, you know, pretty common obvious sources, like the iSight which is a camera that we ship on almost every new Mac now. And also a variety of camcorders and tape decks. And also just about any audio device that you can think of.

And you can take that data and send it to a number of reasonable destinations. So you can record that media to a QuickTime movie. You can preview what you're doing, so that you can make an audio or video preview of what you're capturing. And then maybe there's something we haven't thought of yet in the API. So you want to be able to get at that raw video data directly and do some custom processing on it.

And it's also important to know in this API, Tim said that we're exposing new functionality that's only available in QTKit. And in fact you have been able to do capture in the past in QuickTime using the Sequence Grabber API. But this new API is not a wrapper around that QuickTime Sequence Grabber. This actually sits on top of a new framework that provides a lot of new nice modern functionality that you should come to expect with capture API. So the most important thing that includes is you can now share certain cameras most notably the iSight, between multiple applications.

So if you're running QuickTime Player and you're running Photo Booth and you're running a variety of different apps, you'll not longer get these messages if those applications are using QTKit that say this camera is in use by another application. So, you pick up this API you become a good citizen.

In addition, we also put a lot of effort into synchronizing audio and video together. Everything we do under the hood is sample accurate and time stamped and we put a lot of effort into getting so that your video and your audio line up. We also have support for some new devices.

So if you have the appropriate codecs installed on your system we can now do HDV capture, which Sequence Grabber was never able to do. In addition, there's some new interesting use cases that are just sort of a new paradigm Sequence Grabber never explored. You now will get called back whenever an individual video frame or audio sample is captured, so you can have various frame-accurate capture use cases. So, say you can record a specific number of frames to a movie file, or record along certain timecodes.

And we also give you an API for controlling the tape controls on certain devices which is something you used to have to go to the AVC framework to do. And there's plenty more under the hood that we don't have time to talk about. And I'm just going to highlight these three points up here.

If you have an applications using Sequence Grabber now, and it's -- you have a basic use case, you're not really interested in frame exact recording or any of these new ideas. If you moved your app to QTKit, you would still get these three things basically for free. These two huge advantages. Particularly sharing devices between apps so you become a good citizen. The A/V sync has improved, and new device support.

And because of that, Sequence Grabber is still around. We're not deprecating it. So you have nothing to worry about there. But if you're currently writing an application that uses Sequence Grabber, or certainly if you're developing a new application, we strongly encourage you to move your app over to QTKit.

And similarly, QTKit is sitting on top of a brand new much more modern device model. And we haven't quite made the API for that public yet. So, if you're in the middle of the developing a device and you're writing a current vdig component that interfaces with Sequence Grabber to let you capture from that device, come to the QTKit lab tomorrow and talk to us and we'll figure out how we can get your device working with the new device model. And finally, this is a brand new API. It's a first revision. We tried to get down a lot of the sort of more common use cases that people wanted.

But your feed back here is very, very important. So certainly, if there's something you're doing in Sequence Grabber that you don't think you can do, or can't do using QTKit please tell us, and if there's anything new and cool that you want, that you want at our API level, please give us that feedback. We've already got some great feedback earlier this week in the labs. So let's go over the basic concepts behind what you do when you use the capture API.

So when you're doing capture you have the concept of inputs and outputs. So your inputs are logically various devices, like cameras and microphones. And your outputs are wherever you want to send that data. So say you want to draw a Cocoa view or play audio through a speaker, or write that data to a QuickTime file. And finally you need something sitting there in the middle there that distributes that media to the various destinations it needs to go to. And also performs audio and video sync and decides when to capture and when not to capture.

So the way we abstract that in QTKit is with these three classes. The QTCaptureInput, the QTCaptureOutput, and QTCaptureSession classes. And input and output are obviously your inputs and outputs, as a described. In the sessions this thing sits in the middle, controls when the data is transferred and performs sync and does distribution of media data.

So those are just abstractions. Concretely, we actually provide you with a handful of concrete sub classes of QTCaptureInput and QTCaptureOutput. But what we provide you right now as far as inputs go is the QTCaptureDevice input. Which, as the name implies, let's you grab data from a device. And we give you a variety of outputs. Movie file output will record to a file. Decompress video output will hand you via delegate method every single video frame that's captured and decompressed, so can do some custom operation with it. And then you have ways of previewing the video and audio in your session.

So you can use these inputs and outputs basically like building blocks. And depending on what your application needs to do, you create a Capture Session and then add inputs and outputs as needed, depending on what your goals are. So a really simple example is audio video recording. So this is what the recording feature in QuickTime Player does it we have a couple of Capture Inputs here. One for a camera and one for a microphone, say. And we have those hooked up into the session. And then we have a few Capture Outputs. So we have a couple for doing video and audio previewing, and most importantly we have something that records the file.

So -- an example that you have the source code to that can do this is MyRecorder. And this is actually a sample that we gave out as part of the coding head starts program. So hopefully at least some of you have done your homework and taken a little bit of a look at this. But if you haven't seen it, what it does basically is just what I showed you in that graph, which it does a simple -- grabs the camera and the microphone and records what's coming in there to a file.

I should also point out we had a few bugs in the sample, actually. And when we rushed to release it, we now have an updated sample that fixes those bugs at the web site for this session. So if you want to follow along with the sample you should definitely download it and check that out. That will show you all the best practices.

And so I am not going to go through all the code. Actually, I have a much more interesting demo later on. But I'll just kind of take you through the basic steps that you go through in an application like MyRecorder to do your capture. So in this case it's three pretty simple steps.

You want to find some device to work with, or a couple of devices, if you're using multiple devices. You create the Capture Sessions. So that's the slide I showed you earlier where you have a session and you have inputs and outputs to it. And then you to something with that capture while it's running.

In this case, we're going record to a file. So -- and the first step, finding a device, you're going to go and look at the QTCaptureDevice class. And this is a class that represents all of the devices hooked up to the computer. So in this case we're going to do something very simple. We're just going to ask the machine what's the default device of a certain media type you have on your system. So if you had a MacBook Pro, this would be the built in iSight.

And what you also have to do before you use a device that's very important is open it. And what that tells -- what that does is it tells the device system I'm about to do some capturing for this device. Do any set up you need to do. And also it's important to know when you're done with that device you have to call a closed method so that you free up those resources. The next step we're going to build a Capture Session. And this is just a code version of that slide I showed you earlier.

It's a little bit simpler. I fit it on the slide. But, we'll just create a Capture Session using standard Cocoa allocation. We'll create a device input for that device that we just found. And add that as an input to the session. And we'll create a movie file output which is actually responsible for the movie recording.

And add that to the sessions as an output. And one other extra thing we're doing with the movie file output is setting a delegate on it. So the movie file output is interesting that it does a large number of operations on background threads. So it's important to set a delegate, and you'll see why in a second. So that you can be notified when various operations are running and when they're completing.

And then finally say we have a view in our user interface that's previewing what's going on. We're just going to say preview what's going on in this Capture Session. And then finally we have everything set up so we will tell the session start running; start grabbing from inputs and sending data to outputs.

And then finally we go to step three, which is actually use the session. So we have defined a couple of actions here. The first says start recording. And we just tell the movie file output record to this URL. And then we say stop recording. We just tell the movie file output -- don't record to anything any more. pass a nil. So the interesting thing about this action method is when you call record to output file URL it always returns immediately.

So even if there's some remaining work that needs to be done on the files it's writing. So it probably has to write in the QuickTime header. Maybe in the movie add, maybe it saw some cache data that it needs to write out. This method will still return immediately, and all that work will go on happening in the background.

And the reason why this is very powerful is because you will now be able to switch files that you're writing to on the fly, without losing any media or samples. But the down side is that makes things a little more complex, because you couldn't just call recordToOutputFileURL nil, and expect to be able to open that QuickTime movie.

It's not actually done yet. So you remember before we set a delegate on the movie file output and had the file is done for real, we get called back with this delegate method that we implement. And it looks like a bit of a mouth full, but it's pretty straight forward.

CaptureOutput did finish recordToOutputFileURL for connections due to error. Gives you tons of information. But all we're going to do here is we have a perfectly good movie file now, and so we're going to open it up in the work space, which just means launching it in QuickTime Player.

So I'm just going to do a quick demo of what an application like this looks like. Again, I'm not going to walk through all the codes. I kind of have a more interesting demo to show you in a second. So I'm just going to close this stuff off.

And so what I have here is this is just a built version of the MyRecorder sample that you can get at the session web site. And you'll see I have an iSight camera hooked up to this computer, and I have a nice mascot subject here for my demo. Oops. He has a little bit of trouble standing up, unfortunately. He's old, what do you want.

And -- I can just record kind of a, you know, make him dance around a little bit. So I'm recording all this to disc. That was that Start Recording Action. And when I hit stop, it tells it stop. And then there's a little bit of a delay, and then the application gets called back had the movie file's done for real, and then we can openly it up here and here's the recorded movie. See, there he is dancing around. So, success.

And while I'm here showing this to you. You remember before I mentioned one of the big benefits of using QTKit is that certain devices including the iSight can be shared across multiple applications. And in fact in Leopard a ton of applications and other frameworks on the system will be able to take advantage of this.

For example, we can open up QuickTime Player and create a new movie recording. And we still see that everything is fine there. Let's see. We could open up this bizarre Quartz Composer composition, we're going to run out of the space on the screen pretty soon. But you can see again, everything's being shared. Say I want to change my account picture in System Preferences.

Just go ahead and do that. And this actually uses the standard Image Kit Picture Taker. So if you use Image Kit in the Picture Taker class that takes advantage of this sharing feature in QTKit. And again, you see this all works simultaneously with all these other applications.

All right. So that was kind of a predictable use case. It's very simple. I've got a camera, I want to record a movie with it. What if you want to do something more interesting. What if you want to do something we really haven't thought of yet. You want to get those video frames off the camera and do something special with them.

So in that case what you're going to do is you're going to use a different kind of output. The QTCaptureDecompressedVideoOutput. And what that's going to do is every time a new frame comes in it's going to call you back by a delegate method and say here, do something with this frame. I don't care what it is.

And so -- in this case, the R diagram's actually much simpler. We just have a Capture Session with one input, which is a device input for the camera, and one output, which is the decompressed video output. And what it's going to do is every time a frame comes in it's going to hand us a CVImageBuffer representing that frame.

And so I -- what I'm going do now is I am going to build this application for you from scratch. And this is actually also an application you can get already built on the Web site for the session. It's called StillMotion. And it does exactly what I said. It uses a decompressed video output to create a StopMotion animation. So I don't know if you've seen animations like this.

Various -- you've seen claymation where it takes -- you take a snap shot of a current scene and then you add that on to a movie slowly. And you can create kind of entertaining scenes and animations with this. In a matter of minutes we're going to build something that actually let's you be pretty creative. And so I'll show you around this. And so now I'm going to demo building the StopMotion Animation Application.

Okay. So this will be somewhat similar to the movie player that Tim built for you before. But instead of opening and editing movies, it's actually going to create them from scratch. So we're going to create a new project in Xcode and again it will be a document-based application where the data in our document will be a QuickTime Movie, or a QTMovie object.

And we'll just call it StopMotion -- okay. And again, since we're using QTKit the first thing we want to do is make sure that QTKit is added to our list of frameworks that we're linking against. So we go in and open up the existing frameworks. And we'll go and find QTKit.

And also we're going to be doing a little bit with Core Image and Core Video. So, like, before we also wanted to add in QuartzCore.framework to let us link against that functionality. Okay. And also like before we're working with movies, that'd our document. So I'm opening up the info panel for the StopMotion target type. And I'm going Properties and I'm just changing this to show we've opened movie files. So I'll change the file extension that we support to mov and the OS time to MooV. And great. Now we can open and save movie files. So let's get to some code here.

So I'm going to go to my document class. And the basic idea here is that we have two things going on. We have -- we have on the one hand what's currently -- the image is currently being captured by the camera hooked up to the computer, so the current scene as you've set it up. And then what we also have is the project that we've created before. So the movie that we've built. So we need to show both of those things in the user interface.

So -- what I'm going do is I'm going set up the instance variables in my class to show this. So I'll just replace recall of this here, and I'll walk you through what I've added here. So the first two things I've added are two things that will be in the user interface, two out lets in Interface Builder. So on one side we're going to have a CaptureView, so I'll show you that live scene that I told you about. Then on the other side we'll have a MovieView. And I'll show you the movie we created so far.

We also need a handful of things to store inside the document. So a handful of instance variables. We have a QTMovie object. So this is the movie itself that we're creating and adding to. And then we have these three objects which are -- these are the building blocks for the CaptureSession that we want to use. And they refer back to that slide that I showed you before where we have a session, and then on one end we have a device input, and then on the other hand we have a decompressedVideoOutput. And finally, we have one little extra thing here.

We have an instance variable. And what this is going to do is this is just going to store the most recent video frame that we've captured. So when we go around adding something to the movie to make our scene we're just going to grab it out of this.

And then finally I defined one method and this will be hooked up to a button, and this will actually take whatever is in front of the camera and add that in as a still image to the movie. So now that I've defined the interface to this class I am just going to pop open the document nib, and I am going to build this user interface.

So I'm going get rid of this default stuff. And I'm just going to add -- so remember we have both a CaptureView and a MovieView. So we have this icon in Interface Builder Three for the QTCaptureView class. So we'll just drag that in there. And we also have the QTMovieView that we'll drag in. And we'll make these both kind of a reasonable size.

Oops. And I'll wind those up in the movie, and I'll do the same thing with the MovieView. And the only other thing I need in here is I need a button that will take a snap shot of whatever's currently being displayed in the CaptureView and add it to the movie. So I'll go into the Interface Builder library and go find a button -- there's one.

And I'll just center that in the window a little bit. Ah. Close enough. And I'll call that add frame. And also set up all these things so that they resize pretty nicely when I resize the window. So here's how I set up the CaptureView. Here's how I set up the MovieView, and I'll make it so the button sticks to the bottom of the window.

Okay. And now that I've built my user interface I can hook it up to that header file that I showed you before so I have a couple of out lets here, so I'll drag from the file zone here, and this is my CaptureView. So that's connected. And I'll drag to here. And here's my MovieView. So that's connected as well. And finally I have this action. So I'll hook up this button to that method that I find in the class. And that's the add frame method. So we're done with the user interface now.

So I'll just quit out of the Interface Builder and now we'll get to writing some code. This will be the meat of it. So kind of similarly to what we did with MyRecorder. We have kind of three basic things going on. Three basic steps. We need to get some kind of a device and open it , we need to build up a Capture Session to run it, an then use that Capture Session to do something useful. So what I'm going to show you first is this set up step where we find a device and create a Capture Session.

So -- I'll just replace the window -- no, I need that. I'll replace this windowControllerDidLoadNib method. And it looks like a lot, but it's actually pretty straight forward. So, after calling the super classic imitation -- so remember, our document here is we're building a movie. And unlike what Tim showed off, we might be creating a brand new movie from scratch. So what we're going to do first here is check -- do we have a movie already. And if we don't we're going to create a new QTMovie bject that we're going to write into.

And this is also a brand new API added into QTKit. So it's similar to that initToWritableFile method that Tim showed you earlier. But in this case we haven't really saved a document yet. This is a brand new document. So we're going to use a counter part of that method, which is initToWritableData. Which just takes an NSMutableData object. And all we're doing is saying I want a new movie and I just want to write into this block of memory that we've cordoned off, which we can save to a file later.

And so once the movie is created, now that we have a movie, we have a MovieView in our interface. So we want to make sure that that MovieView is displaying the movie that we've created so far. And then finally we go to set up our Capture Session. So, very similar to before.

We just create a new QTCaptureSession. We find a device, and in this case again I am just going to use the default device on the system that supports video. Open that device up, and then add our inputs and outputs. So create a device, input the device, and add it to the session.

And create a decompressed video output where we're going to get our frames and add that as an output to the session. And kind of like the movie file output, we need to set a delegate on the decompressed video output. And what that let's it do is call us back whenever a new frame comes in. Which I'll show you in a second. And finally, we want to preview whatever's going on in the CaptureView, in the window. So we hook that up to the session. So we hook that up to the session. And everything's set up now so we say start the session.

And what we've also shown you here, I didn't really show you any clean up code had I showed you MyRecorder demo, but it is important to clean up. Especially since you're running captures which can be pretty resource-intensive. And you're also opening devices, which is also fairly resource-intensive. So when you're done with everything you want to kind of clean it up.

So doing that when the window is about to close is a good time. So right here we stop the Capture Session from running. Oops. And we also -- because we've opened that device before, we're done with it now. We're going to close it. An then in dealloc, we just release everything that we allocated.

So now that we've found a device, set up the Capture Session, we're going to do something interesting with it. You remember before I set a delegate on the decompressed video output. So here's the delegate method that you immediate to implement. And I'm just going resize this window so you can see -- I'll have to scroll over. Again, this method gives you tons of information. In this case we're really not going to use much of any of it. But it's Capture Output, did output video frame with sample buffer from connection.

And so all we're really interested in here right now is this video frame. Which is a CVImageBufferRef. And what this method is going to do is it's going to take that CVImageBufferRef and store it in that instance variable we created. So we can always hold onto the most recent image that's been grabbed.

And so this actually a pretty standard assignment. What we do is we retain this new video frame and then we assign it to this instance variable. And then we release the old one, because we don't want to leak memory, 30 times a second. So one kind of interesting thing here is you notice I put this in the synchronize block.

And the reason for this is when QTKit calls these delegate methods, they're not guaranteed to be called on the main thread. A lot of traditional Cocoa programming practices kind of expected delegate methods to come on the main thread. But QTKit is very, very multithreaded. And for efficiency's sake it doesn't do a lot of interthread communication by default. So you have to realize that you shouldn't always make that assumption when you implement a delegate method.

So in that case we just need to do a little bit of thread safety work to make sure that we don't clobber our data or have any erase conditions. So now that we've done that, now that we're storing our most recent frames, we're going to implement that action that's called when you click on that button. So I'll just put that in here.

And here's the add frame action. And what this essentially does is it grabs the most recent frame and puts into the movie using standard QTKit APIs that you may have seen before. So the first thing we're going to do -- and again, we have to do this in a synchronized block so we avoid race conditions is we grab the most recent image buffer out of this instance variable. And we retain that because we're going to be using it on a different thread.

And then assuming we got an image buffer out of there what I am going to do is turn it into an NSImage. And then call this QTKit API, the addImage:image forDuration withAttributes method that takes an NSImage and then just amends that image onto the end of the movie. So in order to make the CVImageBuffer into an NSImage -- this will seem a little bit contrived, but it's actually pretty efficient. We're going to use Core Image, which knows how to draw CVImageBuffer, and NSCIImageRep, which can contain CIImages, and then put that into an NSImage.

So it sounds like kind of three levels of translation. But in fact there's only actually -- all the data is in one place. No one's making any copies -- so -- you don't need to worry. So just think of this -- we're turning a CVImageBuffer into an NSImage.

And so when we go ahead and add the image to the movie we're just going to take an arbitrary time. Let's say we want to make our StopMotion animation ten frames per second. You could add another user interface application that changes that, but we'll keep it simple here. And let's say we'll compress it to JPEG. We don't want raw YUV frames exploding on your system, you know, and having taking of gigabytes of disc space. So we'll do a little bit of compression. And then we'll do some nice interface -- user interface updating.

So we'll just tell the movie make your current time your duration. So scroll to the end of the movie. And that will show in the user interface the most recent frame that you've added to the movie. And you tell the MovieView redisplay yourself when you get the chance. And then we say we've edited the document, so let's mark it as dirty. And finally, we actually want to be able to save these movies. So I'll just add a little bit of simple code to be able to open and save. And oops -- that's what I want.

Okay. And I'm just overriding these standard NS document methods, read from URL and write to URL. And read from URL is very simple. We just create a new QT Movie with that URL, and -- what we also want to make sure is we're opening something, we might be adding images to it.

So we make sure to set that editable attribute to yes. And then write to URL is very simple. We just tell the movie write to this file. And then we pass on a little flag that says flatten this movie. Which just kind of means arrange the sample tables and the chunk sizes in such a way that it's convenient.

But you don't need to do that. That's just a nice optimization that you can do. Okay. And with any luck this should be everything we need. Yes, I do want to save that. And we'll just go ahead and build this. And hopefully be able to create some brilliant movies. Okay. Build succeeded. Let's see.

Okay. So here's a brand new document window. And we see we again have a live preview of our Capture Session in here. So we're going to use again -- our friend Einstein is the subject of our StopMotion animation. So I'll have him kind of walk up to the camera. And you'll see -- it will be a nice animation. So I'll move a little closer. A little closer still. Well, kind of shifted over there. Well, he doesn't want to be in the center of the frame.

You know, maybe a little further. Maybe it's getting a little creepy now. All right. Well, anyway, I could do this for hours. I probably shouldn't keep you here forever. But as you can see we're just taking this data from the camera in -- we've, you know, in -- you know, a matter of ten minutes we've built an application that actually let's you do something pretty cool, which is this creative application right here, to create StopMotion videos.

And if you really spend the kind of time on this, you know, you can hang out in your office and do work on your white board and create masterpieces like this. So you can see this is the artistic talent that goes into this kind of effort. So, yeah. There it goes.

So, just real quickly, I've known a couple of use cases for what you can do with the QTCapture API. I'll walk you a little bit through some of the common classes and methods that you'll use. But definitely won't cover them all. But this will be things that you'll see commonly in your applications. So QTCaptureSession you'll always use if you're doing capture. And you've basically seen these methods. You can add and remove inputs and outputs. And it's also important, you need to start the session when you want to start capturing and stop it when you don't need to any more.

QTCaptureDevice input. Very simple. You just initialize it with a device that you've opened already. And you can add that to the session. QTCapture movie file output as you saw before. The most important methods are telling it where to record. And also there are a ton of delegate methods that you can implement.

But the one that you're always going to implement, basically, is this one that tells you when it's done with a file. And you can actually us it as a useable QuickTime file. Another delegate method that I didn't show you in the demo but it opens up some more interesting cases for movie file output is you can also implement this method, which gets called every single time the movie file output receives a new video or audio sample.

So you can -- you get this QTSampleBuffer object that contains both the actual sample data itself and tons of meta data about it. So if it has a timecode, you'll be able to get the timecode. You'll be able to get timing information. If it has date recorded information, you'll get that. So all sorts of interesting stuff.

And you can use that actually to precisely control where to start -- start and stop your recording, as well as getting information about individual samples. So that's pretty cool. Video Preview outputs. So, Video Preview output is kind of the guts of QTCaptureView. QTCaptureView actually creates one of these and reads frames off of it when it wants to do a Video Preview.

But if QTCaptureView is kind of a little bit too simple for your needs, basically shows a single video frame, then you can use this class instead and get those preview frames directly and do something with them. A good example of that is kind of an obvious way to extend that StopMotion application.

Say you want an onion skinning effect, where you draw a translucent version of the Video Preview on top of the movie that you've created so far, letting you kind of more precisely position the various things in your scene so you know, you have something a little better than what I did with Einstein over there. And you can do that by hooking directly to the frames that are returned with this class. And there are two ways to get at those frames.

You can use a QTVisualContext, which is a QuickTime C API type, which does some nice image timing and queueing, and let's you get at the CVBuffers in good time. The only caveat about that -- that's not available in 64-bit because again that's in the C QuickTime framework. But what you can also do is implement this delegate method that looks a lot like decompressed video output delegate method that will just call you back every time a frame of Video Preview comes on. Here's decompressed video output. There's really only one method that you care about, which is this delegate method that calls you for every frame.

Audio preview output, we didn't go over. But it's kind of too simple to even -- worth mentioning too much. But it let's you play audio -- whatever audio you're capturing from your session, you'll play that through the computer's speaker. And something you'll probably want to do with it is set the volume. QTCaptureDevice is your main interface to finding and opening up devices. This is actually a pretty huge class already. But here are some highlights of what you can do. So these first two methods let you find different QTCaptureDevices.

And you can either get a list of all of them if you want to build some kind of a menu. Or if you don't really mind. If you just want the system to figure it out you can just get the default one like we did in these demos.

And you get lots of information about the device. One good example is a localized name of it that you can put in your user interface. And also very important, you need to -- when you want to use a device you need to open it. When you're done with it, you have to close it.

We also give you some information on the connection state of the device. So in particular we define these two notifications that will call you back whenever a device gets plugged in or whenever a device gets unplugged. So this is a nice new thing we get with this framework.

Before, when you were using Sequence Grabber you would have to dive down into IO Kit to find out when things get unplugged. And now we defined these notifications for you. And we also implement some methods that tell you whether another application is hogging the device, or whether or not that device is connected.

While we're talking about devices I should just go over quickly the devices that we're supporting in Leopard and in your Leopard seed. We support both the VDC over USB and IIDC over FireWire protocols. And they correspond to the built in and FireWire iSights that Apple ships. And also a number of different USB web cams. We support any audio device the Core Audio can see.

We also support a variety of DV and HDV cameras. And we have a little asterisk there and that's because if you're using a camera that uses a pro codec, like DVC Pro HD, or HDV, you need Final Cut Pro to be installed in order to be -- in order to decode the frames coming off that camera.

So that's one thing to note. And we also have a little bridge in there that if we don't have any device that's on the new device model, we do have a bridge that supports Sequence Grabber devices through their vdig component. So those devices should still work with QTCaptureDevice.

QTCaptureView, you've already seen. And all you really need to do is say preview this Capture Session and it will do it. And you can set various attributes of its visual appearance, such as the background color that it fills with, and whether or not it preserves the aspect ratio of what's being captured.

And there are a bunch of other new classes in this API that we don't really have the time to go over in detail. But these are if you really want to get at the sample level of information and control various media work flows and look at the data on a sample by sample basis. There's -- we also have a class that let's you look at the detailed format of those samples. So those are worth looking at.

And definitely if you want more information, this is all fully documented. And this includes both the new editions that Tim talked about, and also the new editions for the capture API. So I definitely recommend -- check out the reference documentation. We also have a really good programming guide that will take you through some of the concepts of building these applications more in depth. And a ton of good sample codes.

So those two samples that I showed you today, and also we have the QT recorder sample, which shows you some interesting things I didn't show you today; like how to build a menu for how to select different devices. So you should definitely go and check that out. You can get all that documentation sample code off of the WWDC attendee web site, and so I encourage you to take a look at it.

Finally, if you have more questions or if you have a specific thing that you want to work with us on we have the QTKit lab tomorrow, and that should be tomorrow morning, I believe. So definitely stop by with any questions or concerns or feedback. We'd love to hear it.