My name is Tim Monroe, and I'm a QuickTime engineer, and I will be joined about halfway through by David Underwood. And this is the first of four sessions that I think should be of interest to you today. This is an introductory session talking about QtKit and our media APIs. After this, session 7.12 in this room is a more advanced look at QtKit, and that's where we'll talk about QuickTime 10. So if you have issues about QuickTime 10 or questions and want to see how to work with it, come to that session. I've also highlighted here a noon talk by Dr. Michael Johnson from Pixar Studios. He was one of the very early adopters of QTKit, and I don't know if he's going to be talking about it in this particular talk at noon, but he's a great speaker, and if you haven't heard him talk, definitely go to that noontime talk. And then later today, we have a lab in the Graphics and Media Lab starting at 2 and ending whenever anyone's done with a question. So I highly encourage you to go to the following session, the noontime talk, and if you have other questions, come to the lab.

So what are we going to talk about today? QtKit, as you probably know, is a Cocoa framework that is designed for working with media files and media generating devices. That's just a fancy way of saying that we can play back existing movies and we can also capture movies from devices attached to your computer. Now to say that it's a Cocoa framework means that it exposes a set of Objective-C classes. And we do all the nice Cocoa goodness with notifications and delegate methods and all that. So if you're experienced with Cocoa, this will have the kind of API flavor that you're used to.

QtKit, as you've heard in several of the State of the Union sessions, is now the preferred way to handle media or media capture on Mac OS X. It really permeates all of Snow Leopard. Anywhere in Snow Leopard that you see a movie being played back or a thumbnail of a movie, chances are very good that QtKit is being used to generate that playback or that thumbnail. Similarly, anywhere that you take a picture or capture a movie in the operating system or the Apple-supplied application, again, chances are very good that QtKit is what's doing the work under the hood.

It's also widely used in a lot of third-party apps and some in-house applications. Again, Pixar has been using QtKit for a number of years, and it's become part of their standard workflow. So all of their recent movies have been built with a process that crucially involves QtKit applications.

So what are we gonna talk about in this session? First I'm gonna give an overview of Qt Kits playback and editing classes. Then I will actually show you here on stage how to build a simple movie playback application from scratch. It really is, you know, 20 lines of code, not even that.

I'll talk a little bit about how QtKit integrates nicely with other graphics technologies on the Mac OS X platform, in particular core image and core animation. Then we'll shift our attention to talk about capture. David will come up and talk about the various capture-related classes, and again, he will build a simple capture application. So let's start talking about QtKit.

These are the basic classes that you will use for movie playback. As you know, QtKit is a very thin wrapper on top of the QuickTime framework. And again, as you probably know, QuickTime works with things called movies. A movie is composed of various tracks, and tracks are associated with things called media. So we have three classes in QtKit, namely QtMovie, QtTrack, and QtMedia, which are object-oriented wrappers on those concepts. And as you can also see, I've listed QtMovieView, which is a subclass of NSView, which is used to display and control a movie.

Here's just some random Qt movie methods that I've selected. One thing you need to do if you want to play back a movie is open it. And so we have the initWithUrl method. You pass in a URL and we'll pass you back a Qt movie object. We also have initWithFile which I'll use in a little bit and there are several other init methods that are of interest.

If you want to set an attribute, change a property of a Qt movie object, you will use the set attribute for key method. You pass in the desired value and the key and the attribute will be changed. There are also some basic control, movie control methods. So play, if you want to change the time, we have go to beginning, go to end, or you can specify a certain time that you want to go to. And finally, a very popular set of APIs is those that give you images of the movie. So here I've listed current frame image. We'll pass you back an NS image of the movie at whatever time it's at.

Qt movie view, as I said, is the subclass of NSView that you use to display and control a movie in the UI. Well, once you have a Qt movie view, the basic thing you want to do on it is set the movie. So you'll set movie and pass in a Qt movie, and then that will be the movie that's displayed in the movie view.

If you have a movie view and you want to find out what movie is associated with it, just call the movie method and that'll tell you what the current movie is. There are also a number of IB action methods that allow you to edit the movie, and here I would have cut, copy, paste. And lastly, as you know, QuickTime can put that little controller bar underneath the movie that allows the user to scrub in the movie or pause it or adjust the volume. If you want to hide that, you can use the setController visible method.

Now aside from the various classes that QTKit exposes, it also defines some data structures. In particular, we use data structures and not objects to work with movie times. The top one here is QTTime, very simple structure, consists of three fields. The one to start with is the time scale field. That essentially is the number of units per second that you want to work with when you're talking about time. The time value field is the number of those things in this duration or this point in time.

And QT time range, as you can see, is just two QT time structures, one of which indicates the start time of the range, and the other which indicates the duration of the range. We have a large number of functions, not methods, that you can use to operate on QT times and QT time ranges. One that you'll call all the time is QT make time.

You pass in the desired time value and the time scale you want to use, and it will pass you back a QT time. If you want to increment two QT time structures, we have the QT time increment. There's also QT time decrement, which you can use to subtract two QT times. If you have a QT time and you want to rescale it, you want to change the time scale field, we can do that for you because it's a little bit tricky. We have the QT make time scaled function. And finally, if you want to make a time range, pass us in the two QT times and we'll pass you back a QT time range.

So that's a good intro to what QtKit has to offer in terms of playback. Let's actually see how easy it is to build a playback application. So if I could go to the demo machine, please. Could I have demo one please? Here we go. Alright, so I'm just gonna launch, not dash code, but Xcode.

So here's Xcode and I'm just going to create a new project. And I want to build a Cocoa document-based application so that I can open one or more movies and display them in Windows on the screen. And I'll just call this WWDC demo. And here's our basic project file. Now the first thing you want to do is add QtKit to your linked frameworks. So I'll come down here and say add an existing framework. And then I'll look down here in my list of frameworks for QtKit, there it is, and add that to my project.

Now the next thing you want to do before you forget is set the kinds of files you want your application to be able to open. So I will select the target and do get information and then under properties, I can set the kind of extensions I want it to open. In this case, I'll let it open mov files, which are of type mov. And I want to add one more type. Let's call it, let's open m4v files, and I don't know their OS types, so I'll just say mpeg, which may or may not be right.

we're good there. Okay, so let's look at our classes. Instead of importing Qt or Cocoa, we can import QtKit. and qtkit.h. And I'm going to have one instance variable in this sample. It'll be of type QTMovieView, and let's just call it MovieView. Okay, now let's set up the user interface by finding our nib file.

So this is our default document layout. Let's get rid of this text box here. And over here in the library, we have this Qt movie view, QuickTime movie view. So we'll drag that over into our window, make it a little bigger. Now, if all goes well, I should be able to just control click here, and there it is, movie view, because I tagged it in my header file as an IB outlet. And I think I'm done there.

Now all we need to do is add some code to the.m file to find a file and open it and set it in that view. So let's come down here to movie controller didload nib. And we want to do the following. If the user has selected a file, so then self file name will have a value.

So if there is a file name here, I want to open that file. So I'll create a Qt movie. And I'll do movie with file. And I'll just pass in that same file name. And error, I won't bother passing in any error because I trust QtKit not to fail.

And what do I wanna do? Then I wanna set that movie on the movie view. Set movie and that will, whoops, I need a, So I've got a movie, I'll set it on the view. And this should work. Let's try building this. Build succeeded. Let's try running it. Let's open a movie. Go to the desktop. I have a movie there.

and let's see if it plays. So that was pretty easy. With that little bit of code, we're able to go out, select a movie file from the desktop or from anywhere and have it play back here. Now let's look at the edit menu. All the good editing stuff is disabled. We can fix that very easily with one nice line of code and we can do that like this. Let us change one of the properties of the movie. So we'll use our set attribute and I want that value to be in this number, Number with bool. And I want the value to be yes. And the attribute that I wanted to change is the QtMovieEditable attribute. And let's see if that builds. Yep, now let's run that. Now let's try opening that same file.

And here, my Edit menu is enabled. So we could go in here and select a portion of it, come up and cut it. Maybe that goes better here at the end of the movie, so we'll come over there and paste it in. Our director says that sucks, so let's undo that. We'll undo the paste, undo the original cut. So you see we have basically unlimited stack of undo's. You can do a lot of stuff and then undo it. So let's quit that. And if I could go back to slides, please.

So this is the code that we just built. We selected a file, we called movie with file to open that file and get a Qt movie associated with it. We set the attribute, its editability to be on, and then we set that movie into the view. It's really very simple, and we could open anything that was a.mov or an m4v, and it would play back fine.

Now what if we had a movie open and we want to export it or create a movie that has a different format? So I've got some movies, but I want to make them playable on my iPod, for instance. To do that, we want to export the movie in a different format. And it's really very simple. We would use the QtMovie write to file with attributes method. And the idea is we will set up a dictionary that indicates the desired operations we want QtKit to perform on it. And if you look at the dictionary here, I'm saying, yes, I want you to export it. And the export type is 3GPP. So if I were to run this code on one of the movies that I opened, I would get a movie whose size was appropriate for playback on some of the devices we have. and the audio would be transcoded into AAC that would play back nicely on your devices.

I mentioned getting a movie frame image with the current frame image method. Slightly more general is the frame image at time method, which allows you to specify an arbitrary time in the movie that you want to take the image from. And you'll notice here that it returns an NS image. Turns out that a lot of people didn't want an NS image. They wanted other kinds of images. So in Leopard, we added a new method called frame image at time with attributes error, which allows you to, again, specify a dictionary of attributes that tells us what kind of image you want and some of the properties of that image. So let's look at what could go into that dictionary. You can specify the size of the image you want returned. That's a very nice feature. The Qt Movie Frame Image Type tells us what kind of image you want, and on the next slide I'll indicate the various kinds of images you can ask for. And the last three attributes are some Booleans that say whether you want the image to be deinterlaced, high quality, and single field.

So here are the five available types of images you can request. You can get back an NSImage if you want. You can also get back a CGImageRef, a CIImage, a CVPixelBuffer, or an OpenGLTexture. So here is a very simple snippet showing how, with an open movie, I can get a CG image ref at a specific time. And in this case, as you can see, I'm asking for the image at the current time.

So as you can see, basic playback and editing with QtKit is really very simple. Like all Cocoa frameworks, it's designed to make the easy things easy. One nice thing about QtKit is that it plays well with others. It is able to interact nicely with core animation and core image. To interact with core animation, we define two classes that allow you to do things in layers.

We have a QtMovie layer where you will have a QtMovie and then say, I want you to play it back in this layer. And then that layer could be part of a core animation presentation that you have. Also, if you're doing capture through a device attached to the machine, you can preview what is being captured in a QT capture layer. We also, because we integrate nicely with core animation, make it very easy for you to apply core image filters to the output in a movie view. And so let's take a quick look at that, if I could go back to the demo machine.

Let me just open this existing project. And the basic idea here is very simple. Let me open my nib file. And here's my user interface. It's got a button to start and stop the movie, and it's got a pop-up menu where I'll be able to pick a core image filter.

Now, this is a Qt Movie View right in the center there. And let's take a look at this particular panel. Notice that the Movie View has a checkbox next to the Once Core Animation layer. So this is telling the underlying frameworks that when it creates this Movie View, it wants there to be a layer that is associated with that, what's called a layer-backed view. And since this view will now have a layer in it, we can do nice things. So let's quit that and actually look at the code here.

Scroll down a little bit. And it's actually very simple. Let's suppose that we have an array with a bunch of filters in it. We can then set those filters on the layer associated with the movie view. And it's really that simple. So let's run this and see what sorts of things we can do. We'll open a movie.

Actually, let's just play this through all the way, because this, well, let's not. Let's start it playing. And I can come over here and select a filter. Here is a posterized, we can do some dot screens. So we can take any one of the available CI filters and apply it to our movie view. Okay, so it's really just that simple. If I could go back to slides.

So let me just close my portion of it. You've seen that it's very, very simple to open a movie, stick it in a movie view, and let the user work with it from there. The two classes, QtMovie and QtMovieView, do 99% of what you want if you want simple movie playback. And as we saw, QtKit is a good citizen in the whole graphics architecture. We vend layer classes, and we're able to apply filters to our view in very straightforward way. So with that, I'd like to bring up David Underwood, who will talk to us about the capture capabilities in QtKit.

Thank you, Tim. So at the high level, what Qt Kit Capture enables you to do is it enables your application to grab media from some sort of real-time media source. And we have a couple of examples of that here. So a really good example is the EyeSight, which is a camera that we have built into almost all the computers that we sell. But there are other sources you can think of, too. So for instance, consumer and professional camcorders and tape decks and also any number of audio sources like microphones and different audio decks. And Qt Kit Capture lets you take the media that you get from those various sources and apply it to various destinations. So a good example is a QuickTime movie. Maybe you want to record it to disk. You also probably want to preview what you're capturing. So you want to have an audio and a video preview. And in addition, you want to capture to -- maybe your application needs to do some special processing on the samples you're capturing. So Qt Get Capture will also give you raw decompressed audio and also video samples that you can use for any additional processing that you need.

So when we talk about these sources and destinations, we can kind of visualize them as inputs and outputs. And we just have a conceptual diagram of this here. This is just an example use case involving inputs and outputs. So say you have a camera and a microphone attached to your computer somehow, and you want to capture that, and you want to send it to, well, say a window on the screen. Maybe you want to preview the audio through the speakers on your computer, and you want to record it to a QuickTime movie file. So QT Kit Capture represents this concept of inputs and outputs directly using classes that are defined in the framework. So your inputs are represented by the QT Capture input class, and your outputs are represented by, logically, the QT Capture output class. And then we also have this thing in the middle here, which we call a QT Capture session. And this class is very important.

Effectively, what this does is this is the traffic manager for everything that's going on. It's responsible for getting that media data from all of the inputs and then distributing it out output as each output needs it. In this example, you see we have both an audio and a video stream coming in on the input. But one of the outputs, which is just to a window on the screen, it only needs video. So the QT capture session knows that and just gives it the video stream. The speakers only want audio. So likewise, the capture session gives it audio. And then finally, we have our movie file. And that wants all of the media. So the capture session knows to distribute both bits of media to that. Now, QT Capture Input and QT Capture Output are just abstract classes. They define a general interface for how you would hook these things up. QT Kit Capture provides a number of actual concrete classes that are subclasses of QT Capture Input and QT Capture Output that you'll use in your application. So for QT Capture Input, we have the most obvious case, which is we provide a QT Capture Device Input, and this will allow you to use any device that's attached or built into the computer as a media source. We also, for outputs, we provide all sorts of things to you. So, one of them is if you want to write to a QuickTime movie file, you would use Qt Capture Movie File output. If you need to preview your video or your audio, we have a few preview outputs that you can use to do that. And finally, we have outputs that give you raw data, so the decompressed video output and the audio data output.

So if you take a look at these, depending on what your application needs to do, you can use these concrete input and output classes, basically like building blocks. So you assemble them together to create the use case that your application is interested in. So in this one example that I'm gonna show you, say we take just these classes that I've highlighted here. So the device input, and say you want to record to a movie file and preview what you're doing. So you'd use the movie file output and these two preview outputs. And going back to one of these flow chart diagrams again, It would look something like this. So say you have a camera and a microphone in your computer. You would use Qt Capture Device input for each of those, plug those into the capture session, which controls the traffic coming out of them. And then your output destinations would be a decompressed video output, an audio preview output, and a movie file output.

So to show you an example of how to do this in code, I'm gonna go ahead and build an application for you on stage. It'll be a very simple movie recording application. You can actually see a more complete version of this sample on our developer page. It's been up for a while now. So if you wanna follow along, you can open it now, or if you wanna review it later, it's available at this URL. And this application basically does one thing. It captures media and records it to a QuickTime file.

So before I dive into code, I'm just going to go over some basic steps, tell you exactly what you might need to do when you build one of these capture applications. And even though we're just doing recording here, these steps really will apply to any QtKit capture application that you might want to build.

And really, at the highest level, these steps are very simple. You're going to build your capture session, so get your outputs and your inputs in your session, and then you're going to use the capture session. Sounds easy, right? So going to a little more detail, one of the very first things you're gonna need to do in building a capture session is you're gonna want to find a device to record from. The way you do this is you use the QT capture device class, and this is a class whose instances represent all of the devices that are attached to or built into the user's computer.

And once you've found the device that you want, and there are a lot of ways to do that, and I'll show you one of them, you need to open the device to capture it. And what that means is you're effectively telling the underlying device system, I'm about to grab data from this device, so allocate any resources you need to do this and make sure that this device is as available as it can be for the application. And this is very important because devices on the system are a shared resource. Many applications can access them at the same time. And so the application needs to be explicit to the device system about when it's using the device so that it doesn't hog the resource unnecessarily.

So once you've found your device, now you're ready to build your QT capture session itself. And the way you do that is you create instances of subclasses of QT capture input and QT capture output. So in the example I'm about to show you, this will be pretty simple. We're just going to use QT capture device input and QT capture movie file output. So we'll capture from a device and output to a movie file.

Once you have those instances, you're gonna add them to your Qt capture session, which is again gonna do all the traffic. And finally, we have another class called a Qt capture view, which is somewhat similar to a Qt movie view that Tim showed you. It's just going to do all of the work of automatically previewing the video going through your capture session. And you're gonna interact with this class and code very little. Most of it's just gonna be an interface builder, and I'll show you that in a second.

So now you have your Qt capture session. Now you actually wanna use it. Your application's supposed to do something with it. So first you tell it to start running. And what this does is this tells it to start pulling media data out of those inputs that are hooked up to it and to start distributing it out to its outputs. And it'll keep on doing that until you tell it to stop. Then we're gonna go and talk to the movie file output that we added to the session. And since this is the class that's responsible for actually recording the file, we're gonna tell it to start recording and then tell it to stop recording sometime later, depending on what the user does. And then finally, we need to implement a delegate method on the movie file output that will get called back when the file's actually finished writing, and I'll show you how that works concretely in a second. So if we could go to the demo machine, please.

All right, I'm just gonna close this off. So we're gonna build this roughly from scratch. I'm gonna cheat a little more than Tim did and I'll have some snippets of code here that I already have written. So we're gonna go into Xcode and create a new project. And this is gonna be a very simple, single-windowed application. So I'm just gonna make a new Cocoa application. And let's call it My Recorder. And just as before, the very first thing that we need to do is add the QtKit framework to our project.

Now we're good there. And because this is a relatively simple application, really we just have a user interface and the actual capture objects in QtKit. So the model objects, QtCaptureSession and the inputs and outputs. So we really just need one controller class that ties it all together. And we're going to create that here now by making a new file. and we're gonna define a new Objective-C class. And let's call it MyRecorderController.

And let's just go ahead and define what we'll see in the header files. Look at the general interface of this class. I'm just gonna get rid of that template stuff. Drag what we need in the header file. So you see there's not a whole lot to it. One thing we have in here is we have an interface builder outlet to a QT capture view, and we're going to need that so we can associate the capture view with the session in code. You notice we also have a few instance variables for those QTKit capture objects that we're going to create. So we have a QT capture session object, a movie file output, and a device input. Finally, this controller class is defining a couple of action methods that we're going to hook up to buttons in our user interface and this will be what the user uses to actually tell us to start recording and to stop recording. So I'm just gonna save that. And now that we have that header file defined, let's go ahead and build the user interface, the application. So let's open our nib file, And right here in the interface builder library, right next to Qt movie view, you see we also have a Qt capture view. And just as before, we'll drag that out. Nicely. And we're also gonna add a couple of buttons to this interface, and those are gonna let the user control recording. So let's have a stop button and a start button, and we'll line those up.

And finally, I'm just going to set some resizing attributes in Interface Builder so that the window resizes nicely and the controls are laid out. So I'm going to pin these two buttons in the lower right-hand corner, and I'm going to make it so the capture view resizes nicely with the window also.

Finally, we need an instance of our controller class to hook up the user interface to the capture. But luckily we can do that right in the nib file. And the way we do that is we get an object from the interface builder library and drag that into the nib file.

And that object needs to be an instance of our controller class. So we go to the custom class pane of the inspector and we say that this is an instance of my recorder controller. And you see interface builder automatically found all the outlets and actions. And so we'll hook this up to our capture view. There we go. And we'll hook up the start and stop buttons.

we're good to go, so let's write some code. So we'll open up the implementation file from iRecorderController. So if you recall, I outlined two basic steps for working with QtK Capture, which is to set up your capture session and then run it and do something with it. So a logical place to do the setup, since our controller class is being instantiated inside of a nib file, is inside of the awake from nib method.

And this method is called for any class that's instantiated inside of a nib file, so it'll just get called when the nib is first loaded. and we'll start doing some setup. So again, the very first thing we're gonna wanna do is create our QT capture session. And that's just simply a matter of doing QT capture session alloc init. It's all you need to do. Then we need to find a device. So we've simplified this a little bit over the sample. It's available online. And right now we're just going to record video. So I'm going to look for a video device.

And one of the ways to do that is to ask the system, what's the default video input device on the system? What's the most logical one that could be used by default? So for example, on the iMac or a laptop, that would be the built-in eyesight. And the way that we get that is with the class method, default input device with media type. And the media type you pass here is what kind of media the device is going to give you. So in this case, we want video.

Now that we've found our device, we need to open it, again, to tell the device system I'm about to capture from this device, allocate any resources that you need to do that. And we do that with the QGCaptureDevice open method on that instance. This method also returns an NSError optionally if you want to find out if it failed for some reason, and it returns a Boolean value just if you need to check the failure case. So one very important thing, it's the reason why it's highlighted in this code, often in these demos we leave out error handling, but your application absolutely must check to see if open succeeded. And the reason for that is that there is always a rational reason that the open method can fail. So for example, say the user had a camera plugged into the computer, and you went and called this method while that camera was plugged in and it returned the QT capture device for that camera. And then between this method finishing and you calling open, the user, you know, they like to try to break your application. They go and yank the camera out of the system. Well, the QT capture device instance, the object is still valid, but the device is gone. So if you try to open it on the device system, it's gonna fail.

And so your application always needs to be able to deal with that case. So in this case, it's not all that elegant, but basically we say, okay, this isn't useful anymore, so we'll mill it out. Probably you would wanna display an error message or something similar to that.

So now that we've found our device, we can create our inputs and our outputs and add them to the capture session. So the first input we'll create is a QtCaptureDevice input, and we just create that around the device that we just found using the initWithDevice method. Thank you. Once we've done that, Qt capture session has an add input method, and so we take that input that we just created and add it to the session, and this also returns an error that you can optionally check.

For our output case, we create a QT Capture movie file output which will be responsible for writing the data to the file. And again, we just initialize that by saying alloc init and adding it as an output using the add output method to the QT Capture session. One additional thing you need to do with the movie file output is set a delegate on it. In this case, we'll just set ourselves as the delegate. And the reason for this is that QT Capture movie file output will call back the delegate with actually all sorts of information.

It exposes a lot of delegate methods. So for example, it tells the delegate when it's started to write to the file. It even actually has a delegate method that's called for every single media sample that comes into the movie file output. So if you wanna control really precisely when to start and stop recording, you can do that. But in this case, we only care about one thing that's very important, which is we wanna find out when the file is finished writing. And I'll show you in a second why that has to be done with the delegate method when I show you the delegate method.

Just sort of to write sane amounts of data to the disk, what we also want to do is compress the media that we're writing through. And the way that we do that is using this Qt compression options API, which is a very simple API in Qt Get Capture for choosing what sorts of options for compression you might want to use for a piece of media. And basically it just exposes a bunch of compression presets. So for example, in this case we're gonna create a Qt compression options object with just this preset, which is we just want to do kind of small H.264 video. So we don't really care about it being big. We just want to record something, have it not take up very much space. And it will compress that live as you're recording to disk.

And when you get that QT compression options object, you're going to apply it to the movie file output. And this goes into a little more detail than I'm gonna go into in this session, but you have to do it for these QT capture connection objects. And if you're really curious to know what these objects are for, what their significance is, stick around for the next session, the advanced session, we'll go into more detail. But I'll just suffice it to say here, this is how you would apply these compression options to this movie file output.

Finally, we want to preview the video we're recording. So we had an outlet and interface builder to our capture view. And we're just going to say, tell the capture view, this is your capture session, which was the capture session we created at the beginning. And finally, we have everything set up. So we tell the capture session, start running, start grabbing media from the camera and sending it out to where you need to send it.

So our capture session has been built. And whenever you set things up, you also need to clean them up, always important. And we do this in two ways. So first, actually I forgot to do this in the nib file, so I'm gonna do this pretty quickly. we're going to make our controller the delegate of our applications window.

And what this enables us to do is, when the window's about to close, it's gonna call this method. And this is a good opportunity to clean up some of the really critical shared resources that our application is using. So most importantly, this is our opportunity to close that Qt Capture device that we opened earlier. And this is really important because again, we don't wanna keep it open for longer than we need to, so the window closing is a good time 'cause the user obviously doesn't need it anymore. In addition, your application should basically always close a Qt capture device that it opens. So wherever you call open, you need a matched call to close elsewhere in your application when you're done with it. This is also a good time to tell the capture session to stop running because that also just uses a lot of CPU and bus bandwidth and we don't need it anymore 'cause the window's gonna close. Finally, in our dealloc method, this is where we just go and do the standard Cocoa releasing of the objects that we've created.

So we have our setup code and we have our teardown code. Let's actually use this thing. And that turns out to be pretty simple in this case. Here are two interface builder actions. The start recording action simply tells the movie file output, record to the file at this URL, and we just have a hard coded path in this case, but you could obviously bring up the user interface to make it more complex. And then the stop recording method says, okay, just don't record to anything anymore. You pass nil.

And an important thing to note about this particular method is when you call this, it will effectively return immediately. So even if the file isn't finished writing yet, this method's still gonna return. And the reason for this is that Qticket capture is very heavily asynchronous and very heavily multi-threaded. And sometimes, for example, with a QuickTime movie file, even when you've run out of data that you need to write to the file, so media data, there's some extra stuff you need in that file so that it'll open correctly as a QuickTime file. So for example, QuickTime movies need a sample table so that they know where the media samples are and what times they correspond to so the user can play and scrub efficiently. And that might take some time to write out, but we don't wanna block the main user interface while that's happening. So even if the file isn't actually finished yet, this is gonna return immediately. But your application most likely wants to do something with that file after it's been finished. So you need to find out when it really was finished so that that file can be open. And the way that you do that, if you recall, we set ourselves as a delegate on the movie file output before, is you implement this delegate method, just capture output, did finish recording to output file URL for connections due to error.

So it's a bit of a mouthful, but in this case, we don't have to do a whole lot with it. It's pretty simple. We're just gonna tell the shared NS workspace, open up this file, which we'll just launch it in QuickTime Player, 'cause we know it's finished and we know it can be played, so we'll just open it in a player application. So we'll save that and build it. And that should be everything we need. Oh, no, no, it's not. Ah, I know what I forgot to do. Just as it's very important to include QtKit in your project, you also need to bring in the header. There we go.

And here's our application. And you see, we have our Qt capture view up here and it's previewing what we have and our faithful test subject here for all of our videos today is gonna be this little stuffed elephant. Yeah, say hello. And we'll just record a little movie with the elephant in it. So let's start this thing. And now it's recording the file and I don't know, we'll make him turn in a circle or something.

Sure, that's pretty good. And we'll hit stop. And there you go. The movie was recorded to a file and we automatically launched it in QuickTime Player. And there it is. And you'll notice it's a little cheesy here, seeing my hand here in the frame turning the elephant. That's not very convincing. He doesn't look like he's moving under his own power. We'll show you how to fix that problem in a second.

And just to show that our compression worked, you'll see that if we open the info window in QuickTime Player, you'll see that we are in fact using the H.264 decoder to decode this movie. So that's what it was written out to file as. So we got a pretty small file for this. So, you know, this is a tiny file for this quality of video. And that's a very simple capture application with the QtDK Capture API. So if I could go back to slides, please.

So that was just one example of the way that you can take the concrete classes that Qt Kit Capture gives you and assemble them together to meet a certain use case of the application. Well, say you have something more creative in mind than writing to disk. Say you have something more open-ended. Well, we can use a different set of classes and use those as building blocks. So again, we're just gonna kind of pick and choose out of what's given. And again, we'll use a Qt Capture device input. It's really our only choice. And we're gonna use this different class called the Qt Capture decompressed video output.

And effectively, what this class allows you to do is it allows you to get each individual frame of video that comes from the capture session as it's captured as quickly as possible, effectively. It tries to really give them to you. And if the device that you were capturing from happens to do video compression, so for example, it's a DV camcorder, that's compressed video, this output will decompress that video for you. So you'll always get raw pixels that you can use for various types of processing. And what this class, this class gives you the video in the form of the CV image buffer type, which is a common class that's exposed by core video that contains buffers of video. It's very appropriate for video buffers. And again, the flow chart here looks pretty similar to what you've seen before. On the input end, we have our device input. The capture session is distributing the traffic. And on the output end, we have our decompressed video output.

So there are all sorts of things you can do with this class, in fact, and just one fun example, and if you were here last year, you've probably seen this one, is stop motion animation. And if you're not familiar with what that is, basically it's the practice of taking some kind of scene, say you have clay characters or action figures or even, you know, bits of paper or even a 2D image, and you just photograph still images of it, and then you assemble those together to to form an animation. So you just change the scene a little bit every time and you put those photographs together and it looks like it's moving.

It's a very traditional way of doing animation. So what I'm gonna show you now is a brief code overview of how we would do this using QtKit Capture. And this class or this application is going to use the QtCapture decompress video output class to actually get those raw frames in order to assemble them into a movie. This application is also gonna use some of the QtMovie API, some of which Tim showed you, to actually assemble those frames together so that we can save that movie and play it back. So this will kind of bring everything together. Incidentally, this is also a sample that's available on the developer website. It's called Stillmotion. So if you wanna follow along, I'm just gonna be walking through the code on that one. And you can also play around with it later. So if we could go back to the demo machine, please.

Thanks. Okay, so I'll just close this off. And I'm not gonna build this one completely from scratch. I'll walk through most of the code. And the reason for that is 'cause you've, a lot of the basic steps are exactly the same. I don't need to show them to you again. I'll show you where they're the same, but not so much different. So, just gonna open this project, and this is the same one that's available on the developer page.

And the first thing we'll look at actually is the user interface. So gonna open up our new file. And unlike the previous application that I built, this one is document based. So you'll actually be able to have multiple animations going at the same time and save them out to separate documents. And in our user interface, it's not that much more complex. On the left side here, we have again, our Qt capture view. And this is gonna show us what's happening live in front of the camera right now. But then on the right side, we have a Qt movie view. And that's gonna show us the actual movie that we've assembled so far in our application. Finally, we have a button here, and this button's gonna be responsible for taking whatever's currently in the scene and appending it over into the movie so we can build our animation. So pretty simple.

So let's take a quick look at the actual implementation. And again, this method is called, it's similar to Awake from Nib, it's called in a document-based application whenever the Nib file is actually loaded and the user interface is about to come up. And you'll notice again, we do very similar things with Qt Kit Capture. We create a capture session, we get our default Qt Capture device and we open it and very importantly, check whether that succeeded or not. And in this case, we're a little friendlier actually, we bring up an alert if that failed for some reason, so it's a little more helpful to the user. We make our device input, we add it to the session.

And the new thing here is we create our decompressed video output instead of the file output since we're not writing directly to file. And this is gonna do the meat of the work of our application. But similarly to the movie file output, we're gonna set a delegate on this class. And in this case, the delegate is going to get called back every single time the output receives a new video frame. So this is what you're gonna use to actually grab those raw frames. And we add that as an output to the session.

Hook up the capture view, same thing, and we start the session running. So this should look very, very similar to what I did before, sort of swapping out parts basically in our setup for a different type of use case. What you might have also seen up here is a little bit of extra work, which is we're actually going to create a Qt movie. And this is the movie object that we're gonna write into to assemble our animation. And we have a special initializer here, which defines some data in memory that we can write into so the movie knows where to store those samples as we add them to it.

Again, we also have basically the same cleanup code. Very importantly, you need to close the device if you've opened it, and we want to stop the capture session from running because there's no reason to be using those resources. Thank you. This code is responsible for reading and writing files. It's basic NS document stuff, but not too important for showing you exactly what you need to do.

And so now you have a capture session and you're running it and here's where we start using it. And this is a little more complex than what I showed you in the simple recording application. But again, it's actually not so complex. So the main thing you want to do is you need to implement this method. And this is the delegate method that gets called by the decompressed video output whenever it gets a video frame. It's called capture output, get output video frame with sample buffer from connection. Okay. And the parameter to this that we're interested in is this CV image buffer ref that it's giving us that's already all packaged up.

And all we're gonna do with it is we have an instance variable defined for this class called current image buffer. And we're just gonna store the video frame that was given to us in this instance variable. There's one additional little bit of complexity here, which is that this method will almost definitely not be called on your application's main thread. And again, the reason for this is that Qticket Capture tries to do everything very, very asynchronously and is very heavily multi-threaded and takes a lot of advantage of multiple processors. So it's not going to block up your main thread by default, calling you back here. So the way that you need to deal with that is you just have to be careful and make sure that if two threads are accessing the same instance variable, which is the case here, you need to lock around that instance variable. And we do this a very simple way here, which is we use the Objective-C @Synchronized block in order to protect that segment of code. You could also use NSLock or Pthread mutexes or anything that you think would be more efficient for your application, but here's just a simple example.

And again, since we're just capturing the current state of the scene to add a frame to our animation, we're really only interested in the latest image buffer that's come in. So there'll be image buffers coming in constantly all the time, but if the user doesn't click that add frame button, we don't really care about the old ones, we just throw them out. And that's the reason why we just store this latest one in this instance variable.

So now that we have that going, we have the actual interface builder action called add frame. the very first thing we do is we retrieve whatever that latest image buffer was. And again, because this is going to be on your main thread instead of the different thread that QtKit Capture is calling you back on, you just need to be a little bit careful and put the synchronized block around here so that you don't get the, you don't read back from that variable when it's in an inconsistent state.

And now that you have an image buffer, you're going to convert it to an NSImage and then use this QtMovie API, which is, QtMovie actually has many APIs and encourage you to look at it, which is called addImageForDurationWithAttributes. And this is a method that takes an NSImage, in this case, the one that we've just created, and you specify how long it's gonna last in the movie. And then maybe you can specify how it'll be compressed. So in this case, again, just save a little bit of space. We're gonna JPEG compression, excuse me. And that's pretty much all there is to it. So we've gotten our CV image buffer refs live from the camera, and we're just gonna add them to the movie. And let's go ahead and build that.

Okay, and let's run this. And so again, we see on the left, we have our live video preview. And on the right, we have the movie that we're gonna build. So you remember before I said, we had a bit of a problem. My movie was really cheesy 'cause my hand was in it. You could tell that the elephant wasn't moving under his own power. We're gonna fix that. So maybe he'll start turning around. Turn around a little bit more. And something about the camera is really making the elephant angry, so he's going to stampede towards it.

Okay, anyway, I could do this for hours. But what you can see here is we have a pretty cool animated movie. I mean, you could really go to town with this, just with a pretty simple piece of sample code, just bringing these few classes together and doing something interesting with them. You can actually write a pretty entertaining application. And because this is just a QtMovie object, we can go ahead and save this file out. And we'll just call it elephant. That needs to be spelled right.

And you'll see we just have a movie now that can be opened and played anywhere. There it is. So we'll just open that up and quick look. There we go. See, and we have our QuickTime movie. So if we could go back to the slides, please. Thank you. So just to do a quick review of some of the APIs that you've seen kind of formally declared as they are on the headers, I'll just go through some of the APIs that you will almost always use if you're building a QtKitCapture application. So first we have QtCaptureSession, which is the class that you will always create because it's the class that needs to mediate between the inputs and the outputs. And the methods you will most commonly use add input and add output, and these methods optionally return an NSError if there was a problem doing it. Also, what you need to do with the session is, once you have everything set up, you need to start it running to tell it to actually grab that media data and send it to the outputs, and that's done with the start running method. And then when you're done with it, when you don't wanna use the bus and use all that bandwidth and that memory, you can say stop running after that.

When you're capturing from devices, which in this case will be every case, you will always interact with the QtCaptureDevice class, and this is the class whose instances, each instance of this class represents a single device on the system. In all the demos that I showed you, I showed you the very simple case of looking for a Qt capture device, which is to ask the system for its default input device of a certain media type. And I showed you video devices. You can also get the default audio device, for example, which is the device that the user has selected in the sound preference pane. But your application, lots of applications probably want to do something a little more complex than that. For example, maybe they want to have a pop-up menu that shows a list of devices from which the user can pick so they can actually decide which device connected to use. And Qt Capture Device exposes that very simply by using an NSArray. So there's this input devices with media type method that returns an array of all of the devices of that media type on your system.

There are various bits of information you can get about Qt Capture Device. A pretty simple example is you can get a localized name for it so you can display it in your UI. And finally, something you always need to do with a Qt Capture Device before you use it is you need to open it to tell the device system that you're using it. And you need to close it again when you're done with it.

QDCaptureDevice also exposes a bit of information about its current state on the system. So the class exposes two notifications that tell you when devices appear and go away, because the user can plug them in and unplug them at any time. And there are also a few methods on the device itself where you can find out if it's connected. And for certain devices, if an application is hogging it, if it can't be used by your application, it happens to be the case that the vast majority of devices, including all iSites and all DV camcorders, can be shared across multiple applications at the same time. So usually that's not gonna be a problem.

With Qt Capture Device, you're going to use the Qt Capture Device input class, and that's just as simple as initializing it with the Qt Capture Device, and that's pretty much all you need to do with it. On the output front, if you're recording to a QuickTime movie file, you'll use the QtCaptureMovieFileOutput class. This class is actually a concrete subclass of QtCaptureFileOutput, which is a subclass of QtCaptureOutput. So if you're poking through the headers and you open up QtCaptureMovieFileOutput.h and you see that it's empty, go look at QtCaptureFileOutput, which will actually show you all the methods that you need to call. And the methods that you almost always will use are recordToOutputFileURL, which specifies a recording destination. and you will definitely want to implement this delegate method that calls you back when the file is actually finished writing. This class actually has tons of other features, including features that let you record on exact frame and sample boundaries, and you should definitely check it out, it's a pretty powerful class.

Also, if you're interested in getting raw video frames, as I showed you earlier, you can use Qt Capture decompressed video output, and the key thing here is implementing this delegate method, which will call you every time a new video frame comes in, and will give you a CV image buffer with which you can do as you please.

Finally, I showed the use of QtCaptureView. This class is actually an NSView wrapper around the QtCaptureVideoPreviewOutput class, which I mentioned a little bit earlier. And almost always, if you're doing a Cocoa or an AppKit application, you're going to use QtCaptureView and never really use QtCaptureVideoPreviewOutput directly, but that class is available independent of a view in case you need to do some custom previewing. And also, in general, QtCaptureView is something you never create programmatically. you'll probably just drag it into your nib file and interface builder, so you generally don't need to interact with it that much. One thing that you do need to do though is tell it which capture session it's previewing, and you do that with the setCaptureSession method. There are also a few attributes it has that govern its appearance, such as the color that it fills in behind the video and whether or not it preserves the aspect of the ratio of the video as it's playing.

A quick note on what devices are supported by Qt Kit Capture, This is pretty much the complete list. We support the VDC over USB standard, and an example of those cameras are, well, the built-in iSight on all of our machines. There's IIDC over FireWire, and an example of that is the external iSight that we've shipped. And QtK Capture also supports any device that's supported by Core Audio for audio, as well as consumer DV and HDV cameras, and also pro DV and HDV cameras. In the context of professional things, there's a little asterisk there and the codecs that actually decode HDV and also some of the pro DV formats like DVC Pro, those codecs are only bundled with Final Cut Pro, so a default system won't be able to decode those formats. You'll still be able to capture the data with this API, but you won't be able to preview it or say recompress it. Finally, if you're familiar with the older sequence grabber API, Qt Kit Capture has a built-in bridge that also supports any devices that worked with Sequence Grabber, but aren't on the list of any of those devices above. So capture cards and other things that have VDIG components, if you're familiar with those.

So to summarize a little bit, and this pretty much parallels Tim's half of the talk, if you're doing real-time capture from devices in your application, use QtKit. It's the preferred API from here on out, and it really provides a lot of power without a lot of complexity. In order to do that, you pretty much will always rely on using QtCaptureSession, QtCaptureInput, and QtCaptureOutput, and you'll use those as building blocks to assemble what your application needs to do, depending on its needs.

If you want more information, first of all, definitely just stay put in this room. In the next session, we're gonna dive into a lot more detail about both the playback and editing aspects of QtKit, as well as the capture aspect. So we're gonna go into some more advanced topics. And we're also gonna talk a bit about QuickTime 10. And again, we strongly recommend that you check out that lunchtime session. It's sure to be very interesting. It's always a very good speaker.

Also, if you have any questions or any problems or specific concerns, you just want to chat, go meet us up in the lab later today, and we'll be around and we'll be there to answer any questions that you might have. And also, if you want to get more information, first talk to Allan Schaffer. He knows everything about everything or knows people who know things about everything. So he's the one to talk to. And also, we have tons of documentation and sample code and other things on the developer page. So just surf through there, and you'll see a lot more examples of all of these things.