Before we get started, have a look up here for some first instructions to get started. What we are going to do during the hands-on session is actually set-up some video conferences with each other so that we can try out iChat Theater as we build the project. So find somebody near by and set up a computer-to-computer Bonjour network. Once you have that set up, when you log into Bonjour you should be able to see the other person there and set up a video chat. I'll give you guys just a minute to do that.

Okay. Let's get started. This is session 202, Sharing Your Application's Content with iChat Theater. My name is Peter Westen. I'm an engineer on the iChat team. And we're going to talk about the demos that we saw during Steve's Keynote, where Phil was able to share content in his Keynote presentations or in iPhoto or even directly out of Finder in a middle of a video chat. So what's iChat Theater?

It begins with content in another application, whether it's Keynote or iPhoto or QuickLook running inside of Finder. And then when you have a running video chat with another buddy, iChat Theater allows that application to send video and audio through that video chat so that it can be shared with the person on the other side, who doesn't need to have the same software on their computer.

Example, clients would be iPhoto and Keynote, which we saw on Monday, QuickTime Player can, of course, also send movies directly into iChat Theater. Other applications could send 3D content into =a video chat and hopefully after today you're application's content as well. Before we get into the meat of things, there are a few things iChat Theater does not do.

First thing is screen sharing. There's no shared control. It's a presentation of the content you have on your computer with a buddy. So events, mouse clicks, any of that stuff does not come back into your application from the other side. It's not video chat in your application either.

Sorry, that's not what we are doing today. This is about getting the content in your applications and putting it into iChat. It also not Quick Look, although it works with Quick Look and that's something that was also shown on Monday. So before we go any further, let's take a brief tangent and talked about Quick Look does. It's a new technology in Leopard that allows small bundle to be written for your application;s document type in order to present a preview without actually having to launch your applications.

It's file based, which means you need to have a document on disk that can be used to generate a preview. If your application's content is stored in documents, then you might want to consider writing a Quick Look preview generator and leaving it at that, because iChat can use that preview generator to produce content that gets shared in the video chat.

But it does have limited interactivity. Quick Look only supports the most basic of operations, like a dancing forward or back player, pause. So if you have more complicated manipulations that you can do with your contents such as changing the perceptive in a 3D scene, then that's not going to work.

There's also limited fidelity. Keynote slides, for example, don't have any of the transitions that are supported in the Quick Look preview. It's the most bare bones simplification of the content that's in that document. But the biggest advantage is that there's no extra work if you've written a Quick Look preview generator for your document type, iChat can use that, the user can even drop the document directly onto a video chat in order to share it with iChat Theater. The session I believe is right after this one. It's 213 and it talks about how to write Quick Look preview generators.

So with that out of the way, let's talk about the iChat Theater API. It's a Cocoa API and it's divided into three major parts. The first is session control. Of course, we need to be able to start and stop these sessions, but more importantly your application needs to integrate those actions in the workflow of that application in the way that's most appropriate and that's the decision that's left up to you and your designers.

Video can be provided to iChat Theater in two different ways. Last year we talked about implemented low-level callbacks to provide video. We heard a lot of feedback that there should be an easier way so, this time around we've got support for NSViews, so you can just set an NSView as the source content for iChat Theater and that will get sent directly into the video conference.

And then audio, there's also two ways providing audio content into your video chat. The first is a very simple, it just uses NSSound, which can be configured to send its audio directly into iChat Theater and there are low-level Core Audio callbacks if you want more fine grain control. So first let's talk about session control.

The most important part of the entire API is the IMAVManager class and there's a single shared instance that you will do everything through. All you have to do is import the IMAVManager.h header into your code file and then ask for the sharedManager out of that class. And that object is going to be the one place where you get state information to find out what state the iChat Theater session is in as well ask it to start or stop and configure audio and video.

It's very important that you register for notifications. The iChat Theater session depends on iChat and so if iChat isn't running you won't be able to start a session. When it is running, you begin a session and there's some extra states that tell you when the conference is up and running and you should be providing frames of video to the conference.

So there are two important parts to registering for that notification. The first is, of course, the notification name itself. IMAVManagerStateChangedNotification, but also it's important that you register for this notification using the notifications center published by the IMService class. That's another part of the InstantMessage framework and the reason we do is that iChat Theater wants to be lazy about connecting to iChat and getting information about iChat Theater.

So subscribing to this notification is actually a hint to the framework that you're interested and if you do not subscribe to the notification, we won't know and we won't connect to iChat. So the state will always be not available. And then each time you are interested in the starting or stopping of iChat Theater, you'll generally want to get the state to find out what you're able to do and that's simple accessor to get the state.

So let's talks about those starts, there are six. The first is IMAVNotAvailable. When you launch your application, this is what the states going to be because it has not yet connected to iChat. Once you subscribe to that notification, we'll connect to iChat, get the state and it will then generally transition to Stopped.

That means that iChat Theater can be started. Now it's important to note that this is the state of the IMAVManager, not the state of the current iChat Theater session the user might be having. For instance, if another application is in the middle of sharing through iChat Theater, the state from the perspective of your application is going to be stopped, because if the user were to start from your application the other one would be cutoff and your application would then step in.

Once you start a session it will transition through StartingUp and from there to Pending and what this means your application is connected to iChat, we're ready to go, but the video conference itself either isn't running or the extra video channel for sending that video from your application hasn't been set up yet.

So it will be in the Pending state for some period of time and from there it will transition to IMAVRunning, which means that the session is active, the video frames that you're providing will be sent to the other user and then when the session shuts down it will go through ShuttingDown and from there back to Stopped and you can start over again.

So running a session only has a few steps. The first is to set a video data source. IChat Theater needs a way to get the video and this can either be an NSView or custom object you've written yourself. And that's simply done through these setVideoDataSource method. And then you have a couple of options, this is a bit mask and you can set two options right now. The first is called IMVideoOptimizationStills and what this is, is a hint to the encoder for the video chat that you're content will remain largely static.

And what that allows us to do is increase the size of the buffer used for the video chat so that image quality can be greater, because we know the bandwidth requirements will be lower, well, encode and send a video frame across the other side and then it won't use up very much bandwidth after that. So, more bandwidth can be devoted to video from the user's iSight.

It's important that you only set this option if you know that your video will remain static for periods of time, many seconds for a Slideshow or even an extended period of time if it's some other application, if you're looking at a picture or some other image. If you set this and you do have a, you know, video that's changing every single frame, the result is going to be pretty poor performance and the quality of the video won't be very good.

There's also another option. IMVideoOptimizationReplacement and what this does, is it forces iChat Theater to replace the local user's iSight video with your content instead of sending both side by side, and the result of that, of course, is that we can devote every bit of available CPU and bandwidth resources to that video that you've sent. For the most part, you won't bother sending this because the person's video is nice to have alongside the content, but if you have really important content that you want to have as high fidelity as possible, you may want to set this option.

If you don't set it though, it's not a guarantee that the video will be side by side, because if we're in a video chat with somebody running Tiger for example, it doesn't support iChat Theater, that's what we'll do automatically as a compatibility mode. Then once the options are set and you have a video data source, you begin the session by simply calling start and when you are done you call stop.

Now that's the basic part of session control, the hands on project that we've got is just going to talk about that part of the API. So we'll get started with that. Michael Estee also from the iChat team will be my buddy here during our video chats. And so if you got the iChat Theater Headstart project downloaded, go ahead and open that up.

Let's switch over to the demo machine. So there are two folders in there, Finish is the end state, we're going to want to be in and Start is our beginning state for this application. So the Slideshow project is pretty straight forward. First off, let's just see what this does.

It's a simple application which loads images out of your desktop Pictures folder and displays them in a Slideshow, but it doesn't do much else. It's not very interesting. So let's combine this with iChat Theater. So does everybody have this project loaded up on their computers? Raise your hand if you got that there and you'll be able to follow along. Great. You'll be able to get a decent amount even if you don't have this loaded up on your machine.

So we'll quit that and have a look at the code in this application. It's very simple in the starting case. All this does so far is have one method to start and stop the Slideshow in a custom view, which displays the images and it changes the image on the start and stop button accordingly. A command from the main menu of the application to toggle whether it's running or not and then a validateMenuItem method, which will enable and disable the start and stop Slideshow menu item.

And then an application delegate method to make the application terminate when you close the window. This class will be instantiated once in the main menu .nib and is the application's delegate, but right now it doesn't do anything else. So in order to make this work with iChat Theater, first thing we need to do is add the framework to the project. So open up your Frameworks group on the side and then in the Project menu, we're going to say Add to Project.

Right there. And in your Group Volumes System Library Frameworks folder, you'll find the InstantMessage framework. In addition to iChat Theater this framework also supports getting presence information about buddies from iChat. That session was yesterday. So go ahead and add that framework to your project and choose the target and then we'll go back to our main controller class and in here we'll go ahead and import the header.

Now, to keep things simple we've just put all of the code into the starter project we need and we're just going to remove comment lines so we can activate different portions of our implementations as we go. So just go ahead and remove the comment line surrounding the import.

And build to make sure that we've done everything we need to. Build succeeded, but it still doesn't do anything yet. So the next thing we need to do is subscribe to that notification, as I've mentioned before it's important that you do this because without that notification subscription the instant message framework will never get the hint that you're interested in iChat Theater.

So in the applicationDidFinishLaunching method just below, we'll remove those two comment lines and again check that we've subscribed to the right stateChangedNotification and that we'll also using the IMService notificationCenter, which is our custom center that we use in order to tell when people are subscribing and unsubscribing to these notifications.

So again, still we're not doing anything very interesting yet, we need to actually have a command to start and stop iChat Theater. So for that we'll go to the header file for the controller where we have the toggleSlideshow action method and right above it we'll have a new one we add called toggleTheater. Go ahead and remove those two comment lines. Save that file and now let's open MainMenu.nib and hook up a new action. So in the Resources group, MainMenu.nib is right there.

Now the new version of InterfaceBuilder you don't need to reimport headers if you create your main menu, open up the main menu here under File menu we'll add a new menu item. I'll do this just by duplicating Play Slideshow and we'll retitle that, Start iChat Theater, and then we're just going to control drag from this menu item to our controller object in order to hook up the action. All this is pretty elementary Cocoa development. And InterfaceBuilder has seen that we added a new action method to the header and we'll just connect that right there.

Go ahead and save that and back to our project and we'll make one more change to actually add the implementation of that method before we try things out. So if you scroll down a little ways, there's a section called Headstart Step 3 and this is our action method. So go ahead and remove those two comment lines and we'll talk for a little minute about what's in this method.

So the first thing we're going to do is grab the shared AVManager and then ask for its state. Since we are going to start and stop based on that state. If the state is stopped then we need to start it. And as we said, there are three steps you need to do.

You need to set your video data source, need to set your optimization options, and then you need to start the session. So that's what we've done here, we have an outlet to the slideShowView, a custom NSView subclass. We're going to set the VideoOtimizationStills option, because we're going to have an image that remains static for a very long time, so this particular video stream isn't going to require a lot of bandwidth and then we start the session. And as soon as that's started we'll also make the Slideshow itself begin. This is something Objective-C too that sets the state of slidesShowView to running.

If the state of the manager is not stopped, then we'll stop the session and stop the Slideshow as well. So let's see how that works. So now we've got our app, the menu item is still there. It is now there and we'll start up the video chat with Mike here and see how it works.

Where are you?

Hey, Mike. I want to show you some really cool photos I got. So instead of just beginning the Slideshow, we're going to choose Start iChat Theater from the menu and there we go. That's all it takes. As you can see, even the transitions between the slides are rendered perfectly into the shared video stream.

So that Mike can see them. And to stop we'll just choose that same menu item again. We'll talk about a little bit more about configuring that menu item. Now I want to talk about another possible way to begin an iChat Theater session. You don't have to have a video chat running first.

The user can either choose who they are going to talk to first or they can choose what they are going to share. So they can go into your application, even if a video chat isn't running, the state will be IMAVStopped, which means you can start it if you want to. So if the user chooses share with iChat Theater at that point. iChat will come to the front and prompt the user to begin a video chat.

In addition, this particular order of operations is important if the user doesn't have a video camera at all. If you don't have an iSight connected to your computer, you go into the other application choose Share with iChat Theater. iChat will come to the front and then it will be as if a camera had been connected to your computer.

You're then able to invite someone to a video chat, which you can't normally do without a camera and as soon as they connect the iChat Theater state will move to running and will start sending frames across to the other side. So let's just try out that opposite order here. We'll go back into our application.

Choose Start iChat Theater and iChat comes to the front and says iChat Theater is ready. And then I'll go ahead and invite Mike to another video chat. Well, what's the problem? Our Slideshow is running, so Mike's missing out on the whole beginning of this Slideshow. Maybe even the whole end if this thing connects slowly.

I'll just go ahead and cancel that for now. So how do we synchronize those two actions? Go ahead and quit the application and let's look back at our implementation of this action method. What we did was start the session and then immediately start the Slideshow. So that means when the session state moves from stopped through starting up and to pending, we've already started showing our slides.

So the way to do that is to use the notifications that we subscribed to in order to synchronize the beginning of the Slideshow with the iChat Theater session. So the next thing you should do is select these lines, which start the Slideshow right away and remove them entirely.

Now if we left things as they are, the user would have to say, begin sharing with iChat Theater and then once the video chat had connected, they'd have to go back to the Slideshow application and then hit play to actually begin the Slideshow, but we don't want them to actually go through that amount of work.

So, we'll look at the method which we subscribe to that notification. If you go down a little ways, the stateChanged method, it's implemented, but there's, the contents of it are commented out. So if you look at what's in here, all we have to do, remove those two comment lines and see that the playSlideshow method, which is the same method invoked from the toggleSlideshow action method.

We're going to call from the stateChanged, so what this means is, instead of starting a session and starting your Slideshow at the same time, we're going to start the session, wait for it to begin, and as soon as it is actually connected and we know video can be shared, then we'll start the Slideshow.

So all this does is grab the manager as well as the state if it's running, play the Slideshow, otherwise stop it. So let's see what that does. So this time when we say Start iChat Theater, iChat comes to the front, prompts me to invite Mike to a conference, you'll see that the Slideshow is still black. It's waiting for us to begin.

Bonjour here seems to be a little bit slow. So as soon as Mike connects, we negotiate the extra video stream and than the Slideshow knows to begin. So the only other thing we need to add to this application to make it a little bit more polished is to validate that menu item. This again is very straight forward.

If you back to controller.m and scroll up a little bit ways, scroll up a little bit there, you'll see the validateMenuItem method, go ahead and remove those two comment lines and look at what we are doing here. So, if the action that we're validating is toggle theater from the menu item, simply grab the state from the shared AV manager, if it's greater than or equal to starting up, which means starting up, pending, or running then the menu items title should be Stop Sharing With iChat Theater, otherwise it should be Share With iChat Theater. And then the menu item should be enabled if the state is not available and that goes for enabling a menu item if you also want to have a button or even a check box in your UI to determine whether or not you're going to use iChat Theater. Simply use that single state to know whether or not you're able to begin or end a session. And with that last change, we'll try our application out again Share With iChat Theater.

Once it begins the menu item will change to Stop Sharing With iChat Theater and if I end my video chat, we know that the session stopped. So our state immediately reacts and stops the Slideshow and if I quit iChat entirely right now, the InstantMessage framework will let us know and this menu item will now be disabled because the user can't start sharing. And that's everything you need to do. Let's go back to the slides. (Clapping) Thanks, Mike.

So that's the most important part of integrating iChat Theater with your application. Beginning and ending a session needs to be integrated smoothly with the workflow of your particular application and you want to give it a lot of thought. For iPhoto and Keynote, for example, we made sharing with iChat Theater an option that you have during an actual presentation.

So with iPhoto you choose a Slideshow, when you say play one of the options you choose is also to send that Slideshow to iChat Theater. With Keynote they have a special menu item to begin presenting the slides, but immediately send them to iChat Theater and begin that session.

In your applications it maybe modal. You might have single option on your entire application to share the content you have immediately live or you might have a specific task within your application that will be integrated with iChat Theater. So, make sure you give that a lot of thought. We gave this element of the API a lot of flexibility as far as what the developer can do.

So that's session control. The next thing to talk about is how you get video into iChat Theater. Again there are two different ways, you can either specify in NSView and let us take care of it for you or you can implement a couple of low-level callbacks so that you can provide exactly the content you want.

The easiest way is to use an NSView. If you subclass a standard NSView then you can simply pass that into iChat Theater using the setVideoDataSource method and when we need a frame we will render what's in your view on the main thread, so you don't need to worry about thread safety.

And we're going to do this by calling -diplayRectIgnoringOpacity:inContext:. So this is a standard AppKit method. We'll have set up our own context, and we'll, and this method will render not only your view but any subviews it has, taking into account any opacity they have below it and taking that image and sending it into iChat Theater.

One thing to remember though is that the size of the video isn't necessarily going to correspond to the size of your view. And the way we get around that is by applying a transform to the context we're asking the view to render into that will scale its bounds into the size of the actual video frame and the result is exactly the same as running your entire application at a different resolution scale factor.

There's a session, I believe it's right after this one on Making Your Custom Controls Resolution Independent, basically what this means in a nutshell, is that instead of focusing on pixels you'll focus on points. So make sure that all of your lines, your text are all aligned to each other according to the coordinates in the AppKit coordinates system instead of assuming that pixels and points are equal to each other.

What this means is, for instance, you could have a view that's very, very small when it's rendered into the video buffer for iChat Theater the quality of the content actually goes up and text will have more detail and your images can have more pixels that actually get shared in the iChat Theater buffer then that are actually displayed onscreen.

There's also a lab, I believe it's on Friday to actually tune your applications to make sure that they are resolution independent. And the only other thing we need from you, if you are using a regular NSView is to -setNeedsDisplay. Again we don't want to render this image more often then necessary, so make sure that you use the regular Cocoa view and validation mechanism, -setNeedsDisplay will see that that's happened and then rerender the view into our buffers. If you do any other weird shenanigans inside drawRect or use other ways to display an update of the views onscreen, we won't know about it and we'll never ask for a new frame.

There are a couple of special cases, QTMovieView is probably the easiest one to use. It will be able to render off the main thread because the QTMovie object that's underneath that QTMovieView can control access to the video frames that are inside of it. So we're able to render simultaneously onscreen and for iChat Theater.

-setNeedsDisplayP:YES is not required, because if the movie is playing we'll know to get new frames from it and we won't rerender frames if the movie is paused, because again, we can just get at the information in the QTMovie. Audio comes for free, if you set a QTMovieView as the video data source for iChat Theater, then any audio that's in that movie will find its way into the conference without you doing any extra work.

NSOpenGLView is also another special case. It will take a snapshot on the main thread. Because we don't have any control over the OpenGL context and the rendering that you do, when we need a frame, we will go onto the main thread and take a snapshot of whatever has most recently been rendered and then use that. -setNeedsDisplay:YES is also required for this one, even though you don't strictly need it with an NSOpenGLView, again it's our only hook to know whether or not we should capture a new snapshot.

QCView is very similar to NSOpenGLView; we'll also take a snapshot on the main thread. In the case of QCView though, -setNeedsDisplay is not required because we're going to go ahead and capture a frame from that view every single time we need one. QCViews generally have animation that's going pretty constantly, so we'll just grab a frame every time we are able to get one, which does mean that the IMVideoOptimizationStills option is not settable for this particular video data source. If you do want to have fine grain control you can drop down to using our low-level video callbacks.

So let's talk about those. What are the advantages? The first advantage is that when you use the low-level video callbacks that we offer, they render off the main thread, which means that any updates that are happening onscreen can be done simultaneously while filling a buffer for iChat Theater.

You have complete control over the shared content. When we ask you to fill a buffer, there's nothing in it. You can put exactly what you want in there. For instance if you want some UI that's visible on screen to the user who's giving the presentation, but don't want that to be visible to the other side, you can do that.

There's been with optimization that is under your control now, instead of relying on tricks using setNeedsDisplay, the video callbacks that ask for frame have a simple return value, yes or no. You can say no, nothing has changed, use the last frame that I've gave you. And the last thing is that you will have is fine grain control over frame scheduling, so that audio synchronization and the synchronization between the user's video and your video can be as accurate as possible. So let's talk a little bit about how this video architecture works.

The InstantMessage framework, which is inside your application space will create a frame. Pass that to a render callback, which you'll implement. When you've filled that buffer, you'll hand it back to us. The InstantMessage framework will then take that frame and send it to iChat where it then can be encoded using the video encoder and sent across the other side to the buddy that you're video chatting with.

Video callbacks, there are two different kinds. You can either choose to implement CVPixelBuffer- based callbacks, which are in main memory bitmaps or you can implement CVOpenGLBuffers, which allow you to render OpenGL content into the buffer. And you should choose whichever buffer type is most appropriate for your content. It will probably be pretty obvious which one you need to do based on the kind of rendering that you do.

Then for each kind of buffer type, there are two callbacks. There's one callback which will set options for the kind of buffer you want to fill and that will be called once immediately after you -setVideoDataSource:. So choose your video data source, it will implement this method, as soon as you pass it into iChat Theater, we'll ask it what kind of buffers it wants, and then, of course, there's the actual frame callback. So you'll implement one pair of these methods, either the option and frame callback methods for CVPixelBuffers or the options and frame callback method for OpenGLBuffers. So first let's talk about pixel buffers.

The first method is the pixel format that we need to know. We'll create pixel buffers in whatever format you want. So, implement this one method, it will get called after -setVideoDataSource and pass out into that parameter the pixel format you want. This one is kCVPixelFormatType_32ARGB, which is the kind of pixel format you want if you want to derive a graphic context out of the pixel buffers that we hand to you in the frame callback. Not every single pixel format is supported by CVPixelBuffers. So you'll have to choose one that does work for those.

And then in the actual frame callback we will give you the CVPixelBuffer that you should fill and then also a timeStamp that you should be filling for and we'll talk a little bit more about that. And then in each one of these callbacks, whenever it's made, if nothing is changed since the last frame, return no. This means, of course, that the very first time that we make this call you shouldn't return no because we do need a starting frame.

Make sure you don't retain the buffer. These are being recycled in a buffer pool and we don't want to rapidly fill up memories. So make sure you let this buffer go one you filled it. And then lock the base address as you would in order to actually fill the buffer directly.

The other thing you can do and this is in the sample code that we have. In the Headstart if you look in Slideshow view it takes the CVPixelBuffer and uses the base address and the pixel format to create a CGGraphics context so that you can issue Quartz or Cocoa drawing commands in order to fill it.

And then unlock the base address, very important, before you return from the method. So let's look at a sample implementation. The first thing is to determine whether or not the content is changed or not. That's something you know. So you'll need to determine that in order to return no when necessary.

Then, lock the base address. When that succeeds, make sure that you get the dimensions of the pixel buffer. They are not guaranteed to be the same for every session and they are not even guaranteed to be the same for every callback. iChat is going to dynamically scale these buffers as the bandwidth availability to the conference changes over time. So you want to make sure you get the dimensions every single time and render appropriately for whatever size that we give you.

Then get the base address, fill the buffer in whatever way is easiest for you, unlock and return yes to signal that the frame is ready for us to use. So, OpenGL buffers work similarly. The initial callback that we will make when you set the video data source is to specify the context and the pixel format that you're going to rendering with.

And that's looks like this. It's parallel to the other method. There are two parameters that you need to pass out; one is the CGLContextObj and the other CGLPixelFormatObj. Notice that, at least in this little sample here we're using mySharedContext and we'll talk about why it's important to use a shared context if you're going to be rendering off the main thread.

And then in the frame callback method it works very similarly. We'll hand you a CVOpenGLBuffer to fill as well as the virtual screen number you should be using. I'll talk about that in just a second and the timeStamp as usual. So, just as with the pixel buffer callback, if nothing has changed, return no. Again, don't retain the buffer. Then you'll attach that buffer to your context using the virtual screen number that we give you, render your scene and then when you are done call glFlush(); and then return yes.

So the sample implementation here is pretty similar, again, if nothing is changed return no and then take the shared context that you are using for this rendering and we're going to sign it to a special value called CGL_MACRO_CONTEXT and this is very important if you want to be able to render off the main thread while you are still rendering on the main thread. So let's talk about how that works for a minute.

Multithreaded OpenGL actually is possible. And the key is to drop the idea of having a current OpenGL context. If you have two threads that want to be able to render at the same time and one sets its context as the current one, starts drawing, and then the other thread sets its current thread and starts drawing, well, it's changed the context that you have and then suddenly OpenGL functions are going to get start getting dispatched from two different threads to the same context and things go south very quickly.

So in order to make this work, if you import this special header, OpenGL/CGLMacro.h, into every source file that does OpenGL rendering, it will redeclare every single OpenGL function to invisibly take an extra parameter. And that parameter is the CGL_MACRO_CONTEXT. So any commands that you issue from that point forward are going to go to the CGL_MACRO_CONTEXT that you have declared no matter what scope that it's in. You can declare in a global, you can declare it in an ivar of a class, you can declare it in a method parameter or even just use a local variable in your function.

So the way that this will work is if you have two threads, the callback thread that is rendering for the iChat Theater and the main thread rendering onscreen. In each one you would first set the CGL_MACRO_CONTEXT to the appropriate context even if these are going to be calling the same methods.

In order to render the scene, all you have to do is pass that CGL_MACRO_CONTEXT into each one of those methods so that they get directed to the appropriate one. So the offscreen context is rendering for iChat Theater gets the shared context and the main context will be used for the main thread and from that point on they can each issue commands to the GPU that will go directly to the context that they are intended for.

So that's how that works. So once you've declared the appropriate context you want to do your drawing in, attach to the buffer and make sure that you use the screenInOut parameter we've passed in. This is the OpenGL virtual screen number, 99 out of 100 of you are going to be able to just use this value directly in the attached function. In very rare cases you're going to want do rendering on a specific rendering device.

Cases are very rare, perhaps you have a machine with five-year-old Rage 128 Pro and you also want to support having a modern OpenGL, a modern graphics card in the same machine. You might want to do rendering on a specific device. That's not going to apply to very many of you, but if you do want to render to a different device then the one we've passed into the function. It's important that you write that value back into that parameter so that we know what's happened.

The CVOpenGLBuffers don't have a width and height the way a CVPixelBuffer does, so the appropriate function to use for this one is the CleanRect, which will give you the bounds that you should be using and you can pass that directly to glViewport in order to configure the rendering that you do from that point on.

Any other GL commands you dispatch after that will go into your context, get rendered, call GL flush and then return yes. And at that point iChat will be able to take that OpenGLBuffer and read it back into main memory for the video conference encoder to use and it will get across to the other side. Now the other parameter that we haven't talked about yet is timestamps.

Both of the frame callback methods have this extra parameter and we only talked about it briefly last year. So there's a very important way to use this when you are implementing your callbacks. This is a CVTimeStamp, the relevant field in here is called hostTime. We'll have just called CVhostTimeGetCurrent before making the callback into your code. So you want to pull that value out.

Do your normal rendering of the frame and then before you return from the method you'll write an adjusted value back into that hostTime before giving the buffer back to the InstantMessage framework. So the question is, how do you get from the timeValue that goes into the method to the adjustedTimeValue you'll pass back out?

Here's the problem that we're trying to solve. Suppose iChat is making callbacks into your video data source at 20 frames per second, but your content is, for example, a movie which has 30 frames per second. What's going to happen? Of course, we're going to have to drop any frame that comes and goes in between two subsequent frame callbacks.

If the timestamp parameter is ignored and you leave it as it is, then iChat will take those frames, encode them, send them to the other side and they'll be displayed at 20 frames a second. But the video won't look quite right, because these frames are being displayed uniformly in time, but they weren't sampled from your content at a uniform rate. So how do we fix this? Instead, whenever you receive a frame callback adjust that timestamp backwards in time to correspond to when that frame first appeared in your application.

When the next callback comes in, because our callback is coming in at the tail end of when that frame is visible, the adjustment will be larger. If you do this for every single frame, iChat will be able to use that information and when the decoder on the opposite side of the conference displays those frames, it can actually schedule them for display at the frame rate that is consistent with when they were sampled from your content.

And this corresponds more accurately to the situation that we are really in, which is dropping every third frame. And when you do this, this will allow us to synchronize the audio from the user's microphone with the audio provided by your application and we're able to then synchronize that audio with both video streams much more accurately.

In our Slideshow sample it's a little bit different. It doesn't have frames, per se, instead it shows a still image for an extended period of time, it does a cross-dissolve between two different images and then it shows the next slide for a period of time and then cross-dissolves again. So what's the right behavior for this kind of pattern?

The first callback, of course, gets adjusted back to the beginning of the Slideshow. Any callbacks that come in during one of the cross- dissolves can be rendered for exactly that time, because you can just use that timestamp to determine where in that crossdissolve you need to render. So those can actually go straight through and render for exactly those periods of progress in the transition.

Each callback that comes in after a transition has completed should be adjusted back to the end of the previous transition so that on the remote side that cross-dissolve ends at the time it's really supposed to. If that timestamp were left as it were, then the last frame of the cross-dissolve would last a little bit too long on the other side.

Now an added convenience is that you can use this adjustment to determine whether you should be returning no right at the beginning of this frame callback and this is exactly what the Slideshow view sample in the iChat Theater Headstart does. Whoop. Too far. Let's go back to where we were.

So, the same code that adjusts the timestamp back to the beginning, to the end of the previous dissolve can be used to compare to the previous timestamp that you rendered for. So what the Slideshow view sample does is it adjusts the timestamp back to figure out whether it should be displaying a single image or a cross-dissolve and sees that it's the same timestamp that it returned from the previous call and says, I can just bailout and return no in that situation. So that's how video works. And the last part is audio.

Again there are two ways to provide sound to an iChat Theater session. The first is to use NSSound. This is a new bit of API added to Leopard and with any NSSound that you can have you can have it send its audio directly into the iChat Theater session. First thing you need to do with the iChat Theater session itself is configure the number of audio channels you want to have.

Typically you'll just set one. The only allowable values for this are 0, 1, and 2. 0 indicates you're not interested in sending audio to the session at all. 1 means you have a single channel of sound you liked to provide. 2 is really just a convenience in case you have stereo sound and you don't want to have to do any mixing on your own. At the end of the day, all of the audio that gets sent into iChat Theater is mixed down into a single channel with the user's microphone input and sent across the conference.

So then once your iChat Theater session has connected and the state is IMAVRunning, you can get an audio device and a channel layout used for sending audio into the session. And the two methods for that are audioDeviceUID and audioDeviceChannels. The first is an NSString which identifies a special audio device, which is used for getting sound into iChat Theater.

And then an array of channels that will be the same number that you've asked for. So if you asked for one channel, that array will have one NS number in it and if you've asked for two channels, it will be an array of two and its numbers. You can't count on the channels being zero and one. They might be three and four or seven and nine.

So once you have the NSSound you'd like to play, all you need to do is set the audio device that you've got out of the IMAVAudioManager and the method for that is called setPlaybackDeviceIdentifier and that's a new API in Leopard. And then you need to set the channel mapping.

What the NSSound needs to know how to do is which, is direct certain channels of its sound into the channels that have been set up in iChat Theater. So if the sound has one channel, then that channel sound should be sent into all the channels you've set up for iChat Theater, which is probably also just one.

It is important that you know how many channels that particular sound has. Ninety-nine percent of the time it will just be a single channel sound and this is how you set it up. So the mapping works as an array, so the first item in that array corresponds to channel zero of the NSSound.

If you only have a single channel sound you take the single channel from the IMAVManager, which is in the channels array there, you package that inside an array which maps index zero to the NS number inside the other array which is the channel of the audio device that we've got and set that as the channel mapping on the sound.

So again, for a single channel it's direct. That mapping array maps to a single, either NS number or an array of NS numbers. If you have a two channel sound, index zero should contain the first channel you got out of the channels array from the IMAVManager and the second should contain the second.

And then simply play the sound and its contents will go into the iChat Theater session and be heard on the other side of the conference. One thing that is very convenient, if you download the full iChat Theater sample from the developer site, it also includes examples for using OpenGLViews, threaded and nonthreaded, that has a pair of files which declare a category method on NS Sound, adds two simple methods, play mono for iChat Theater and play stereo for iChat Theater and that has the most correct implementation of how you should play your sounds into iChat Theater. I recommend that you just download that project, dump those files straight into your application and use those. That way you don't need to worry about the details.

So, low-level Core Audio callbacks are also available if you want to have finetune control over the content. Again, you'll use the same audio device and channel layout that you get from the IMAVManager, but you need to convert those into a Core Audio device that you can then use. So it's a little bit involved. Bear with me, we can get through this bit of code.

The first thing you need is an AudioValueTranslation structure. This is used for Core Audio function to convert the NSString we gave you to the audio device ID that you'll use with Core Audio. The input to that is the deviceUID, the string that you got from the IMAVManager and the output is the AudioDeviceID that you want to have at the end of the function call, which is AudioHardwareGetProperty. You'll pass in this parameter here, kAudioHardwarePropertyDeviceForUID that says, I'm going to give you a struct, which has an NSString for an audio device and I'd like to get the AudioDeviceID out.

Pass in that AVTranslation struct and when that call is completed, you will have the AudioDeviceID that you need. And what do you do after that, that's outside the scope of this session. Using the Core Audio device is a whole other topic, of course. I think those sessions have already taken place, so you may want to get the DVDs from the conference to check out what was in those, if you didn't get a chance to see them. Those two sessions were 402 and the lab I think has also taken place, so that might not matter too.

So that's how you send audio into iChat Theater. So in summary, again there are three parts. Most important part for your application is session control. I want you to think very hard about how you are going to integrate starting and stopping iChat Theater with manipulating content inside your own applications, synchronizing the presentation of that content with the state of the iChat Theater session.

Two ways to provide video, the first is to use NSViews. They do have some pitfalls, and then using low-level Core Audio callbacks if you want the maximum amount of control. And then for audio, NSSound and Core Audio are both available to you in order to provide sound into the iChat Theater session.

If you'd like to get more information, our evangelist is Matt Drance. He works on all the sharing technologies that we have in Leopard. We have a new mailing list called [email protected]. Until Leopard ships we won't be able to talk in too much detail on that list about the new Leopard APIs, but that will be ready for general consumption as soon as Leopard is ready. And of course, all the documentation and sample code that we talked about today is on the attendee's site. And with that the iChat Lab is actually immediately after this session down in Lab D and the entire iChat engineering team will be there to help you guys if you like to get started trying to see how you can integrate iChat Theater into your applications.