See some familiar faces from the introduction session. I'm still Dave Howell, FxPlug engineer and at the introductory session, we talked about some changes that have come out in 1.2.1 and the previous version, 1.2 point of the FxPlug SDK and we showed some examples of how to use those things and we went over the very basics of the FxPlug SDK and in this session we're going to go more in depth about some topics that people ask about quite a bit on our mailing list, which is one of the good reasons to join the mailing list and ask questions there. Because we, we can go search through them at the end of the year and see what people had a lot of questions about and have engineers from the Motion and Final Cut team come in and explain deeper issues. So we'll do some of that today.

First we'll have somebody talking about porting plug-ins to FxPlug from the After Effects plug-in architecture and then we'll talk about FxPlugs in Final Cut Pro, particularly how to work with digital video images. Some of the issues with fielding and pixel aspects and so on. We'll talk about the new FxTimingAPI for getting information from the host application about the start times, durations, frame rates, field information and so on, about clips and effects and your time line.

And also demonstrate dynamic parameter hiding and showing in response to a change in another parameter. So it's a way of setting parameter state based on another parameter change. And we'll show how to use a custom UI based on a nib. The examples that we've shipped so far have shown how to, how to create a very simple NSView sub class, programatically and this one will be one that was created in Interface Builder within nib. Then what we'll go through some advanced topics related to openGL, in particular b buffers and shaders and other things as well. So to kick off this, this crazy fun, we've got Darin Cardani from the Motion team to talk about porting plug-ins to FxPlug.

Thanks Dave.

Hello, I'm Darin Cardani, I'm the third-party liaison for Motion and just like to talk to you today about porting your plug-ins. I think it was Steve Ballmer from Microsoft who said and I'm paraphrasing, developers, developers, developers, developers or something like that. I forget the exact quote. But that's a philosophy that I'm really trying to get Apple to embrace. There are a lot of third party developers out there who have been wanting to do more with the pro apps and hopefully you'll be able to now.

One of the things I'm going to talk about is the difference between C-based API's that you might use in other applications for writing plug-ins and the Objective C API's that we have with FxPlug. I'll talk specifically about the similarities between After Effects plug-in API and the FxPlug API and then I'll show you how you can write some code that will work in both applications.

So with a C-based API, you usually have a single entry point to your plug-in. It's your plug-in main function and it will get passed a selector which will tell you what function you're supposed to perform and that selector's generally something like a frame setup or a render frame and you'll get some pointers to some parameters as well. If you want your plug-in to talk back to the host app, what you generally do is call a function that's in a large block of function pointers and request some data or request a service.

And we'll talk about how that's a little different. You may get passed large blocks of data in your plug-in, because I mean, it's kind of like going through a straw. You have a single main function and you're trying to do everything from that main function, whether it's creating parameters or rendering a frame or whatever.

So with Objective-c, your plug-ins will be written to conform to a protocol. And a protocol is similar to well, I'll get into protocols in a minute. But basically, it describes all of the functions that your plug-in performs and so the host application will call your render method rather than calling a dispatch function and sending it a selector telling you to render. So it cuts out a little bit of the middle man.

When you need to talk to the host application, it's a similar thing. You ask for an object which implements a particular protocol and that protocol can be for say retrieving or creating parameters and you simply call the method you need on that object that's returned. So you only have a few objects and pieces of data that are passed between the plug-in and the host application and you don't end up with large blocks of data like in After Effects, you have your in data and out data structures.

So what are protocols? Protocols are abstract descriptions. They're interfaces that describe what your, in this case, what your filter is supposed to do. So they aren't classes in and of themselves, just descriptions of what, what the class should have and they're similar to abstract base classes in C++ or Java and there are three of them that you'll be interested in in FxPlug. FxFilter, FxGenerator and FxTransition. And the host application will also return objects to you that implement other protocol such as the FxParameter retrieval API for getting the values of parameters that you've created.

So here's the FxFilter protocol. It inherits from the FxBaseEffect protocol and it asks you things like what is the output width and height of your filter given an image that's a certain size. It will tell you to set up any memory you need for rendering a frame. It'll ask you to render the actual frame and then will ask you to clean up anything you set up in your frame set up method. So rather than going through this dispatch function and giving you a get output width selector, we'll just simply call your plug-ins, get output width function or method I should say.

So there's a lot of similarities between the After Effects API and the FxPlug API and here are just a few. In your After Effects plug-in, the main function will get sent a selector for setting up your parameters. Well in FxPlug we just have the addParameters method. For sequence data, you'll get a sequence set up selector passed to you in After Effects and in the FxPlug API, you can just set that stuff up in your, in it with API Manager method, because that's what we call every time we create an instance of your filter. Likewise there's, there's equivalence for setting up the framer rendering and setting down and getting notified when a user changes a parameter. If you want some other parameter to also change at the same time. For example, you might have a check box that enables or disables some sliders.

Okay, so let's look specifically at adding parameters. Something that you're going to do in every plug-in is set up the parameters that the user can use to change the output of their, of your filter. So in After Effects, you'd call the PF_ADD_FLOAT_SLIDER macro and then the PF_ADD_POINT, say and then the PF_ADD_COLOR macro and that would tell the host app that you need a slider, a point control and a color control. Likewise in FxPlug, we have the Fx parameter retrieval or parameter creation API and it implements similar functionality. ADDFloatSlider, addPoint, addColor.

So what it would be nice to do is to only have to write your parameter set up code once. Because it's going to have the same controls in both applications. So I'm going to show you how to do that. I'm going to create a little FxHelper class or not really class, it's an interface. I'm going to write it in straight C, because you can call straight C from C, from C++, from Objective-c and from Objective-C++.

So it should work with any other host app that you're likely to be writing for as well, because most of them have c interfaces or C++ interfaces and it'll work with Objective-c as well. So we'll call the Fx addFloatDlider, FXHAddFloatSlider, FXHAddPoint and FXHAddColor and then you've only written it once and it works in both applications.

So we start by creating an interface for the different functions that we're going to need in both plug-ins. In this case, we're going to need something that creates a slider and then something that gets the value of that slider and we pass it as the first parameter, you'll notice that's a pointer to avoid and it's called add specific.

This is going to be different in each application. In the case of After Effects, this is going to be the pointer to your pf in data. In the case of motion, it's going to be a pointer to an object which implements the parameter getting or setting API or creation API.

So let's go back for one second here. You've have this header and you're going to include this header in both of your plug-ins. Your After Effects plug-in and your FxPlug. In After Effects, you're going to write the implementation of these functions for After Effects, I called this AEFxHelpers.c and it does exactly what you'd expect. It casts the add specific pointer to a PF_InData pointer.

It then creates a parameter def structure, clears it and calls the PF_ADD_FLOAT_SLIDER macro, just like you would have done inline in your After Effects plug-in in the past. Likewise, we make a file called FxPlugFxHelpers.mm and this is where you'll implement the same functionality for your FxPlug. So we've got FXHCreateFloatSlider.

In this case, the app specific parameter is a pointer to an object that implements the FxParameterCreationAPI and we simply cast it and then call it's addFloatSlider method and you'll notice we pass mostly the same parameters. You have a parameter id, a default value, a minimum and maximum, slider minimum and maximum. If we look back at the After Effects version, it has the same thing. Minimum, maximum, slider min and max and things like that.

So in this case I'm going to show a simple set up function for a gamma correction plug-in and basically what I've done is, I've taken the standard gamma filter that ships with the After Effects SDK and modified it to work in both applications. So in your AE plug-in source, you would have as the param set up method or function I should say, it would simply call SetupGammaParameters and pass it the in data pointer and then do it's accounting that it needs to do for After Effects setting the number of parameters.

Likewise, in your FxPlug, you'll simply get the api manager for the FxParameterCreation API object and you'll pass that to SetupGammaParameters. Now you'll notice, we're calling SetupGammaParameters from our After Effects plug-in and passing it in data and we're calling it again in our FxPlug, but passing it the parameter API object. That's because we've written different actual functionality for those in the different applications, but the interface is the same.

So here's, here's our SetupGammaParameters. We're actually going to write the set up parameters once and in this case it just creates a float slider and then returns the error, if there was any. I've never had an error be returned from creating a parameter in eight years or writing plug-ins but (laughs) I suppose it could happen eventually. And this will also be included in both your After Effects plug-in and your FxPlug. And of course, the FXH function is different in each app, so it calls the correct one.

So that's, that's a simple way that you can write functionality once. You can do the same thing with your render method to actually do the processing of the pixels and I've included on the attendee website, some sample code that allows you to take the After Effects example and modify it so that it'll run in both applications.

So basically FxPlug implement one of the protocols, either filter, generator or transition. The methods that you would normally call in a C-based API or rather in a C-based API you get passed selectors and have to dispatch them. Whereas Objective-c you'll just write the methods and the host app will call them.

And you can write most of your code once and have it work in both applications, saving you a lot of time. With that, I'm going to turn it over to Paul Schneider and he's going to talk a little bit about Final Cut Pro and some specifics of things that you like to do with FxPlugs. Thanks.

Hi, I'm Paul Schneider. I'm an engineer at apple working on Final Cut. Today I'm going to talk about writing FxPlugs in Final Cut. Like Dave said, I'm going to try to answer some of the most popular questions that people have asked us over the past year. I'm going to start off with describing some common problems that people run into when they initially start trying to run their plug-ins in Final Cut.

Then I'm going to move on to doing more interesting things that you can do for the first time in Final Cut 6 and I'm going to use an example plug-in for most of my talk and it's going to be part of the FxPlug 1.2.1 SDK, which we'll be shipping any day now.

So the first thing I'm going to talk about is images in Final Cut. One of the things that you have to watch out for when you're writing a plug-in is that when you're writing a plug in on image in Final Cut, you're really processing the pixels as they come out of the camera or codec and as they're going to an external device.

So if you're used to working with like with a still image like an rgb, jpeg or even another motion graphics application, you might be surprised by how images look in Final Cut. So with digital video images, there is a couple of things you have to watch out for. The first one is that your pixels aren't actually square. Your pixels are rectangular, there's a non 1.0 aspect ratio. So you need to generally take that into account if you're doing any kind of geometric effect.

The second thing that you need to watch out for is that the images are interlaced. You're going to be processing individual fields rather than a full progressive frame and you need to be aware of that as well. The third thing is not specific to video, but it is specific to Final Cut and Motion. We will at times render on a low resolution proxy. Which means we'll give you a smaller image than, than would actually, than you think you're working on.

An example would be like for playback. We might switch down to fifty percent to save time, to play back faster and then we'll scale the result back up to full size just to give the user a lower quality but higher performance preview. So your effect needs to be aware of when we do that and render accordingly so it doesn't look wrong during playback.

So the first thing you need to be aware of is like I said, aspect ratio. Your pixels won't be square when they appear on the computer monitor or on an external NTSC or HD device. So when you're rendering a plug-in, you need to know the aspect ration of your image and you do that by asking the FxImage what's your aspect ratio using the pixel aspect selector.

The second thing are the single field images. This is sort of a difference between Final Cut and other some other applications. Not all other applications. You can see here, you know two fields are interlaced together to create the full frame and we will pass you one field at a time because the fields are actually occurring at slightly different points in time.

The second field is half a frame past the frame temporarily. The other thing to be aware of here is that these fields are half the height of the full assembled frame. Some other packages will stretch the fields to full height to make it easier to process. Final cut does not do that for a couple of reasons.

One is performance. This way you only have to work on half the data and the second one is actually, if you're writing an effect that's aware of video and wants to do interesting things with the field, it's better to have the actual fields available and not some scaled version of the fields.

So the first thing you need to do is check to see if you're doing an interlaced render and you can do that by asking the FxImage if it represents a field and that will return and instance of the FxField enum which will either be a lower field, the upper field or field none, which means this is a progressive frame. This image does not represent a field.

The second thing you need to check is whether the fields have been stretched to full height or if they're acting their native half height. And you do that using the FxHost capabilities object. This is an object that we added in the FxPlug 1.1 that lets you check various capabilities of the host.

You know, which version of which app you're running in, which work arounds you might need, which bugs have been fixed hopefully and one of the things that you can check for is this upscales field call. So you can ask the host whether it stretches a single field image to full height or whether it keeps it at it's native size.

So one way that you can account for, oh I'm sorry. The third thing you have to watch out for are the low resolution proxies. If we were rendering you on a small image that will actually be scaled up for display or for performance reasons. Final Cut will do this during playback, which means it's something you have to, especially watch out for because, it's not a user selectable thing like it is in Motion. When the user hits play in Final Cut, we will probably drop you down to half quality.

So you check for this using the render info that we passed to you when we asked you to render a frame. The render info struct contains information about the current render that we've asked you for and part of that is what scale the rendering is happening at. So be prepared for a scale X and scale Y of say 0.5 for half resolution render.

Now all three of these things, you can interpret this information any way that makes sense for your plug-in, but all three of these things can be thought of as just a scale distortion on your rendering. So here's a little snippet of code that sort of handles all three at the same time. This is also included in the sample plug-in in the SDK so, don't worry about writing it down.

You see here, I want to know how I'm going to scale my render so it'll look correct when Final Cut displays it and I want to scale horizontally by the aspect ratio, I want to check to see if I'm rendering a half height field and if so, I want to scale vertically and then I also want to take into account any renderInfo.scale for the low resolution proxy rendering.

So now I'm just going to run a demo. What I just talked about might be sort of hard to visualize, but it's actually pretty easy to see when you watch it happen in practice. So here's this example plug-in called scrolling rich text and this will be part of the FxPlug 1.2 SDK. And what this is, it's a generator and it just simply draws some text and scrolls it up and down as you move through time.

And you can see down and the bottom here, I've got three controls to handle, aspect ratio, half height fields and render scale. In your shipping plug-in, you just want to handle these all the time, but for demonstration purposes I'm actually letting you see what happens when you don't handle it correctly. So the first thing is aspect ratio.

You can see here, if I ignore the aspect ratio and I just render as if the pixels are square, my text is a little distorted horizontally and a lot of people miss this. Because for the format that a lot of people use, just DV, the pixels are almost square. You can see that that's really not that different from that and if you just saw one in isolation, you might not know which was right and which was wrong. What I recommend doing is, testing your plug-in in an anamorphic sequence.

That's got a more extreme aspect ratio and you can easily see you know, whether you're handling it correctly or not. So here you see the text is pretty obviously distorted because I'm ignoring the aspect ratio and then if I take the aspect ratio into account, scale my text to render it appropriately, it looks correct even with an extreme aspect ratio.

The second thing you want to take into account are these half height fields. This one's pretty obvious. If I ignore it, my text is twice as high as it should be and the reason for that is I've drawn it at the full height into each field and then the fields were interlaced together creating an image that's twice as high and the text is twice as high as it should be.

So again, I just want to scale my rendering appropriately so that when Final Cut interlaces the fields together to create the final output, the text looks correct. And the last one is the one that's easy to miss. Which is the render scale. You can see here, if I'm just parked on a frame, there's actually no difference.

But if I start playback and I'm not listening to the render scale, the text looks twice as big during playback and the reason is is that Final Cut is actually dropped down to half resolution in order to play back faster and so I'm drawing into a smaller image but I'm using the same text size and then Final Cut scales that image up for display.

So what I want to do whoa, what I want to do is pay attention to that render scale and then during playback, it looks correct. So this is the kind of thing where it's easy to see in practice, it's hard to imagine when we try to explain it in email or documentation but it's pretty obvious what's going wrong when you actually start writing code.

Now I just want to talk about the other interesting things that this plug-in in does besides render text correctly in Final Cut. You can see that it just draws a line of text that scrolls as I move through time. One nice thing about this plug-in is that the animation always takes up the full duration of the item in a timeline. So if I shrink this item down, make it shorter, the animation knows that and catches up.

So on the last frame of the generator, the text is completely off the top of the frame. On the first frame of the generator, the text is completely off the bottom of the frame and it just interpolates in the intermediate frames to draw the right thing and this will work even if I have more than one line of text.

So if I just copy and paste this a few times, you can see that it wraps correctly and it scrolls correctly and I'm doing my text rendering using actually code built into the operating system. I'm using a class called NSAttributedString which is a really nice class that it will take, supports a string with various formats in various ranges of the text.

So NSAttributedString knows how to draw itself, it knows how to measure itself, it knows how to tell you what area it will take up when you render it. So I didn't have to write any code for all that at all. And so I have some you know, fairly simple controls here. I can enter a line of text, this is using a basic string parameter which is built into FxPlug.

I can make the text bigger or smaller by changing the attributes. I can change the color. There's a lot more that I can do with the text and it might be sort of a pain to provide a different parameter setting for each type and there's things like you know, I only get a single line of text entry here, I would like to be able to have a little more control over that and that's where the custom UI comes in.

So what I can do is, I can switch from my simple text entry controls to using an external file on disk and when I do this, you'll see that the simple text that it controls disappear and they're replaced with this custom UI which I defined myself and what this will do, this lets me do, I can just pick any rtf file, I happen to have one on disk and then it'll use that for the text rendering.

So I've got a huge range, I've got a huge text file here, not huge, but I've got a fairly lengthy text file here, simply more than I could have typed in myself in Final Cut. It's got different font characteristics, at different ranges of text, different colors, different spacing. This paragraph is you know, center justified and if I want to edit this, I can use a great text editing facility that comes with the operating system. I'll just open it and TextEdit.

So I can you know, take this, make the text a lot bigger, type in some more and then just use that as my text editing application. So that's pretty nice. It didn't take me very long to write this plug-in and I get very rich text editing capabilities that are already there. So let's switch back to the slides and I'll talk about how I did that.

So the first thing that you might have noticed is that the generator actually knew how long it was in the timeline. So that it was able to animate correctly and I used that, I did that using the new FxTiming API, which we introduced in FxPlug 1.2. I'm just going to talk briefly about the timing API, we talked about it a bit in this morning's session and I'm not going to talk about all the things it does. I'm just going to give you a taste.

So the timing API gives you information about your effect in the timeline. You might want to know the frame rate of the timeline you're in, you might want to know where your effect starts in the timeline and how long your effect is in the timeline and you can find that out with the timing API.

It'll also tell you about the effects input. So you can get information about different images that Final Cut is feeding you. You might want to know the start time of the clip you've been applied to, if you're a filter, the duration of that clip, the field order of the clip.

If you've got a mixed format situation where your filter has been applied to a clip that's got a different frame rate or field order than the timeline, you might want to know that information so that you can intelligently do something appropriate if you're doing a retiming effect or something like that. And we also give you facilities to do that. If you've got a different frame rate in front of your input than you do on the timeline, we'll let you convert between those different times.

So this is the concrete example, a very simple one, here's how I wrote my generator. I just used the durationForEffect call, to find out how long the generator was in the timeline and then I looked at the current frame, the frame I'm being asked to render right now, divide that by the duration and that's how far through the generator I am right now. So I know how far through my animation I should be. The one little gotcha here, I had to subtract one off the duration because, the renderInfo.frame that we passed through the current frame, is zero based.

The other thing you might have noticed that happened was when I switched that pop up menu, I hid some of my parameters and showed some other parameters and this is something that's new in Final Cut 6. It's pretty straightforward. The first thing you want to do is create some of your parameters hidden by default and you do that with the parameterFlag_HIDDEN flag.

So when you create parameters, you pass them some flags. One of the flags is I want this parameter to be hidden on creation and then the next thing you want to do is hide and show parameters in response to another parameter changing and whenever one of your parameter values changes in Final Cut, we'll send you a parameter change message.

So here you can see, I want to know when the menu changes so I know if I should hide or show a different parameter. So I checked to see if a different parameter that changed was my hide show menu parameter, if it was, I get the new value of the hide show pop up and if the pop up says I should hide my parameter now, I'll set the, I'll set the hidden flag on the parameter, the hide otherwise I will clear the hidden flag on the parameter to hide, which means show this parameter if it's hidden.

And then the final thing was my custom UI. Actually created that in Interface Builder with a nib. Now from the beginning of FxPlug, we've let you create custom user interfaces just be returning an NSView and we've always said oh, you know, it's an NSView so you can create it in Interface Builder. You can be as fancy as you want. But we've never actually shown you how to do that before so now we have a sample that shows that off.

So the first thing I do is in my nib, I define my plug-in class. In this case, my plug-in is implemented by class called ScrollingRichText. So I tell Interface Builder that a class with that name exists. The next thing I do is define some properties for the plug-in class, so I will create an outlet which is just a pointer to the custom UI and I'll create two actions. One called chooseTextFile and one called editTextFile. If you remember the demo, I had two buttons. One to let me choose a file on disk and one that opens that file up in TextEdit. So these correspond to those two buttons.

And then the next thing I will do is I'll actually make my UI. So I'll create a view in the Interface Builder, I'll drag in the controls I want from the pallet, so there's my choose button, my edit button and a little static text that just displays the name of the file that I've chosen. Now this stuff is a little bit tricky, but what I'll do is I'll make the owner of the nib in instance of my plug-in class. So in this case, my plug-in class is called ScrollingRichText, so I'll say, this nib is owned by the ScrollingRichText class.

Once I've done that, I can wire up the connections. So, I've got in instance of the ScrollingRichText class in my nib, so I'll wire up it's outlet, it's custom view outlet to the view that I created and then I'll wire up the buttons inside the view to the classes actions.

So that I'll drag that choose text button to so that it fires off the choose text file action and similarly for the edit button. So that's all I do in the nib. Now it's time to write some code. Here's what my plug-in class looks like. It's a generator and it's got the outlet and the action defined. These are the ones that created an Interface Builder.

Now my next step is I will create the custom parameter. So in my add parameters function, I'm going to add a custom parameter, I'm going to give it a default value which in this case is just an empty NSString. You recall that custom parameters in FxPlug can be an instance of any Objective-c class that conforms to the NSCoding protocol and that's necessary for reading and writing your data from a project. So in most of our sample code, we will define our own classes and we'll implement the reading and writing routines but it turns out that appKit already provides you with a whole lot of classes that do this for you.

So if my needs can be met by an NSString or an NSData or an NSArray, I can just use that class as my custom data type. I don't need to write my own class. And then the final thing is I would set the custom UI flag on my parameter, to let the host know that the plug-in itself will be creating the user interface for this parameter.

And when you set that flag as far as the host calls you back when it's time to create the user interface. So in this case, I'm being asked to create a view for my custom parameter. And this is the really cool part because, all I have to do is load my nib passing myself as the owner.

You recall that the owner of the nib is an instance of the ScrollingRchText class, so I pass in myself as the owner and loading the nib wires up my outlets and my actions. So after I've loaded the nib, my custom view pointer points to the NSView that I created in Interface Builder. So I'm done. I just return that view and it's also the buttons inside the view are wired up to my choose text selectors and my edit text selectors. And that's it.

So I'll just show that one last time in case you forgot how the plug-in works. Here's my pop up, I respond to changes in the pop up by hiding or showing certain parameters, in the more complicated example, I've got my user interface that I created in Interface Builder, this choose button is wired up, this edit button is wired up and that was literally all the code I had to write to do it. So that's it for me. I'm going to hand it over to Pete Warden now.

Good job.

So hi, I'm Pete Warden from the motion team and one of the biggest problems for plug-in developers, one of the biggest questions or set of questions that comes up on the list is dealing with openGL, dealing with some of the advanced operations that you typically want to do when you're implementing image processing filters on the GPU and we figured for these advanced real world filters, often times it's not the individual building blocks of the problem, but it's time to put it all together and the best documentation is very often having a working piece of sample code that you can refer to and use to help you figure out problems on your own and to illustrate some of the, some of the issues.

So we actually took one of Motion's internal filters, one of the shipping filters and tied it up, made it stand alone and we're going to be shipping that as sample code with the 1.2 SDK. So this means this is code that's been pretty heavily through our QA mill, it's been optimized as much as we're able. So it should be quite useful to refer to and it shows two issues in particular that come up time and time again that are really tricky to deal with.

Rendering to a texture. If you've got a complex filter, you're very likely to have a graph internally to the filter of image processing operations where you need to pass data from one to another and dealing with floating point rendering which is a fairly obscure area of openGL and isn't documented all that well. So this is all going to be in the SDK which we're going to be shipping shortly.

Now why is the openGL side of writing an FxPlug so hard? It's fundamentally because openGL was written to be a 3D rendering API. It was written to be geometry processing. We've had a lot of success using it for image processing, but some of the very basic features that are building blocks of writing image processing filters, do require some API's and some parts of openGL that are more arcane, more obscure, less world documented and harder to get information on.

Rendering to an intermediate image is one of the most common things for more complex filters to have to do where you have operations, where you need to pass image data from one to another, where you're building up from individual pieces, your rendering graph and your drawing multiple passes and on the GPU, it's quite different from the CPU. On the CPU all you have are images. You can usually access the data directly, you can read from that data or you can write to that data. It's all very orthogonal and it's all very flat.

Whereas on the GPU, the things that you draw into are very different beasts than the things that you read data from. That's fundamentally part of the design of the GPU, part of the way that they get their massive speed up is they've got a very strict pipeline about what's input and what's output. There's several different ways of rendering into a texture using the GPU. I'm going to be focusing on PBuffers.

One important thing and probably one of the most crucial things about PBuffers is that they stay in VRAM. If you want to get performance out of the GPU, you really, really, really need to make sure that you're operations are staying on the GPU as much as possible. If you're coming from a CPU-based background, it's very tempting when you're running into issues, to try and do a read back from the card and do some CPU-based processing and then write it back up to the card and that's really death for performance.

It's very, very bad news because the bus between the graphics card and the CPU isn't all that fast compared to how fast the graphics card can process and it means that the CPU has to wait until the GPU has finished whatever operations it's doing so you no longer get the parallelism that you'd really like. So unlike some of the older techniques of rendering to a texture, this stays entirely in VRAM.

Frame buffer objects were introduced fairly recently, we're modified fairly recently but, they also have this advantage and they are nicer in some other ways as well.. They actually have nicer API and we're using them in some parts of the Motion engine but, since almost all of our filters are still using PBuffers, we wanted to illustrate the way that we're most familiar with and that's still working very well for us.

So within the code sample itself, this is the small set of utility functions we actually give you for doing common operations with PBuffers. The first two hopefully are fairly self explanatory, you use them to create and destroy PBuffers to allocate and deallocate PBuffers. I am going to talk about some of the hidden gotchas next about dealing with allocation and deallocation because it isn't as trivial as it might appear.

When you want to start drawing into a PBuffer, when you start want to start sending drawing operations onto a surface, into a texture, you call PBuffer_Begin and then subsequently all of your GL calls that you make after that will go into the surface that you set up as a PBuffer.

It sets the current context, the current geo context to point to the PBuffer surface that you set up. And when you've completed all of your drawing and you have finished up, you can call PBuffer_End and that will finalize all of your drawing operations and will send any subsequent GL calls back to the surface that you were drawing into before you started drawing into the PBuffer.

And then finally, when you actually want to use the contents of that PBuffer, when you want to use what you've drawn into that PBuffer as a texture, you call PBuffer_Use and that's actually very similar to using a normal texture. So wherever you'd use Fx texture use or do a GL bind texture, you can just call this PBuffer_Use command and it will make the text unit you got as the active texture use the contents of that PBuffer that you've drawn into as a texture so you can use it in fragment programs or whatever other drawing operations you're using.

The allocation issues I talked about are all based around that fact that allocation and deallocation of PBuffers is very, very expensive. It's not like on the CPU side where allocating an image and deallocating and image is normally around the same cost as a malloc, a fairly lightweight operation.

It's a kernel level object. It involves a lot of housekeeping on the OS side and it's generally a very expensive operation. So in practice, this means that if you're coming from a CPU background, you're used to writing render function where you would actually allocate any intermediate images you needed at the start of the render function, use them as you do the processing and then at the end of the render function just delete them. That is a very bad pattern to use on the GPU when you're using PBuffers. Because, that means you'll be doing an allocation and a deallocation over PBuffer every single frame that you render.

Instead, what we recommend and what the sample code demonstrates, is trying to keep PBuffers around as long as you can. Trying to create PBuffers at the start of your render function, use them as before but then at the end of the render function, have them stored in member variables so next time you're called, you're actually able to check the size and the bit depth of those PBuffers that you previously created and if they're big enough to reuse, you've completely skipped the allocation and deallocation that you were doing before. And this to give you an idea of the magnitude of the difference this can make, we've had plug-ins where we've gone from four or five frames a second up to thirty or forty frames a second, just by making this change.

So it's a bit of extra housekeeping, it has a bit more complexity, you do need to make sure that you destroy them in your dealloc call so that you don't end up leaking, but it can make a massive difference to performance. So it's highly recommended that you actually do this to get the best performance out of the GPU.

Paul talked about using proxy resolution within Final Cut and the same thing happens within Motion. Very often when you're writing simple plug-ins, all that means is that you have to scale the parameters that you're using internally by the resolution. Once you're rendering into intermediate images, one trap that we often fall into ourselves and that we've seen third party developers fall into, is still doing your processing at full resolution, even though the input and output are actually a low resolution and the reason this is a problem is, the user has switched or the app has automatically switched for the user to a lower resolution so they can sacrifice quality for performance.

They expect to see complex projects speed up when you're actually using proxy resolution scales and if your filter isn't respecting that and it's still processing the same number of pixels internally, it's going to be a pretty bad experience for the user and they're going to have a bad day. So just another thing to watch out for. Just make sure when you switch to a low resolution that your filter seems to speed up. If it doesn't, have a look at the intermediate images and see if maybe there's something going wrong with that.

So the final piece of GL arcana that I'm going to talk about is floating point. Now this is obviously a crucial thing if you're going to be doing high quality rendering and we actually require filters to support on the GPU 16-bit and 32-bit float. But this is something that was introduced pretty recently on consumer graphics cards.

It's only been the past two or three years that cards have even begun supporting this at all and for technical reasons, there's a couple of capabilities that we take for granted on the 8-bit side, but very many graphics cards don't support. Many of the more modern ones actually support these two at 16-bit float but almost no cards support them at 32-bit float.

And those two capabilities are bilinear filtering where you're doing a fetch from a non-integral texture coordinate and you're expecting to get the weighted average of the four neighboring pixels and blend mode compositing hardware blending on openGL where you're actually expecting that the text that you're drawing will be combined with the background that's already there and then drawn into the background of the Frame-buffer. So both of those don't work on many cards, just the hardware doesn't support them.

Unfortunately, there isn't any good API for discovering if the card that you're running on at the moment and the bit fetch that you're running on at the moment supports these. These are always supported 8-bit, but once you enter float, there isn't a good technique for using this. Initially we did try doing our own card id checks which have all the usually problems of new cards coming out, id strings changing and just all the reasons why that's usually a bad idea. So what we ended up doing in Motion is actually running a test at start up for different bit depths.

We will run operations that rely on bilinear filtering and operations that rely on blend mode compositing and that should produce known values when they're drawn into a, into a buffer and then we will read back the pixels that were actually drawn to tell if the operation worked or not.

Now this is obviously pretty scary stuff, pretty complex to write yourself so we very strongly recommend that you try and avoid relying on these two features of openGL within your GPU FxPlugs. If you can, design your algorithms so that you don't rely on these. It will make life a lot simpler, it will make life a lot easier. If like us, you've already written a lot of your algorithms to rely on these, there are some alternatives that you can use. There are some work arounds if you're willing to put in a bit of extra work and deal with some performance loss.

The sample that we're going to be shipping, the directional blur sample actually demonstrates this for bilinear filtering. We have a fragment program that does four nearest neighbor texture fetches and then does the weighted bilinear filtering internally. It looks at the factional part and adds them together and it's actually was surprisingly tricky at least for us to get that right, because of the placing of the center pixels and a lot of the flawing and stuff. So hopefully if you're doing any bilinear filtering that's going to be useful for you guys. It does mean four times as many texture accesses, so it has a serious impact on performance. So that's a good incentive to try and avoid it if you can.

Blending you can also emulate. It wasn't needed for this particular filter so it's not included in the sample code but, the way you do that is you actually take one PBuffer that you're using as a background and the texture that you want to blend over that background, put them both as inputs into a fragment program and because you cannot read from the same PBuffer that you're drawing into, you actually have to draw into a second PBuffer after you combine the two original inputs in the fragment program and then when you want to composite something further, you then ping pong back from the one you've just drawn into using. That is the background into the second PBuffer and just continue doing operations like that.

So again, you can probably see, this is a pretty heavyweight operation compared to the normal hardware openGL blending, especially if you're trying to blend a lot of small triangles. It rapidly gets very, very costly. So again, if you possibly can, try and avoid using either blending or bilinear filtering. It will take you a lot of time to get right and it really won't give very good performance.

And that's it. I'm hoping you'll get a chance to look through the directional blur sample code when it's out there with the SDK and you'll be able to send plenty of questions onto us and I don't know if Dave wants to wrap up. Thanks.

(Inaudible) Thanks Pete.

We saw a few things today. We saw a brief overview of the FxPlug features that have been added and we talked about methods for sharing code when you're trying to support more than one plug-in architecture. That's also useful for porting, the concepts in there are good as a reference when you're porting to FxPlug.

And we saw examples from the directional blur example and the rich scrolling, ScrollingRichText example and those are both in the plug-in examples in the new FxPlug SDK which will be uploaded to developer connections soon so get to know them. There's a lot in them. It's worth really kind of stepping through and looking at all the code in there as you're getting started with FxPlug or if you're trying to refresh your knowledge or move on to some to attack some difficult problem that might be covered in them. For more information, please do join the pro-apps-dev mail list.

This is an open forum that other plug-in developers can share their wisdom with you on and on the Apple engineers from the Motion and Final Cut team do frequent this list and share information too. If you have a confidential topic to discuss, you can write to an internal group at Apple, the [email protected]. And watch for the 1.2.1 FxPlug SDK. The code for sharing on After Effects, sharing your code with After Effects is up on the attendees site on the, on WWDC site, so please look for that.