Good afternoon everyone. Welcome to the third and final Quartz Composer session during this WWDC. My name is Pierre-Olivier Latour and I'm the Engineering Manager for Quartz Composer at Apple. In this session, titled "Advanced Techniques of Quartz Composer," we are going to learn a number of things. First of all, like the title of that session implies, a number of advanced techniques using either the QC view or the QC renderers and how to do low-level rendering of compositions. But then the biggest part of that session is going to be how to write your own patches, so we'll have a deep look at that.

Through that session I'm going to assume a number of things that you are already familiar with the Quartz Composer concept, which would be fine if you have already played with it or if you went to either a session this morning or the one yesterday. I'm going to assume you have the basics of OpenGL as well as the basics of the Objective C 2.0 and the changes that went into that language.

So let's start with advanced techniques. First of all, the QC view. Well, the QC view is the easiest way for you to display Quartz Composer contents inside your application. The way it works is it's pretty much an autonomous view that's driven by a timer that's running on the main thread, and every x number of times per second, it's going to render a new frame. So because the QC view is kind of living in its own world, the question becomes how to best interact with the rendering that's happening in a QC view.

In part, we're introducing a new method that's called "renderAtTimeArguments". That method is only for subclasses of the QC view, so you don't want to call that directly. And when you subclass QC view and override that method, you do not want to modify the time or arguments parameters when passing them to the super implementation.

So what's the point of that new method? Well, it's to cover two usage cases when you want to interface tightly with the QC view. It's either to do synchronized communication, when talking and retrieving results from the QC view, or if you want to even add your own OpenGL rendering as underlays or overlays.

So let's look at the synchronized communication. What does that exactly mean? Well, remember, in the QC view, the composition is rendering its own world and its own rate and so on. So the problem becomes, what if you want to set some input parameters of the composition, but you want to set it for every frame? So the way you would do it in Tiger is by probably having your own timer that's running a little faster than the one of the QC view, say, 70 hertz or something like that.

And then you would end up setting the parameters all the time, but it wouldn't be synchronous. So the idea here is that you can now know precisely when the QC view is about to render a frame, and therefore set your input parameters only when necessary and only at that time. And then it renders, and you retrieve the result. So let's look at an example.

You subclass QC view. You override the renderAtTime arguments method. And the first thing you do is you're going to call super so that the QC view is not broken and actually performs rendering. But then before calling super, you can call a set value for input key and set your input parameters at that time just before rendering happens. Then rendering happens, and you can retrieve the results. So it's pretty straightforward.

A more interesting and let's say more powerful use of that is if you want to add some custom OpenGL overlays or underlays combined with the rendering in the QC view. A couple of restrictions there. First of all, we're going to do OpenGL drawing and the current context does matter.

Instead of changing the current context to the one that the QCV uses, it's simpler and more efficient to just use the CGL macros. So there'll be more in a minute. A golden rule to follow when you work inside Quartz Composer and do OpenGL code, and we're going to repeat that a few times throughout the presentation, is make sure you save/restore the state you change. Open--Quartz Composer is tightly integrated with OpenGL.

OpenGL is a giant state machine. If you change the states and you don't restore them properly to what they were before, then the whole system end up in an inconsistent state and your rendering is not correct. And finally, you want to make sure you check for OpenGL errors.

So here's what it means. In that case, we subclass QCVue, override render at time like in the previous example. You will notice at the top of the--of that example sample code that there is the inclusion of that CGL macro file, which basically is going to make it so that whenever you call an OpenGL function, it will be targeted to the context, to a specific OpenGL context that's defined by the variable CGL_CTX. And it's basically a bunch of preprocessor macros that expand. And this way, you don't have to touch at all the current context. You don't have to try to set it, to restore it, anything like that.

It's very straightforward. You just retrieve the context that the QCVue uses by doing, in that case, self OpenGL context, and then CGL context object to retrieve the underlying CGL context object. And then every OpenGL code you're going to put there from that point on will be sent to that context.

So about the OpenGL code, so here's an example that would just draw a red quad. Remember what I said that was very important. You want to save, restore the steps. So the first step at the top is saving the current step that we're about to change, changing them, then perform the drawing, restore the original steps. instead and handle any errors.

Another thing we added to the QC view in Leopard is the ability to retrieve directly the image that it's rendered. It's very straightforward as well. All you have to do is call on the QC view you want to retrieve the rendering, create snapshot image of type. And what's great about that method is you can specify a number of types.

So you specify the type as a string, which can be for instance NSBitmapImageRep or could be CVPixelBuffer. That method would take care of downloading the image from the GPU and flipping it vertically if necessary, because the OpenGL rendering on a GPU usually happens upside down, so you don't even have to deal with that.

It will provide you directly an object ready to be used in the format you want. Now you've got to be careful because that method starts with "create" for a reason. It means it's going to return you a brand new object. It's not an auto-release, it's not on the auto-release pool, so make sure you call "release" on it or the appropriate CF release type, whatever it is for the image type.

And you will also want to make sure for best performances to ask for the type that is best suited for what you're going to do with it afterwards. So for example, if what you want to do is get the image, pass it further to QuickTime, well, QuickTime, like the compression session APIs, they're typically using CVPixel buffers. So you ask for a CVPixel buffer. If you were to pass it to Core Image, you would ask for CI Image and so on.

One of those cases is I want to pass the rendered image to another QC view or QC render or something like that. The best type you can use in that case is CVOpenGL buffer because it is a buffer that is on the GPU and therefore your content will stay on the GPU. There is no extra download and then upload that would be done. I'll show you a quick demo of some of those features.

showcasing those two features, you can see here, I got a basic composition with an Apple logo, and I can just rotate it like that, do some inertia in the logo. Well, here, that red frame could be, for example, a set frame area in the video world. And that is drawn by a few lines of OpenGL code that were inserted after the drawing of the QC view.

So it's completely synchronous with the rendering of the QC view. Now, you can see here I also have a secondary window, which has a timer, and it's simply an NSImageView inside. The timer fires 10 times per second. Take a snapshot of the main view and put it as an NSImage inside the NSImageView here. So it's really simple to do overlay rendering or underlay or grabbing snapshots from the QC view. Back to slides please.

So like I said earlier, you can interleave the rendering of the QC renderer with your own OpenGL code, which allows interesting underlays and overlays as well. In that case, you can see that I took the same loop as before, but I added a bit of OpenGL code before and after, where you would draw some kind of background, then you draw the composition. Obviously, we're assuming here that the composition is not covering the whole drawing area. Otherwise, it's pointless to draw something before. And then you can draw some overlay with some GL code.

If you want to communicate with the compositions through the QC renderer, it's the same exact set of methods as on the QC view. So, set value for input key will set an input parameter on the composition. Value for output key will retrieve an output result from the composition.

Now, regarding the arguments you can pass when you call renderer time arguments, there are two of them and they're completely optional. The first one is QC renderer event key. If you have an NS event around, because your QC renderer, the result is being displayed in Windows or something and then you have NS events flowing in, you should pass it in to the QC renderer when you call renderer time, pass only the current event.

And this will allow patches that depend on events, for example, the keyboard or the mouse to work properly for the mouse clicks and such. If another optional argument is passing the current mouse location, so the way we do that here is you put the mouse location as an NS point stored inside an NS value and the coordinate system for the mouse must be normalized 01 to 01.

A great thing about the QC Renderer is that you can do offline rendering. You could have a custom thread that's a separate thread from the main thread that's rendering composition in the background or something like that. Or you can just have a common line tool that's rendering composition directly to a bunch of TIFF files, things like that. Well, we made it easier in Leopard where there is a new method on QC Renderer called InitOffScreenWithSize, Colorspace and File.

So what it does is taking care completely of creating the OpenGL context, the P buffer to render on the GPU, all of that, you don't even need to know about how it works. You just use it. You specify a file, a color space. You can even pass null instead of a color space to get the default one, and you're good to go.

Now, that rendering still happens on a GPU, always happens on a GPU, which means you cannot render at size that won't be supported on a GPU, for example, like 16,000 by 10,000. That wouldn't work. Typically, the size limitation is 2K, 2K on the current hardware. But-- most of the machines we're shipping today are probably more than 4K and 4K. And depending on the amount of VRAM you will have around, the rendering will happen more or less fast.

I would like to now look at a very interesting possibility we added in Mac OS X Leopard, which is the ability to do chaining of compositions. The idea behind that is that instead of having a big monolithic composition with a ton of patches inside, you divide it into a set of primary composition and then a bunch of sub-compositions that are well-defined and that you load dynamically into the first one. So that was not possible in Mac OS X Tiger and you can now do that in Leopard using a new patch that's called the Composition Loader. So let's do a demo of that.

The Quartz Composer is a powerful visual programming tool for using graphics and animation on the Mac. With Quartz Composer you can easily explore the graphics stack in Mac OS X. Mac OS X Leopard brings a number of new APIs as well as the ability to write custom ""patches"" for use in your compositions, allowing you to take Quartz Composer further than it's ever gone before.

We'll demonstrate this new technology as well as advanced API techniques for integrating Quartz Composer technology into your applications, allowing you to take Quartz Composer together with your computer. is applying some kind of neon effect to an input image and you can also pick the color of that neon effect. What I want to do now is build a new composition from scratch that puts these two sub-compositions into this master one. So let me organize that quite a bit.

Okay, so what we need is that new patch called the Composition Loader. I'm going to create an instance of it. The first thing you expect from it is the ability to specify the composition to actually render, and there is an input for that, composition location, which will accept the file path, the URL, usual things. So let's take that gradient composition.

It's loaded and now it's rendering here. However, that would be pretty limited if you were only able to load composition and render them. What we want to do as well is communicate with the inputs and outputs and we can do that. If you display the inspector for the new composition loader patch, go to the settings pane. You can configure a kind of template with the inputs and outputs of the composition.

If I look again at the gradient composition, there is one input and the key is "image." So here in the settings of that patch, I'm going to say I have an image of type-- I have a-- sorry, an input of type image with the key image. Create that. You can see the image is now available as an input. And I can take my Apple logo and connect it there.

And it just works. Now, what would exactly happen if I were to load a composition that doesn't have an image input with the key input image? Well, nothing. I mean, you wouldn't get an error. It wouldn't get connected. So you can see the inputs and outputs that you define on the compositor, on the composition loader, as kind of a weak template of inputs and outputs.

If the composition that's loaded has one or more of those inputs and outputs, they will automatically get connected, and you can talk to those inputs and outputs through the inputs and outputs of the patch. If they don't exist, well, no big deal. You don't get errors. It doesn't crash or anything like that.

Now let's insert our effect, the neon effect. I'm going to create another Composition Loader, and in that case we also have something interesting, which is We don't want that neon effect to render. We want it to be inserted between the Apple logo image and the image that's displayed in the second composition loader here.

So let's look at the settings. And you can see that the top setting is the type of patches. So remember, there are several types of patches in Quartz Composer-- three to be precise-- processor, consumer, and provider. And obviously, the system, when it loads a composition from disk, the type definitely matters. So in our case, we are going to use that neon composition as a processor with the input image and color inputs.

And it also has an output, which we can see by looking at the composition. Because right now, we don't have a way to display the outputs on the viewer. We only display the inputs in that area. You can see it has a published output, which has the key output image.

So I can go back here, set the execution mode to processor, and now I can define inputs and outputs because consumers have that particularity that they're not allowed to have outputs. So now the two sets are visible and I can add-- An image input with the key input "image", an image output with the key output "image". Let's specify the composition location.

So as you can imagine, that opens the door to pretty powerful possibilities. And as a demonstration of that, for those of you who were present last year, there was this hands-on session that I gave where, at the end, we were building a fact TV system that was called the QC TV. And it's a sample code that you can get from the ADC website.

The way it was implemented is that there were a number of composition for each part of the TV setup, where you have one composition that handles the crawler at the bottom, another one that renders the upper bar with the logo and such. Each of them was a different composition. And then in the source code, there was a QC renderer for each of the composition. And everything was put together in an OpenGL context.

Well, now doing the sensing would be a lot simpler, because I have here a composition folder with all the sub-compositions, which is really convenient when you want to have artists or programmers work on separate composition without having everyone touching on that big master composition. So it gives you granularity. It gives you an easy way to replace sub-compositions.

And here, it's really a bunch of composition loaders that are configured with the proper inputs and outputs. And now, on the code side, you would simply have one QC renderer or one QC view to be able to render that, instead of having a number of QC renderers and do all the logic to render them in a proper order and such. So that would simplify your life quite a bit. Let's go back to the slides, please.

Now it's time to get down and dirty with the real thing, which is writing your own patches. This has been the number one developer request, as you can imagine, so the nice thing about it is that, yes, it does make sense for you to send all your requests and bugs to [email protected] and it actually works, so we listen to them.

We wanted it, that ability to write custom patches, simple and powerful like the rest of Quartz Composer, which concretely means we wanted it easy to be able to manage your set of inputs and outputs on the custom patch, easy to make, to do image processing or OpenGL rendering or even provide a custom user interface to edit kind of the internal settings of your patch. So to achieve that goal, we actually built completely a custom patch mechanism on top of Objective-C 2.0, which as a side effect implies that those custom patches that you write will only work on Mac OS X Leopard. Let me introduce you to that new class called the QC plugin.

This is the base class for writing your custom patches. It is itself a subclass of NSObject. And the way it's going to work is that you subclass in turn the QCPlugin class. You implement one or more-- you implement or override one or more methods, the one you need, define the inputs and outputs, implement how exactly the patch executes and so on. And then you end up with a plugin that you can put at the appropriate location on disk. It gets loaded inside Quartz Composer and turns into a custom patch.

There are a few requirements when you write custom patches. The first of them is that, obviously, you want to be a good citizen and your custom patch must be able to have-- well, let's say your QC plugin subclass must be able to generate multiple instances of those custom patches.

And that's really a requirement. The other one is that you've got to be able to write code that is going to work on any thread. We're not talking about reentrant code. But we're talking about code that doesn't matter if it's executed on the main thread or a background thread or those kind of things. In the same direction, you also need to make code that works if you don't have a run loop around.

So for some system services, the way they work is they require a run loop to be able to post notifications and things like that. If you want to use those APIs inside one of your custom patches, you will likely need to spin off a custom thread. And then in that thread, have a run loop and then communicate with the patch.

Here are the basics of the QC plugin class. So you subclass it, and then the first two methods you are likely to implement are attributes, which is called so that the system can retrieve the name, description, and so on for the UI. And if you need to, you can implement init and dialog, which are the way the plugin instances are created.

Now the important thing is how do those custom patches execute? Well, there are a number of methods on the QC plugin class that you implement whenever needed. You will notice that there is a certain symmetry in those methods. So what happened is when the engine starts, start execution is called on your plugin instance. When the engine shuts down, stop execution is called.

When your patch starts being used by the engine because somebody's pulling data from it, pulling, let's say, results from it, then enable execution is called. And when nobody needs the data anymore, for example, for several minutes before it's being used again, disable execution is called. And at the core of it, we have the execute method, which is called by the engine whenever results are needed from your patch.

You will have likely noticed that all those methods were taking a parameter that was an opaque object complying to the QC plugin context protocol. That protocol defines the number of methods that you can use to get information about the rendering destination, like obviously the bounds, the color space, and the OpenGL context to render to it, as well as a couple utilities so that you can send a message to the Quartz Composer log or store some user information. The interest of that user information method is that this mutable dictionary is shared between all your instances that are running in the same Quartz Composer context.

So that's a great way if you need to do caching that is shared between your various instances in the same context to use the user info dictionary. You want to be careful. Do not retain that QC plugin context object. Do not make calls to it outside of the scope of those execution methods. It's really a dedicated object for that, for only being used inside the execution methods.

So the core execution, like I mentioned earlier, is all about that execute method. So remember, in Course Composer, we have patches. Those custom patches are going to execute in the environment of the QC plugin context. They get their input parameters. You read the values. So that means reading the values from the input port. Then you do computations in your execute method to build the result. You can also take the time into account if necessary. And finally, you output those results to the output ports of of your patch, or you render it to the destination.

There are two more things we need to define regarding execution. Like I mentioned earlier with the CompositionLoader, those three types of patches, Provider, Processor, and Consumer, are fairly important to the system. So you need to implement the plus execution method in your class to define what's the execution mode for your patch.

So as a reminder here, providers are the ones that are executed whenever their outputs are needed, but no more than once per frame. Processors are kind of the lazy patches. They're only executed when their input change and their output change are needed. Or the consumer always are the one putting the data from the two others, and they're pretty much always executed.

You also need to specify the time dependency of the patch. Three cases. Either your patch doesn't care about the time at all in its execute method-- it just ignores the parameter, so the type is none-- or it can definitely take the time into account its computation, so you would say the patch has a time base.

Or there is kind of an intermediate case where you have some patches that just need some idle time, because it could be a provider patch that's connected to some hardware, and you want to make sure it pulls data from it whenever it's needed by the engine. So you don't truly depend on time, but you need some idling going on. So that's the kind of a special case that is defined by the KQC plugin timeModeIdle. But it's pretty rare. You shouldn't use that very often.

So we define how the patches execute, when they get executed, and so on. The last thing we need to look at is, well, how do I define my inputs and outputs? So we're completely leveraging the Objective-C 2.0 properties. And the way we do that is if you define a property of your subclass whose name starts with input and output and whose type is one of the supported types, then we automatically turn that into an input and output-- or output, sorry-- on the patch.

So for instance, you can see here I have a dynamic property called input value one, double, and that will turn into an input port on your patch with the key input value one and of type number. So the same thing for input value two, and it's pretty much the same thing for an output result property.

You still need to define-- it's not mandatory, but it would be nice for the UI-- that to provide some description for your inputs and outputs so that when you're in a Quartz Composer editor, you see a normal name instead of saying input value one. So for that, you override the method attributes for property port with key, and that will get called by the system. And it will pass, for instance, input value one, and you can return a dictionary that contains the name and the default value, those kind of things.

How do you read and write to those property ports? Well, the beauty of Objective-C 2.0 is that it becomes completely transparent. You want to read from one of those property ports, input value one. You just access it like if it was an IVAR of your class. It's that transparent. It will automatically make a round trip to the port, query the value, and return it, as if you were using an IVAR. If you want to write to an output, it's the exact same mechanism. So there is a little thing to be aware of.

It's definitely more expensive than reading and writing to an IVAR because of that round trip to the input or output port. So if you have a loop, like a for loop or any kind of loop, it is definitely recommended to actually cache the value before you do that loop and use the cached value in the loop.

As a hint, what you may do in your code is instead of typing input value one in your source code to read or write to a property port, you can do self.inputValue1. It works exactly the same. But in that case, when you read your code again later, it will remind you, oh, I'm actually not reading and writing to an IVAR here. I'm accessing a property of myself. And therefore, if you see self.foo inside a loop, then you know you should move that out of the loop.

Now, not all patches want to have a pretty fine number of inputs and outputs. So obviously, we provide a way for you to add custom inputs and outputs. And it's pretty straightforward. You call addInputPortWithType. The type is going to be, for example, Boolean, string number, those kind of things.

And then the key to use for the port, as well as optional attributes, which are the same attributes as the EWD return for property ports, like name, default value, those kind of things. And it's the exact same mechanism to create output ports. Because those are not property ports, when you want to read and write to them, you need to go through an explicit method. And it's called valueForInputKey, or it's called setValueForOutputKey to write to the outputs.

Here's a detailed list. And you can see on the left column, the first one, the type of port in the course of the world, for example, Color. You can see the type of the Objective-C 2.0 property that is required so that your property turns into a Color port. In that case, that will be CG ColorRef. If you create it directly by calling addInputPort, the type would be the QC Port Type Color constant. And if you were to talk to it using setValueForInputKey, valueForOutputKey, the type you would get is CG ColorRef.

You may notice here that we're taking a break from the usual Course Commoder API, which are really flexible. And they would typically allow you to pass an NSColor, a CI Color, a CG Color, pretty much anything. Well, the idea here is that the code you write is executing inside the Course Commoder Engine. So for performance reasons, we need to make sure we minimize the number of conversions.

And if we allow people to write a patch that take an NSColor, the next one that takes a CI Color, well, we would need to convert that all the time, which might still be OK for colors, but as you can imagine, for images, it would be pretty expensive. So we have restricted it to one type of object per type of Course Commoder Port.

The last thing we need to do is when we build our plugin to package it somewhere, and QC plugins, you simply put them into standard Cocoa bundles. The only specificity they're going to have is a special entry in the Info.plist of the bundle that list the subclasses of QC plugin that you have in that bundle.

Then you put that plugin in the proper location, library graphics, course composer plugins. You cannot put them in the tilde library graphics, they won't get loaded here. Only library graphics, because we want for security reasons, have administrator only have the rights to install those plugins. Or you can also load the plugins directly from inside your application if for whatever reason you don't want to install the plugin and make it available for everyone. So you can either load a plugin from an arbitrary pass, or you can have the code mixed with your application code and just called registerPluginClass, class being a subclass of QC plugin.

One thing I did not mention here is that even if you install your custom plugin at the proper location, that doesn't mean it's going to get loaded in all environments. Because in WebKit, for example, we have to have restrictions for security reasons as well, and we're not allowed to load any kind of patch in a web environment. So your custom plugins won't get loaded. But for regular clients, from the editor to applications you write or anything else, they will get loaded.

So in a theory for now, it's time to build our first patch. And what I'm going to build here is a very simple patch to demonstrate all the best principles, something that I call the "iPatch," which is kind of a marketing tool where you start with a regular name and you end up with an "i" name. So it's very simple.

Input string, output string, pure string manipulation. OK, thank you. So we start in Xcode and we create a new project. There are two templates for Quartz Composer plugins. The one we're going to use, the simplest one, Quartz Composer Plugin, that's the name. Those templates are not in the Leopard seed that you currently have.

We'll upload them on the ADC website or the WWDC website as soon as possible, so likely later this week or the week after, so that you can download them and put them at the proper location so they appear in Xcode. So let's go ahead and create what we call the eyepatch project.

And let's start by looking at the structure of the template. So we have on the left side the series of files that are used by default. Here are your two files that define the subclass of QC plug-in. Then you have your usual prefix file so that it compiles faster, which includes by default here the Quartz framework, which in turn includes all the Quartz Composer APIs. Then we only need two frameworks, the Quartz framework, Abusely, and then the Cocoa framework.

Now if we look more precisely at the ipatchplugin.h file, so pretty straightforward, remember, subclass of QC plug-in and called ipatchplugin. First thing I'm going to do is add those input properties we talked about so much. So we need two properties to define the input string and the output string. So the type would be NSString. First name would be input string. And then let me create another one that's of type NSString as well, except it's called output string.

That's all we need to do so far. Then we can look at the .m implementation file. So by default, it's pretty much a placeholder. And you just have to fill your code at the proper places if you want to. You can see the structure here. Every method is implemented, except it does nothing or returns the default value. So you can either delete the one you don't need, or you can leave them as is. It won't make a difference.

So let's start at the top of the file. First thing I'm going to do is change the description of the patch, which is conveniently defined as a preprocessor constant. So let's say a patch that converts any name to an iname. OK. The attributes here are defined directly through the preprocessor constants we have there, so we don't need to touch that method. Then remember, it would be nice to define for the UI attributes for the inputs and output ports. So let's look for the key output string.

And if we have a match, then let's return a dictionary that contains A UI name for that input part. So the key would be "nameKey" and the name, let's say, "name". Now if the key is input string, we're going to do a little more than return an ""m". We're also going to define the default value to have on that part when your patch instance is created.

So let's see, dictionary with object and keys. And my first object is going to be "iName". "It's going to be the official name of our port in the UI. Then we're going to have some default value. And that's pretty much it. And default value, I'm going to define that to a conveniently pod.

Execution mode, that patch is a processor, just like data in, data out, nothing to change. This is what we have by default, processor. Time mode, nothing to change at all. That our patch doesn't depend on the time. We don't have anything to do in inter-dialog. We don't need global resources or anything like that. We can skip it. We don't care to know when the engine is starting, stopping, or enable execution, so we don't touch any of those methods, we leave them as is, doing nothing. And the only one where we need to do something is in the execute method.

So remember, in execute, you take your inputs, process them, produce some result on the outputs. So our output is output string, so you can just write to it, and we're going to have that ising happen, so "String by Append in String" input string. And to make it even better, we're going to capitalize the input string.

OK. Now, we are almost ready to build. Let me tell you first that by default, the Active executable is the course composer editing environment, so that's pretty nice, you don't even need to add it. You can build a debug and release, that's configured. We also provide, as a convenience, a secondary target, that's called build and copy.

So typically, if you don't use that target, you build your patch, and then it's in the build directory, but it's not going to get loaded by course composer because it's not located in the appropriate course composer plug-ins. So at that point, you would either manually copy it there, or you would put a symlink, something like that. To avoid that annoying operation, that target is simply calling the normal target, and after, run a script phase that copies the built result directly into the course composer plug-ins folder. So let's build.

Pierre-Olivié Latour: OK. Now I can look into the Quartz Composer plugins folder, which is located into library graphics, and my """patch"" is there. So let's launch -- well, actually I need to quit it and relaunch. There we go. And I already have -- let's see, a composition to host.

Here we go, my new patch. Just zoom on that. What this composition does is it has simply an image with string to generate the image with some string on it and display it on a billboard. Very straightforward. And now I can look for my eye patch. Whoops, not too fast. Create an instance of it connected there. You can see the name is proper, iname, name. The default value is-- let's wait for the tool tip-- is pod.

And so the result is iPod, but it definitely works. So I can type book, and you get iBook, and you can even invent completely new products like chair or whatever and get iChair. So it's very simple to implement that plugin. And we almost type no code, thanks to Objective-C 2.0. Back to the slides, please.

So that was a very basic demo just to make sure you understand the principles there. Now we're going to go a bit further and do some interesting OpenGL rendering. So what are the conditions here? Well, first of all, your QCplugin subclass is going to have to be of type consumer, because they're the only ones which can render to the destination. If you have a processor or provider, that won't work.

Now, the same way we have those OpenGL, let's say, good citizenship rules, they also apply when you write custom batches, which means use cgl macros instead of touching the current OpenGL context, and make sure you save restore all your OpenGL steps, except the one that are part of the GL current bit state. So that basically means the current vertex position, current vertex color, current vertex texture coordinates, because everyone is changing them anyway, so there is no point in saving or restoring them.

So here's what the execute method of your patch that does OpenGL rendering would look like. Remember, you included the top OpenGL cglmacros.h to get the cglmacros. And then the first thing we would do in execute is retrieve the OpenGL context to render with and assign it to a variable with the name cgl_ctx.

So we simply do that by calling the cglcontext object method on the context parameter. For extra safety here, you can always check-- that should never be null, but you can always do if it's null, return null, fail execution. Then you insert your gl code. Exact same thing as the example for the QC renderer and the QC view. Save the current state, change the one you want to the values you're interested in, then perform your OpenGL rendering, restore the states, check and handle errors. Really straightforward.

Well, the next question becomes, "Okay, I can do OpenGL rendering, but what about images? I would like to draw images, obviously, or get images from the system." So the way you use images within QC Plugin is by dealing with OPEC objects. There are two OPEC objects. The first one is called-- I use a protocol that is QC Plugin Input Image Source, and the second one is QC Plugin Output Image Provider.

The reasons we use protocols there is because we have numerous image types in the OS, you know, NSEmage, NSEbitmapRef, CGImageRef, and so on, and each of them is really suited to a given usage, and Quartz Composer is a really generic system, so none of those was really fitting, and so by picking one of them, it would have a number of restrictions to deal with, or by, on the contrary, allowing any of them, then we would have to deal with, again, impedance mismatch, which can be really expensive. So we solved that problem, and the nice thing we get as a side effect is that we can defer, through those protocols, as you're gonna see in a minute, all computation until it's really needed. So that improves-- improves performance. First thing, input images.

Like I said earlier, those are OPEC objects that simply comply to the QC plugin input image source. The way you would create input image ports is either through Objective-C 2.0 properties, where the type would be ID because it's a generic object and then a protocol, or it would be dynamic ports where you can just create them calling self-add-input-port, where the type would be QC port type image.

Now you have those images, you have those ports, you get images through them. How do you access the pixels to do something with them? Well, we use what we call representations. There are two types of representation. The first one is on the CPU. So the idea here is that when you want to access the contents of the image, the actual pixels, on the CPU you call log buffer representation with pixel format on the OPEC object and you specify one of the supported formats right now that can be RGB8, RGBF for 32 bits float, or intensity 8 or intensity float.

And in case of success it will return yes, which theoretically should always be successful, unless you already logged the image in a different format obviously. But at that point on, you have the image, the pixels of the image accessible in the buffer on the CPU. So you can call regular methods like, you know, buffer pixel wide, buffer pixel high, and the usual base address and byte sparrows to read from the pixels.

Finally, when you're done, you simply have to call unlock pixel buffer represent unlock buffer representation to release that representation. You cannot have multiple representation at a time. You need to unlock the current one and relock a new one. The pixel buffer is read only, so it's very important you don't touch that memory. Only for reading.

The second case is accessing the input images on a GPU. It's pretty much a similar mechanism, except instead of locking a pixel buffer in CPU memory, you're going to lock a texture. And you can specify the target, which can be 2D or rectangle. If you specify 2D and the actual image does not have power of two dimensions, it will be automatically resized to fit.

So you will get your 2D texture, which will allow you to do repeat and various operations that can only be done on 2D textures at the expense of a potential quality loss because of the rescaling that had happened to get a 2D texture out of a non-power of two-texture original image.

Once it is locked as a texture, it's the usual stuff. You can get the width, the height, the target again, as well as the name, obviously. As a conveniency, and also the fact if the contents inside a texture is flipped or not, vertically flipped. As a conveniency, we provide a texture matrix.

The idea here is that if you get a rectangle texture, you have to express the texture coordinates in pixels versus the regular, let's say, normalized units that you use for 1D, 2D, and 3D textures. Also, if the texture content is flipped, you would need to flip your texture coordinates to take that into account. Well, you can do all that transparently and always have to deal with unflipped, normalized texture coordinates just by loading a texture matrix on the OpenGL texture matrix stack.

And then your coordinates are sent through that matrix, which will do the proper transform before producing the texture coordinates, producing the real coordinates to use for the texture. You don't have to use that mechanism, but it simplifies your life if you do. So we provide a code that will return a texture matrix that you can directly load on OpenGL. If there is no need for a texture matrix, this will return null. The same way you cannot modify the contents of the pixel buffer on a CPU, you cannot modify the content of the texture.

Then the third case to look at is producing images out of the plugin. So once again, we have an OPEC object, which is going to have to comply to the QC plugin output image provider. But you're the one creating that object, so you can return any object as long as it conforms to that protocol. The way you would create output ports for output images, exact same thing as for input images, Objective-C 2.0 properties, or dynamic ports.

So let's look a bit closer at that protocol output image provider. So it's going to--your OPEC object that you return, that you built and return, which complies to that protocol, is going to be curated by the engine whenever necessary. And it's going to be responsible for providing pixels for the image on the CPU, on the GPU. So it's the inverse mechanism that getting image in your plug-in. There are two methods that are generic on that protocol, just the dimensions of your image, pixel wide and pixels high.

Then, once again, two cases, CPU and GPU. In case of CPU, your provider object is called, and the engine is going to ask what are your supported buffer pixel formats? So if you're able to draw your image into, let's say, an RGB8 format, you will return RGB8. If you can support more than one format, you return more than one in SNES array, and you can order them by order of preferences. If you don't support rendering in CPU, then just return an MTRR or NIL.

If that condition passes, then the engine is going to call when it actually needs the actual pixels, render to buffer, giving you a base address and a row bytes and the actual pixel format of that buffer. And at that point, you just access the buffer and write to it and put your pixels in there.

In the case of producing pixels for the image on a GPU, it's pretty similar. Pierre-Olivié Latour as well say, I'd like to render into a floating point buffer on the GPU, if that's possible. So that's the point of that method, support a drawable pixel format. Because we're in the OpenGL world, to be able to render, you might also require from the OpenGL context a number of extensions to be supported. So you can check for those, and return yes, if you can work with that context. And finally, if the above conditions are met, render with CGL context is called.

The context is ready to be used with CGL macros, as you can see, and you just draw, and the viewport is already set, and so on. The projection and model view matrices are set to identity. You just render. Remember, very important, preserve the stats you change, always. And at that point, you're done. So you don't have to implement both CPU and GPU method, but obviously, you need to implement one of them. So let's do a demo of OpenGL rendering.

In that case I'm not going to build a patch because there is a bit more code, especially GL code, so I'm going to show you a project that's already written. OK, so this is our usual plug-in project. The difference here is the inputs and outputs and the execution method compared to the previous one we wrote.

So you can see here dynamic properties that define the inputs of our plug-in. I should mention first that our plug-in is just drawing a square with an image in it and colorizing the image. So the inputs we need are x and y for the position of that square on screen, as well as the color to modulate the image with. And finally, we need an image.

Now let's look at the actual implementation of that file. By default, Quartz Composer, the project template in Xcode already has OpenGL, CGL macros at the top. So you don't need to bother with that if you have actually forgotten. So same thing here. We define a nice name and a description that's going to go in there in the attributes.

Attributes for property port with key, exact same thing like we saw before. Check for each of those keys and return an appropriate set of attributes. Execution mode, that's the difference. Remember, we have to be a consumer to be able to render. We don't depend on time, so it's still known.

Then to make the code clear, I deleted all the methods that were actually not implemented. So we're only left with one, which is the execute method. So let's rapidly look at it. The first thing we do is we retrieve the OpenGL context to draw with. And for extra safety, we check to make sure it's not null, because I should have mentioned before that when you actually use CGL macros and CGL context is null, you will crash. So you want to make sure it's not null.

Here, for convenience, remember it's-- when you access the image many, many times, you might as well cache it in a local variable rather than read from the input property. So that's what I'm doing here. I am copying the image on the input into an image local variable. And now it's time to get to the real thing.

So what we want to do is we have an image as an input. We want to get an OpenGL texture out of it. If this image exists, and if we can successfully get a texture representation of type rectangle here, then let's just put the texture name inside another local variable. Otherwise, we put zero.

Then we set up our model view to do the translation. And remember, we have to save the previous model view matrix. If a texture does exist, if we do have a texture, then we configure the texture unit in OpenGL to enable texturing, to use the texture corresponding to the texture name. And if we need a texture matrix, we just load it on the texture matrix stack.

We're almost there. One more configuration we need to do is set the current color, so we can get-- remember, input color will contain a CG color ref object. So we can get the components out of it, and it's guaranteed to be RGBA components. So we retrieve them, and we can directly set the current color, which we don't need to save, because it's one of the few stats that is part of the current stat bit. Everything is set up. At that point, we're ready to draw.

So we draw a quad, specifying the four vertices of the quad, as well as the texture coordinates for each of those. Because we know the texture, we actually load the texture matrix that is provided to us. We don't need to deal with rectangle texture coordinates and specify the coordinates in pixels or anything.

We just specify normalized coordinates for the texture, like 0 and 1, and it just works. So if you were to change the constant above here to be 2D instead of rectangle-- you would just replace it there, and it would still work. You wouldn't have to change that code here.

Finally, our quad is drawn, and we do the inverse operation, restoring the previous stats. So if there was a texture name, we unload the texture matrix. We restore the original state of the texture unit by disabling texturing. And we restore the model view matrix, and finally check for OpenGL errors. And don't forget to unlock the texture presentation at the end. So it might seem a bit complex. But all of that is OpenGL code. The real QC code is very simple. Let's build a project.

Okay, let's run. Yeah, should be installed. Just double check. Yes, it is. Create a new document. Look for our brand new Square patch. Create an instance of it. As you can see, X, Y color and image inputs. So let's take an image. Okay, and I can obviously change the color and the position as well.

So that was a simple demo of how you would implement OpenGL rendering. Well, obviously you would start from that and then replace with your own OpenGL code, however complex it is. Let's go to the -- oh, before we go back to the slides, I wanted to show you a more complex plugin that I wrote the other day. So it's a hide-fill plugin that does a lot more complex GL operations. But the QC side of things is still very simple. Let me open that composition here.

So what I have in that plugin is, we start-- well, let's look at the composition, and then I'll explain what the plugin does. So we start with the video input. We get the image out of that. Then there is a custom Core Image Kernel that extracts, computes the luminosity out of that image. So you get basically a grayscale image representing the luminosity.

Then we have another image kernel that is going to take the luminosity and build a new image where the RGBA components, instead of representing colors, are representing vertices. So R maps to X, G maps to Y, and so on. So you get an image where the color components are actually XYZW, coordinates of vertices. So we're building just a mesh of vertices stored in an image, in a way.

And what happened is, we take that special type of image, and it goes inside-- well, here I have a trackball so that I can rotate-- but inside, it's basically passed to the custom patch I was talking about, height field. And here inside, you have some fairly complex GL code that is getting that image, copying it into an FBO, then going from the FBO to a VBO, and basically processing a mesh of vertices.

So the whole operation from processing the video captured frames to building the vertices in real time to generating the mesh, it's completely done on a GPU. It doesn't leave the mesh. It leaves the GPU. So that's fairly advanced OpenGL. And that is to show you here that you can really leverage the Quartz Composer environment to do complex GL code, where you don't have to deal with all your stuff.

Like, now I need to write some code to capture the video input, I need to write some code to import images and create textures. You can leverage all those facilities provided by Quartz Composer to do fairly advanced GL. So you can kind of see it here that it's basically-- well, I need to rotate it, but you might see my face in the mesh. There we go.

So it's a mesh of about 16,000 vertices and a bunch of triangles as well. Here you see on the bottom left the result of the first Core Image Kernel that is computing that luminosity and here you can see that special image where RGBA are actually vertex positions in a kind of normalized in a 0 to 1, 0 to 1 space in X, Y, and Z.

All right. To learn about advanced OpenGL techniques like this one, I would recommend you definitely go tomorrow to the GLSL session. Now it's time to look at the final part of writing custom plug-ins, which is using internal settings. The idea behind that is you cannot have a plug-in where necessarily all the inputs and parameters needed by that plug-in are defined through input ports.

For instance, you write a custom plug-in that is going to access some hardware device, and you might have a number of parameters for the configuration of that hardware device or the way you communicate with it, and you simply don't want to have those settings available on the inputs because you don't want them to be animatable or it might just not make any sense at all. In that case, you have what we call internal settings for the plug-in.

And the way people are going to be editing those is by going through a custom interface, which appears in the inspector of the editor. So in that schematic drawing here you can see the RQC plugin which gets some values from input ports, produce results, while the internal settings are set from the inspector and the communication between the plugin and inspector happens through key value coding.

So how exactly do we implement internal settings? Typically, you would define IVARs in your class, or even better, Objective-C 2.0 properties. Then you make sure those are accessible through key value coding-- remember, value for key, set value for key. You also need to list the keys corresponding to those internal settings by implementing the method PluginKeys, which return an array containing those keys. We need that list of keys, because then we can do a number of automatic things. Like for now, we do automatic serialization of the values for those settings. But we can also do in the future automatic undo, redo, those kind of things.

Here's an example of internal settings. It's the same subclass that I had earlier in my slide, except I added two Objective-C 2.0 properties that you can see at the bottom to store my internal settings. In that case, I have a system color. So just--it's completely arbitrary for the sake of that example. So an NSColor, you know, kind of thing, and another one that would be a completely opaque object, and let's call that system configuration. We need to declare the corresponding keys, system color, system configuration, when implementing the Plug-in Keys method.

Now how does serialization work? Well, remember, you have your plug-in, you have your internal settings, and obviously when the user creates an instance of your plug-in and then save the composition file, you want those internal settings to be saved, and you want them to be restored when the composition file is loaded. Well, we do that automatically for you as long as the object you return for that internal setting complies to NSCoding.

If it doesn't, you will have to override SerializedValue for Key and set SerializedValue for Key methods, which basically are used to convert an arbitrary object into a value we can store in the composition file. And that has to be either nil or it has to be a class, like NSString, NSNumber, and so on.

Here's an example serialization still leveraging our previous class interface. Remember, the first one we had as an internal setting was system-color. Because it's an NSColor type of object and it already conforms to NSCodding, there's nothing to do. However, for the second one, that special configuration object completely opaque, well, it doesn't conform to NSCodding, so we have to handle the serialization ourselves.

So we override serialize value for key, check for the appropriate key, and for the sake of the example here, I'm assuming that this object can return some kind of just binary data of its representation. So we just return that. It's an NSData, so it's PLoS compliant. We're fine. Then on the other way, when the composition file is loaded, setSerializeValueForKey is called, and we need to override that, check for that custom key, do the opposite operation. We got that kind of data object, so we recreate a configuration object from it and set it.

Now, the user interface, how does that work? Well, you are going to create an NSView with a number of controls inside to edit those internal settings. And it's displayed in a settings pane of the Inspector. This NSView is going to be managed through a QCPlugin ViewController instance. So the controller, it's your usual model view controller mechanism in Cocoa, is going to act as an intermediary between the plugin instance, which is the model, the view, which is your view full of controls, and the controller in the middle, the QCPluginViewController. And it also handles all the loading of the Nib file and those kind of things.

The way you return the controller is by implementing that method, which has to return a brand new controller object for your plug-in instance, and it's called createViewController. Typically, you would not subclass QC view controller, you only need to do that if you want to do some fairly advanced things.

But for now, I'm assuming simple case-- just return an instance initiated from self, your own plug-in instance, and as well as providing the name of the Nib file to use. Now the Nib file, how is it going to work? Well, we use Cocoa Bindings to simplify our life. So whenever possible, your controls are simply going to be Cocoa bound to the plug-in settings.

So using the plugin.xxx, where xxx would be simply the key corresponding to the internal settings. If you want to use the more-- the kind of the former target action communication model, it's completely doable. You would just have to subclass QC plugin view controller. So practically, how does that work? We start with our QC plugin.

It returns a QC plugin view controller which is going to communicate through KVC to set and read the internal settings. That plugin view controller is loading a Nib file and it makes itself the file's owner of the Nib file. Then the plugin view controller as the file owner has a view outlet which you need to connect to a view in your Nib file that contains all the controls.

And finally, all those controls use Cocoa bindings bound to the QC plugin view controller. And that's pretty much it. So in that final demo, I'm going to quickly build some kind of simple text thing where you have the ability to enter a long text in the interface in the inspector of Quartz Composer editor and then this is just produced as an output.

Okay, new project. In this case I'm going to use another template which is the one which already provides support for all the UI. We need files and everything and let's call that simple text. Alright.

Which will be the string that we-- the text and the string that we actually produce, so output string. And my internal setting is going to be--

Now I can go to the implementation file. So, simple text, I'm going to change here. Usual attributes, so let's do if key.

Alright, default name, and I'm just going to retrieve... I forgot to copy that... All right, name key. Execution mode, processor, nothing to do here. Time mode, still not in it. So remember, we have an internal variable, an internal setting that is storing our text. So let's give it some default value.

This internal setting is stored using an Objective-C 2.0 property and when your class is dialogued, your instance is dialogued, you need to set it to nil. If you don't do that, then it will basically leak. And you cannot release the text because it's a property, so you just set it to nil, which will trigger, you know, replace the property by the nil value and release the previous string that was stored in there. To plug in keys, remember we have to list our keys here. We have only one, r with object, text.

We don't need to change the serialization because NSString conforms to NSCoding, so nothing to do here. CreateViewController is already all implemented for us, nothing to do. And the only method we actually care about is execute, where we do output string. So we do it very simply, we take the text that we have internally and we copy it to the output string.

We're almost done here. The last step we have is edit the Nib file. So the templates provide a Nib file that's already ready with a view. You can just put your control on. The view is already, the file owner is already configured to be of class QCPluginViewController. Its view outlet is already connected to the NSView, so everything's pretty much ready. Let's add an NSTextView.

configure the NSTES view to not have multiple fonts. And let's see, small sliders, there we go. And finally, we're going to bind the content of the NSTES view to the internal state of our plugin. So we make sure we select the actual NSTES view inside the NS scroll view. Then we go to bindings. The value here is going to be an NSString if the NSTES view does not have multiple fonts enabled, which is exactly what we want.

So we bind to the file's owner, which turns out to be our QC plugin view controller. Then the key pass is going to be plugin. Then at that point, we retrieve the plugin instance. And remember, the internal setting is text. That's our internal setting. So we just bound the contents of that NSTES view to our internal variable. And the last thing we need to do here is make sure it updates continuously. So let's save that and build it.

All good. Make sure it's in there. It is. Simple text. Let me hide that. Hide this. Quit this. Okay. And now I have a patch that's conveniently pre-populated. So this patch is-- sorry, this composition has an imagery string and applies some bloom effect on the text before displaying it on a billboard so you get a nice kind of look. And let's create the instance of our simple text. Oops. Oh, I might have to quit it. Yeah, it's there. Simple text.

Pierre-Olivié Latour: Here we go. It wasn't loaded, I guess. Connect to the string input. So now I have my initial value, which if you remember was Bonjour. What becomes interesting is I can go show the inspector, go to the settings pane, and now I have my text view with the text. And I can type some text here, and it's all real time. So do a nice thing with the effects, you know.

I can save the composition, quit, and reopen it, and a text is there, which shows that serialization worked perfectly fine. Back to slides, please. So that was all for the advanced Quartz Composer session, pretty intensive context. Remember that there are many advanced features you can leverage out of Quartz Composer. Quartz Composer is definitely more than just to do pretty motion graphics. You can do pretty professional work with it, especially if you leverage the advanced feature of the QC view or use the QC renderer to integrate into your own imaging pipelines.

And when you reach the limits of Quartz Composer, the Composition Loader will allow you to build pretty complex compositions by dividing them into sub-composition and a master one. Or you can even write your own patches. And we've said it's not that difficult. All the basics of writing your own patches is really simple. So you just have to, in a way, put your code for the actual patch in the pre-populated templates, and you're good to go. Thanks for coming.

I don't think we have obviously time for Q&A. The one thing worth noticing is that we have a number of labs regarding Quartz Composer. There will be Quartz Composer folks at the lab tonight starting at 6:00. Our official Quartz Composer lab is tomorrow at 3:30, so there will be a number there. And finally, I would like to remind you that we have a great developer mailing list called [email protected] to which you're welcome to subscribe. Thanks.