Good morning everyone. Welcome to the Quartz Composer Overview session. This is going to be the first of our three Quartz Composer sessions this year at WWDC. I'm Pierre-Olivier Latour, Engineering Manager of Quartz Composer at Apple. Can you go back to slides please? Thanks. All right, so what are we going to learn today in these overview sessions? Well, first of all, the basics of course of Composer, how the technology works, what is it made of? We'll look at building a simple composition as an example, then we'll look at where you can use those compositions out of the box inside Mac OS X, how to leverage the new composition repository that we're introducing in Leopard. We'll also have a rapid look at how you can integrate compositions directly inside your applications using our APIs, interface builders and so on, and we'll finish on an overview of what's new regarding course Composer in Mac OS X Leopard.

So first of all, what exactly is Quartz Composer? Quartz Composer is a technology that's deeply integrated into the US at a fairly low level. It's completely built around OpenGL, which gives us high performances and a great visual quality. Now what we do with that technology is we use it as a visual synthesizer, and we build compositions.

And compositions, we're going to learn a lot more about them in the upcoming hour. But for now, you can just consider composition as some kind of black boxes that have some input parameters and produce some output results. Those compositions can also interact with the user. And there are various possibilities, like the mouse, the keyboard, tablets, media devices. It's actually pretty large.

A big difference between this technology and a number of other graphics technology we have in the OS is this one comes with a full featured graphical editor that is part of the DevTools. So if you install the DevTools on Tiger Leopard, you will have the Quartz Composer tool.

And finally, we have made it so it's very easy to integrate Quartz Composer inside your applications. I mentioned earlier that Quartz Composer was a really integrated into the OS technology. And what it does exactly is it leverage a number of graphics and non-graphics technologies. So as you can see on the slide, we have typical graphics technologies like OpenGL of course, but also Core Image, Core Video and so on.

And a bunch of non-graphics one that can be, you know, Safari, RSS, JavaScript and so on. All those technologies are integrated through Quartz Composer which you use to build compositions. And then those compositions are in turn used everywhere into the operating system system or inside your own applications.

We have a number of clients that are using Quartz Composer one way or another in the system or applications we ship and there is also of course all the applications you guys have made. So for my first demo, I would like to show you-- can you switch to a demo machine, please? I would like to show you a number of compositions so that you get an overview of what exactly are those things.

So let me start with those two compositions here. So those are your typical motion graphics. So it's animated, it has colors, and it's moving with time, and so on. We have another one here that is a little more complex than the first one here. This one has text.

And you can see that there is also another text in the background, and that text is blurred and has a real-time blurring effect that's going on it. Now the first thing to notice with compositions is that I can actually resize the window. "And the rendering remains sharp. I can go full screen and the rendering will stay perfectly sharp. And that's a very important thing about compositions is the fact that they're visually, they're independent of the rendering destination size.

A usual way to do animations is to use particle systems. So Quartz Composer has a simple particle system that's built in and I just want to show you rapidly two examples. This one is, you know, a typical kind of fireworks thing made with a particle system. But you can do completely different things with particle system. For example, this blobby object that's moving around, it's also a particle system.

I mentioned earlier that compositions can have input parameters. So what I can do here is I can display them and this composition has a number of settings and take a primary input as an image. So let's use a "I have a demo image for that and now what I can do is control all the parameters of the composition to control how the effect is applied.

So in that case I can apply a glow, I can kind of warm the image, control the fading, those kind of things. So that's the second big point about composition. They're not completely opaque black boxes. They're actually, you can communicate with them and control the inputs." Among the technologies that I mentioned earlier that are fully integrated inside QuartzComposer, there is Core Image, obviously.

And you can use QuartzComposer as a development environment for Core Image. You can write your own Core Image kernels. So I have two examples here that were made by Apple engineers. The first one is kind of rotating gears, and it's completely done on a GPU using Core Image.

It's not two images that are rotating around or anything like that. It's computed on a GPU and drawn on a GPU. I have another example here that's great. It's July iteration fractal thing. And it's also done on a GPU. And it's a Core Image filter that is written inside QuartzComposer.

I mentioned you could use various input devices, the biggest of all being the camera that we now have on a number of Macs. So here I have a composition that is actually using the video camera that's on that machine and a number of effects with core image and feedback rendering and so on and you can achieve some pretty cool effect with that.

Still on interactivity, I wanted to show you another one. So here I have a regular tablet and what I can do with it and the pen is I have a composition that reacts to pressure sensitivity and also to the speed and so on. And we use that to do some rendering in Core Image with feedback and you achieve that very nice trailing effect.

And to conclude that series of demo compositions, I'm going to show you another interactivity that's possible in Quartz Composer, and in that case, it's using MIDI. So here I have a regular MIDI keyboard with a bunch of controls on it and various parameters of that composition have been just wired to the MIDI. So now what I can do is by tweaking the sliders and everything, I can control the color, the size as well as the rotation, everything regarding that composition. Back to slides please.

All right. So now how exactly does Quartz Composer work? How is it conceived? Well, at the base of Quartz Composer, we have what we call patches. They're the basic processing units. That's what we use to build those compositions. So a patch is something that is executed by the Quartz Composer engine. It gets some input parameters through input ports, and it will produce a result either by producing that result on output ports or by rendering it on the destination, which typically is the screen.

So you can say that those patches are like programming functions. They take a number of inputs. They might also depend on time, and they produce a given result. Let's look at a few examples here. So the first patch here is LFO, which stands for Low Frequency Osteator. So it means it's a patch that produces a wave that's oscillating with time.

So typically, it will depend on time. Its result would be the height of the wave at a given point in time. It will have parameters like the type of the wave, if it's a sinusoidal wave, a square wave, triangular, and so on, as well as the period and the amplitude. Now, not all patches need to have inputs and outputs.

For instance, the mouse patch here only has outputs, because its goal is only to output the x and y position of the mouse. The other way, you can have patches that only have inputs. So the sprite patch, which draws a quad on screen with an image, has two inputs to position that quad, one to receive the image. to draw in that quad, as well as a color you can use to modulate the image.

So all those patches receive the data and potentially produce data through inputs and output ports, and we support a number of types. We support values that can be numerical values like Booleans, floating point values, string, and colors, but we also support more complex objects. Two of them specifically, a structure, which is a generic data container, like an NSRA, NSDictionary, that kind of thing, as well as a generic image container, and that's one of the most powerful things.

I'm saying generic image container because it doesn't matter what the image contains. It could be texture on the GPU, it could be an image on disk, or whoever knows what, but what you manipulate inside Course Composer is an OPEC image container. I'd like to show you now the basics of Quartz Composer and a rapid tour of the editor.

So let me launch the editor. The editor is located inside the developer applications folder in Leopard and located in developer application graphic tools on Tiger machines. I'm going to change a quick thing here and create a blank document. So that's the Quartz Composer environment. The biggest thing is obviously the viewer on the right here, which renders the composition, where you can visualize in real time what the composition does. Then we have the editor on the left side here, which is where you assemble the patches we just talked about.

This editor is made of a number of pens. The biggest one is the workspace. We assemble the patches. And then here on the left, you can see a pretty big list of patches, and we have probably 150 to 200 patches right now. And core image filters and Quartz Composer built-in patches and so on.

So what I'm going to do here is build a simple composition to demonstrate to you the principles of Quartz Composer. Now for extensive knowledge on how to build compositions and way more complex than that, you would need to go to the Wednesday session of Quartz Composer in the morning, which will be a hands-on session and cover deeply how to build compositions and build efficient ones. So the first thing I'm going to do is paint the viewer with a gradient.

So I can search in the list of patches for a patch that's going to allow me to paint a gradient. So here it is. You can see-- let me just zoom in-- you can see that the gradient has a number of inputs. It should be pretty straightforward, you know, start color, middle color, end color, and so on. To edit one of those inputs, I can just double click on them. So I'm going to set it to blue. Edit the three to various shades of the same blue.

Okay. Now what I'm going to use is an image. I would like to draw an image on top of that gradient. So let's import an image. In our case, the one of the new Quartz Composer logo. And there are many ways in Quartz Composer to display an image, but I'm going to use the simplest of all, which is using a billboard patch.

The billboard part has inputs like, you know, what's the image to display and where to display it. I have my image in that patch here and I got the billboard on the right there. So what I need to do now is connect the image output to the image input of the billboard. And now the billboard has the image it needs to display. What I'm going to do here is slightly move it to the right.

So that's better for what's coming. Then a cheap way to add animation, like I mentioned at the very beginning, is to use a particle system. So let's add an instance of the particle patch. The particle system, for those of you who don't know, is a system of particles, like the name implies, that are actually evolving with time.

And the evolution happens taking into account a number of parameters, like the velocity of the particles, their sizes. And then particles can disappear after a certain amount of time, and gravity is also taken into account, and so on. Right now we see a number of white boxes that are moving around. Those are the particles, but they don't have an image on them. So let's import an image. Same technique as before.

And the first thing I'm going to do here is change the way the images are composited on top of each other. There are several what we call blending modes, and in that case I'm going to set it to "Add." So now we get a nice saturation effect and the blackness of the image has disappeared.

When you want to edit a number of parameters in the editor, you don't have to double click on every of them. We actually have an inspector with a number of pens, and one of them is input parameters, and it lists in one single area all the input parameters of the selected patch.

So here I'm going to do a couple of changes. First of all, I'm going to remove the attraction so that the particles just fly away. I'm going to add some gravity so that they slowly fall down. And what I would ultimately like is to have the particles emit from the top of the WAN and go towards the left side, like that. So I'm going to have to control the initial velocities of those particles.

Now I'm also going to want them to be, to go higher, so that means make sure they don't start going with a velocity that would make them going down. That's good enough. And finally, we don't care about Z, so I'm just going to-- we don't care about the depth, so I'm just going to have them go towards the left. Now what I can do is position the origin of that particle system approximately on the top of the wand. So here we go. Something like that.

Now the last thing I can do is add some colors. If I click on the color input, then I can see directly the color picker where I can select, you know, various colors. But ideally you would like to have that changing with time so that it's automatically animated. Here's the color input. What I want to change in the color is the hue. I don't care about the other components. I want them constant. Fortunately, we have a patch that's called HSL Color that takes hue saturation and luminosity inputs and builds a color out of that.

So I'm going to connect the output of the color patch of the HSL color patch to the color input. And now just reduce the luminosity to 60%. Probably going to be good. And now I can play with the U. And you can see all the colors are changing in the particle system. There is one last step to make that automatically animating.

Basically what I need is to use a patch that's going to generate a value between 0 and 1 that's continuously oscillating. So the LFO is a good thing for that. And its default settings actually generate a value between 0 and 1, so it's all good. And you can see here what's happening.

So now my particle system is starting to look pretty good. What I have to do is add some text, and it will look even better. So how do you add text in Quartz Composer? Quartz Composer is ultimately built around images, so we're going to generate an image which has text inside it.

And there is a patch that's called Image with String that does exactly that. How do we display an image while still using the billboard as it's the simplest way? Image Restring provides us a convenience output which is the optimal width at which to display that image. So I'm just going to make that as well. And now I can change a couple of things on the text. Namely the font, so I'm going to set it to Lucida Grande, bold, let's see. Here we go. Change the character size to a little bigger and then just move the image slightly to the left.

And to have that text be a bit less boring, I'm going to add an effect on it. We have all the core image filters available in Quartz Composer, so I'm just going to pick one. Keep its default setting, the Bloom filter. And now, I have my texture. So remember, at the very beginning, I mentioned that one of the very important things about composition is that they can have input parameters. So how do we add input parameters to that composition? Well, it's pretty straightforward.

First of all here, what could be our input parameters? Well, the text is an obvious answer, but there is also the general color of the background. So let's make those two input parameters. I can control click on the ImageRisString, and there is a contextual menu that appears. And in there, I can select the input I want to publish to the upper level, and that's going to turn it into a parameter of the composition. So I'm going to select the string input and read a text. And now, if I display the input parameters of the composition, I have a text here, which I can change to whatever I want, and it's becoming an input of the composition.

The Gradient is an interesting case because it has three color inputs. However, that wouldn't be very useful to publish those three colors. What you want is to have a single color and control it and have the two others be built upon the first one. There is a pretty convenient patch that we have called a splitter when you want to connect a single input to multiple outputs. So I'm just going to create one, make it color because that's what I want to move through that splitter, color subject. Connect it to start.

And now I can set the input of the splitter to the same blue we originally had. So we're pretty much back to the beginning, except now I'm going to compute the middle and end color according to that start color. And for that we have a patch that's called Color Transformation. So let's create two instances of it.

Have the color go through them and back to middle and end. And finally, all I have to do is change The luminosity so that each of those two colors is darker than the very original one. And finally, I can take the color input that's on that splitter, publish it, Give it a name, and now I have a composition which has two inputs, text and color. So that concludes our simple composition. I would like to get back to slides, please.

For the next few slides, I would like to do some theory, but be assured it's going to be short just so that we get the principles right. So this is pretty much what we build, a number of patches with inputs and outputs on the workspace. Then we interconnect those patches. Some of those patches are going to pull data from the other one and render something. So you can see an example here in red.

Then what we can do is assemble those patches into a macro patch. Pretty much like you group your functions when you're programming, it's the same concept. What becomes interesting is that we could say for instance that the central patch here is itself another macro patch made of sub-patches whose inputs, whose summing of the inputs and outputs have been published. And are therefore connected to the patch at the above level. So we can go a long way with that and build a hierarchical patch of three, uh, tree of patches, that's better.

Then what you have, since it's a tree of patches, is obviously a root. So that's what we call the root patch of the composition. And this is pretty much what a composition is. It's just a tree of macro patches and at each level they're all interconnected. And they produce a topology from which you have data that goes in, data that goes out, and a result is rendered.

Now how does that exactly execute within the engine? Well, it's pretty straightforward. The engine starts at the top level of the composition. And then it's going to see it's a macro patch. So it's going to go to the lower level to compute all the sub-patches. One of them turns to be a macro patch. So it goes another level down and so on until all is computed, and it can go back up and produce the final result. And how are things executed exactly at a given macro patch level? So here's a typical example.

You can see we only have five patches, a clear one that paints the screen, a sprite that draws an image. And then you can guess that the position of the sprite is determined by the mouse position, and the width and height are kind of oscillating on their own.

The first thing to know here is that not all patches are equal. We have three types of patches. Most important of them are the consumers. We call them consumers because they're the ones pulling data from the other patches, and they're the ones that actually produce the result and render something.

In the Quartz Composer Editor, you will see them, recognize them, because they have that pink color. And another important thing about them is the fact they're rendering a given order. That's important when you draw on screen. So you can see as well on the top right hand corner of those sprites, of those patches, there is a number which indicates the execution order.

Then we have processors, which are pretty much lazy patches. They just execute whenever the inputs are needed and whenever the inputs are changing or the time is changing if they depend on the time. They just do data processing. Pretty basic. The third type of patches we have are providers. So they're required when you want to have data that comes from the outside world inside Quartz Composer.

And they have the specificity that they execute whenever the outputs are needed, obviously, but not more than once per frame that Quartz Composer renders. The idea being that if one of those patches are typically connected to a hardware device, there is no point in curing the hardware device more than once per frame when you want to use the data in the composition.

So now it's time to look at how it exactly executes. So remember, consumer first in the order. So we start with clear. Then we go to Sprite. And Sprite has inputs that depends on other patches. So they're undetermined. They need to be computed first. So the other patches that it depends on are computed. Then when all the results are available, the Sprite can render.

It's now time to look at where you can use your composition out of the box in Mac OS X. And we've got great support in Mac OS X and it's getting even better in Leopard. You can use your composition as screen savers, you can use them as animated desktops, we have full QuickTime support, and we now have also WebKit support, so that means Safari, Dashboard, and so on. Let me do some demos here.

Let's start with the composition we just built. And the easiest way to use the Composer in Mac OS X is simply as a screen saver. So I can open the screen server folder that's located into the library folder, drop the composition into it, go to System Preferences, Screen Servers, And in the part they appear in the other category. Here we go. And that's my screen saver. Now what's cool is the input parameters of that composition are available as options on the screen savers. So here I can change that. Whatever. Okay.

Oops. Here we go. And that doesn't touch the original Composition file. Those are just input parameters that are saved as preferences. New in Leopard, we can use the same composition as Animated Desktop, which is pretty cool too. So here we go. See, now you have those nice obnoxious backgrounds now all the time when you work. And it can get even better. So for instance, I have a great composition to use here as an animated desktop.

It takes a couple seconds to work, but here we go. So very relaxing and you know it's really animated desktop. The whole thing is working fine on top of it. Except I'm not going to keep that around for the demos because I want you guys to focus on Quartz Composer and not on some ducky. So let's see. Here we go.

All right, so those are pretty easy way to use Composition, but that's not very professional. I think we all agree. And we have professional support kind of thing for Quartz Composer. So I can still use the exact same Composition we built, open it in QuickTime, And it's natively supported. I can resize. Remember, composition is independent of the rendering size and QuickTime supports that. So when I go full screen, it's still perfectly sharp. I can even control the composition playback, you know, back and forth in time.

When it gets even better is what if I save the file? You can see Self Content Movie, they're saying here About 170 kilobytes. So for something that's resolution independent that you can use in any application that supports QuickTime, that's pretty powerful. I'll let you imagine the size of that file if you wanted to have it in, let's say, 1600 by 900 in, you know, 25 frames per second, full quality. It would be gigabytes of data. Here, whatever the length, it's going to be-- the duration, it's going to be 170 kilobytes.

Another support that we added in Mac OS X Leopard is WebKit. So I can just launch Safari and still use our same composition. And it will play within WebKit browsers. Well, that's pretty limited, but hopefully we can do a lot more than just playback. You can actually control the composition from JavaScript. We have a full JavaScript API that allows you to control the playback and as well as communicate with it.

A typical example is Dashboard. That's a great place where you can do amazing widgets using Quartz Composer. So I got a few demos here. Let me start with this one. It's a simple widget that's rendering a composition. You can see it's interactive, the glowing effect, transparency, all the things you can get with Quartz Composer.

This other one here is a bit more complex and it's rendering a graph, it's a very simple graph, it's basically a proof of concept and the data, the names that are displayed in that area as well as the bars, the height of the bars is coming directly from JavaScript. So you can start seeing that you can do pretty powerful AJAXing with JavaScript doing data processing and so on, giving the result to the composition for display.

As another example of that concept, I would like to show you this photocast. Let's hope we have an internet connection. OK, we do. What this one does is there is a JavaScript back end that is retrieving some images from the internet. And then they go through the composition where some filter is applied. We made it very simple in that case, just a CPR.

And the result is displayed into that album kind of thing, where each of those subviews is a composition. Now each time I actually show Dashboard again, it refreshes the images. As you can imagine, you can build way more complex widgets than those simple examples. I'd like to go back to slides, please.

Let's talk a little bit about the details of QuickTime support. Well, first of all, it's Micros 10 Tiger only part only. That means no prior version of Micros 10 or no Windows or anything else. If you want to actually use a Composition on the platform, you would have to render it in QuickTime as a regular movie file. We have a few restrictions as well. If your Composition requires user events like keyboard and mouse, those do not get through right now in a QuickTime pipeline, so we're not handling them at the Composition level.

For security reason, we have to disable all Internet communications, so that means if your Composition download images or RSS feeds or anything like that from the Internet, it won't work within the QuickTime environment. And finally, for QuickTime programmers, you cannot access through the QuickTime API at the Composition input parameters. So it's pretty much playback, but remember, we have a great quality.

However, in QuickTime, there is no concept for tracks that have infinite time--I mean infinite duration and no size at all. So we need the track that is created when you open a Composition inside QuickTime is a track of type QDZ for Course Composer. And this track is going to have default dimensions and duration.

It doesn't really matter regarding the dimension when you draw it on screen or display it because remember, it's going to be displayed at the final size anyway. However, if you were to export it or if you were to convert it to another format, then the track size are going to matter and you're going to have to pay attention to that.

Another important point is the characteristic of the track, especially if you want to interleave it with other video materials. The frames that are produced by the track are progressive and use square pixels and the alpha channel is completely valid. So you can do completely valid blending with all the tracks underneath and so on.

For best performances, if you have a QuickTime application and you want to support Course Composer, you would need to use the new API in QuickTime 7, namely the visual context and core video or for high level programmers, QD Kit and HIMovieView. If you don't and use the old API, GWorld and Friends, it's still going to work but it's just going to be pretty, pretty slow.

Another important thing is the WebKit support. It's Mac OS X Leopard, or we also have it in Tiger 10.4.7 now. It's only working in WebKit-based browsers, like I mentioned earlier, the best client being Dashboard. Not all patches are available for security reasons. The same thing, we have a white list of patches that we allow in the WebKit environment.

Typically, we block access to some devices, like the video camera, for instance. What's very interesting and powerful is the fact you can access the composition through the DOM and talk to it through JavaScript API to control the playback or access the input outputs. And finally, we also have transparency. Remember my first example.

Now I would like to cover the Composition Repository. So that's brand new. We're introducing that in Mac OS X Leopard. The concept here is that let's have in a system a central location for Quartz Composer composition files. Anything you put at that location is shared system-wide. Apple is putting compositions in there, a number of effects and animated background and such. And to install new ones, you just have to drag and drop them. I mean, you just have to drag and drop them in the appropriate locations.

It's exactly like a phone folder. And we provide an API and a way for applications to curate that repository. We have a few clients right now at that seat of Lepard. They're PhotoBoost and iChat, which leverage the repository for all their effects. And the client's going to be your own applications.

Now, the compositions that go into that repository, they're a bit special. We're really focusing on the concept of the composition as a graphic module that takes inputs and produce output result. So it has to communicate with the host application through some standard inputs and outputs. Otherwise, you get a composition from the repository and you don't know how to use it.

So we're defining a new thing here, which are composition protocols. And we're going to talk about that just more in a minute. But for now, just consider defining inputs and outputs. We have an updated version of the editor that is building such compositions which are tagged with the proper protocols.

So what exactly are protocols? For those of you who are familiar with Objective-C, it's pretty much the same thing as a protocol in Objective-C. It defines a set of inputs and outputs in the scope of a given usage, and it guarantees that any composition that complies to that protocol will have those inputs and outputs.

For instance, let's look at a transition composition. So a transition takes two images. They're going to come in through inputs of type image, one with the name input image, the other one with the name input target image. And the result is going to be another image that comes on an output port with the name output image, and also depends on time in that matter.

We have a few protocols that are available in that Leopard Seed. Animation, in the simplest of all, it simply defines that you can use that composition that is suitable for background animations or foreground, like screen savers, animated desktop, those kind of things. We have filters, we have imaging, image out, we have transitions, two imaging, one image out.

Using the Composition Repository programmatically is really simple. We introduce a new class in Objective-C that's called QC Composition. And it's an OPEC object that represents a composition in a repository. And it's OPEC because you don't know what the backup is, where the file is, or those kind of details. The only thing you can do with that QC Composition object is curate and access to the usual attributes, name, description, and so on, as well as obviously the protocol list.

But you also have access to a unique persistent identifier, which allows you to refer to that composition later in time. And the identifier is persistent, so you reboot your machine, you move the composition among the various repositories, all those kind of things. You will be guaranteed, as long as the composition is somewhere, that you can retrieve it with that identifier.

To query programmatically the CompositionRepository, there is that QCCompositionRepository class. It's a shared instance. So you just get the shared instance. And then you ask, give me all the compositions that comply to all those protocols and that have those attributes. So the protocols are end operation. So the compositions have to comply to all of them at the same time, because potentially, you can have compositions that implement multiple protocols.

And you can do kind of post-filtering of the results, specifying attributes, like I only want the composition that complied to the image filter protocol, but made by Apple or made by this other manufacturer, those kind of things, so that's what the attributes are for. To make it even easier to use the Composition Repository, we're introducing a new UI, Standard Composition Picker, that you can either embed in your application as a standalone panel version or as an embedded view in an existing window.

Now, you get those composition, QC Composition objects either programmatically or using the pickers. And how do you use them? Well, we extended our playback APIs in the Quartz Composer API so that they can take those QC Composition objects. So the QC view, the QC Composition layer for core animation, we'll talk more about that in a bit, the QC renderer for low-level rendering, it's very straightforward. They all take a QC Composition object.

So let's recap the basics of the Composition Repository. First thing, you want to show the picker on screen so that the user can pick a composition. So we get the composition panel, we configure it to display compositions. Now, you can see here that you configure it to display composition of a given protocol and a subset of attributes.

We don't say protocols because it would be pretty inconsistent from a UI point of view if you were able to display in the panel compositions that are transitions and other that are filters and whatnot. It would be confusing. So you have to settle down on one protocol and have the user select one composition among that list.

Then we need to set a delegate so that the panel can call us back and finally display it. Then in a delegate, you're simply going to have to implement that method. Composition picker panel did select composition, which is called whenever the user selects a composition object. And then you get a QC composition. And from then on, you can pass it to one of the numerous QC APIs, like if you had a QC view around, you would simply do load composition. So let's do a demo of that Composition Repository.

So let me launch this Composition Repository Browser, which is a simple demo app that we built. You can see the picker in its HUD version that I mentioned earlier. And we have a number of compositions that are right now in the repository. Most of them are placeholders. We'll have, obviously, the final set of composition for the final version of Laypart.

But for now, you can experiment with the repository using those. So selecting one of those compositions, they appear here in a standard QC view. Here you can see I can edit all the parameters of the composition, change the color, and the various speeds and so on. We have plenty of image filters that are built in.

As you can see here, some of them are coming from PhotoBush. Some of them are new ones. So I can just select them and then display in the QC view. And I can change the input image to something else. So here we go. Works fine. And it's very straightforward. There's no code in that demo.

and you can browse the composition picker, you know, just clicking letters and so on. And finally, we have a few transitions that we put in the repository. Some of them are coming straight from Core Image. Other ones were added like this cube one or this sliver effect as well.

Now let's look at building such a composition for the repository. The editor has been updated so When I launch it, I can select "New from template" and you will have a template to build an appropriate composition from the repository. What we're going to build in that case is a simple image filter.

So the composition workspace is prepopulated with the required inputs and outputs for that filter. If I display the input parameters of the composition, I can see, obviously, an input image. And there is actually-- I didn't mention that in the slide-- but there is an optional input for the filter protocol which the host application may use to indicate if it's OK to display the filter and can have a preview mode that is lower quality and therefore faster. So we're going to ignore that for now.

So let's pick up demo image here. And if I connect the input image, : I'm going to show you how to create a simple blueprint effect. It's fairly straightforward to build using some core image filters. First one I'm going to need is Color Posterize. Going to insert it in the middle.

And now all I need to do is kind of colorize the result to make sure it's in a single color. And we have a convenient filter for that which is false color. So I'm going to insert that in a chain. And now I can look at the colors and change it to blue and white. And now I have a great kind of blueprint effect. You can go through a few more iterations and improve that effect. So I'm going to show you here the final version.

Okay, so you can recognize here the original color posterized and false color. Here there's some more core image filters on a custom kernel to kind of build those little edges that you can see and improve the quality of the effect. And the other thing that was done here is take the overall blue color and make it an input parameter as well so that we can change that to whatever we want.

Now what I can do with that composition is just take the composition folder that is located in library compositions, take the blueprint, drop it there. Now if I relaunch the composition browser and go to image filter, well now it's just there and you can see the color is there. Oops, there we go. And I can also launch PhotoBoost.

We have numerous ways for you to integrate Quartz Composer content inside your application. The easiest of all is the QC view, which is a subclass of NSU that allows you to do Quartz Composer playback. It also has--it can also be used directly in Interface Builder and has Cocoa Binding support, but that doesn't mean you have to use it this way. You can also use it completely programmatically. We have new in Mac OS X Leopard the QC Composition Layer for fully native integration inside a core animation environment.

And for people who want to integrate Quartz Composer into kind of already existing engines or do low-level processing of composition and control completely what's happening during the rendering, they can use our low-level QC renderer class, and that class will be covered extensively tomorrow during the advanced Quartz Composer session. So let's do a quick demo of that QCview integration.

I'm going to use here the old version of the interface builder because the components we require for that demo, it's not complete yet. So first of all, let's create a blank application. Then the first time you use Interface Builder, you need to add the Quartz Composer palette. So you can do that through the preferences and it's located in, let's see, Leopard Developers There we go. Developer, Extra, Palettes, and Quartz Composer. That's good. All right. Okay, I'll just try it again.

: Go figure. OK. So now I have that palette which showcases a number of elements, but the two we're only interested in today are the QC view here and the QC patch controller for Cocoa Bindings. Let's instantiate the QC view. Oops, I'm just going to configure it to resize with the window.

And what we can have here is in the attributes the ability to load a composition on the view. So I'm just going to click on the load button and load our very initial compositions. Now I can run the Nib file and you will see the composition just plays and it resized correctly and everything. So it was zero cutting.

But we can do something that's more interactive, still without leaving IB and cutting anything. So I'm going to upload the composition and just resize that to 320 by 240. And when you want to do Cocoa bindings, you need a controller. So we have a QC patch controller. So remember, it's the model view paradigm in Objective-C where you have a model that is your data set.

Then you have a controller that acts as an intermediary between the model and the view. In our case, the view is a QC view, the model is the composition, and the controller is a QC patch controller. In the attributes of the QC patch controller, I can load a composition file.

What I did is slightly modified the BluePrint effect that was built earlier so that it's kind of autonomous. And I added a few things here. First of all, there is now a video input patch that grabs video from the camera on the computer. Then the video is processed to improve the quality of the image by doing fairly simple noise reduction.

Then it goes through the Blueprint effect that we built just earlier, the exact same one. And finally, there is a macro here that draws the composition with the frame that you can notice around here. And we also have another macro here that handles the text display in the lower left corner. This composition has two input parameters, a color, the name color, and an arbitrary text with the name text.

So back to IB. On the QC patch controller, I can select that composition file I just showed to you. Then I can go to select the QC view, show the bindings span, and the QC view has a binding that's called patch, and it's simply the patch that the QC view is going to render. So let's configure that. We want to bind to the QC patch controller.

And that's, remember, that's a controller that is loading our composition and so on. What we want to retrieve from it is the top patch of the composition, the root patch, because this is what we want to display, and we can do that simply with the patch controller key.

Now I can run this little nib file, and it's pretty much as if you had loaded the file directly on the Composition. It does the same. When it becomes more powerful is that I can now use Cocoa Bindings to talk to the Composition. Remember, we have two inputs. So let's take a color.

Add it there. And I can bind more of that color well control to the patch controller, retrieve the root patch of the Composition. Then retrieve the color input, and from that color input, get the value. Okay, so now Cocoa Bindings, it just works. I have a second input that was text, so let's build that as well.

And I'm going to have that button perform the copy action on the view.

All right, now I have a little autonomous application that allows me to type text and

I'm sure you're all wondering where I'm going with this demo. Well, I am going somewhere and I'm going to build a little comic here. You'll see how you can build a little application very fast and use it. So let's switch. Some text. Okay. Okay, I'll put that here. Okay. Okay. Here we go. All right.

Live comic, no Xcode, no code, nothing. Okay, back to slides please. The QC view you have in Interface Builder, you can access it programmatically in your nib file. And the API is pretty straightforward. You can control rendering completely, start, stop, pose, all those things. You can load a new composition file on it, load composition from file. In Mac OS X Leopard, we have a new method that's called unloadComposition.

The point is that if you have your QC view around, and you don't need the composition anymore, you can just unload it, and this will save resources from the video card and the CPU and so on. So it's a good thing to do if you can do it.

To communicate with the composition, the API is also straightforward. You can retrieve the values properly, like the attributes, you know, name, description, those kind of things, as well as the list of inputs and outputs. To access the inputs and outputs, you need to call setValueForInputKey or valueForOutputKey. New in Leopard, we have that new method, valueForOutputKeyOfType, which allows you to explicitly request for the output object to be in a given class.

So let's look at what's supported here. Remember, we have a number of port types in Course Composer, Boolean index numbers, and so on. And for each of them, there are a number of object classes that are supported when you pass input objects. For instance, a color input port, you can pass an NSColor, obviously, but you can also pass a CGColorRef and CIColor. And if that color port was an output, you would also be able to retrieve one of those three. When it comes to images, we support a great number of image formats.

Inputs and outputs. What's important here, however, is that you make sure you always pass the original image to Course Composer. For instance, if you have a Core Video Buffer, pass the Core Video Buffer to Course Composer. Do not pass, like, the Core Video Buffer converted to a texture and then pass that texture object. Or the Core Video Buffer converted to a CI image and pass the CI image. Always pass the primitive. It's important. This way, you ensure maximum performances.

Now it's time to look at the new Quartz Composer API and patches we have in Leopard. First of all, the most important request from developers is the ability to write your custom patches. We made it very simple and powerful like the rest of QC and we completely leverage Objective-C 2.0, so that's really a great thing. You're still going to have to wait a little more to discover that.

It will be fully covered during tomorrow's session in the afternoon, the Advanced Course Composer 1. Then we have a number of new patches, one that is the Composer that allows you to load a Composition within Composer Composition. That's also going to be covered in details tomorrow afternoon. And then other patches like GLSL support and so on.

We added a number of methods to the QCVU to be able to do basic things like going full screen, in and out of full screen, retrieve the current image on the QCVU as a snapshot, and what's great here is you can get the snapshot in your favorite image format. So, you know, CG image, CI image, and the QCVU will take care of everything. Downloading the image from the GPU, putting it on a CPU, flipping it vertically if it's necessary, all those kind of things.

We made the QC view more powerful. We added a number of methods so that you can suspend rendering, pause it, and resume it. We have advanced method that allows you to, if you subclass QC view, kind of customize an interface directly with the rendering of the QC view. So that's a pretty powerful feature, as well as the ability to retrieve the internal OpenGL context for sharing or using that subclassing. And those last two points will be covered as well in a two more session.

We made the QC renderer better with the ability to integrate it even lower in your existing OpenGL pipeline so you can now use a CGL context object as well as provide the output color space into which Quartz Composer should render and we only support RGB color space right now.

For those of you who want to do off-screen rendering of composition, well, it's never been easier. You just need to call initOffScreenWithSize, and it will take care of everything-- creating the OpenGL context, creating the OpenGLP buffer. So you don't have to deal with OpenGL at all. You will just create a QC renderer off-screen, and you can get the image out using the same snapshot API that is available on the QC view. You call snapshotImage, or createSnapShotImageOfType, and you get it. One important note about that is createSnapShotImageOfType does not return a notori release object. So be careful about memory leaks here.

Last but not least, we have full integration in the new Core Animation. So there is a new Core Animation layer that's called the QC Composition Layer. Very straightforward to create, just Composition Layer with a file, file being a composition file on disk. And then you can use similar APIs to the one on QC View and QC Renderer to be able to retrieve the attributes of the Composition and communicate with it.

Finally, we have a conveniency that is the new Composition Parameter View. If you have your composition and you want to display in the UI all the parameters at once, no need to build it yourself anymore. You can just use the standard QC Composition Parameter View. Very easy to use.

All you have to do is call setCompositionRenderer on it and you can pass either a QC view, a QC renderer, or a new core animation layer. And it exists in two versions, either as an embedded view or as a standalone patterned version as well. Let's do a demo of those new features.

First one I wanted to show you is a new patch that's really cool. It's called Mathematical Expression. So here I have a basic composition that displays two Apple logos on the bottom left and top right corners and they're actually anchored to the screen. So when I resize the composition, the Apple logo stays in the corners.

It might sound simple, but it's not that simple because in Quartz Composer, our coordinate system, well, we want one unit horizontally to be always the same as one unit vertically. So which means when you resize and change the aspect ratio, well, the coordinate system slightly changes. So you need to do a little computation to make sure the Apple stays anchored.

If you were to do it using the Mat Patch, well, each Mat Patch can do a number of operations, but they have to be sequential and so on. So this is what we would need. We would need two Mat Patches for the X and two Mat Patches for the Y coordinate of the Apple.

Well, I can do the same thing here with the new Mathematical Expression Patch where where whatever the number of mat operation, you just have one. The way it works is you simply type formula and automatically it populates the inputs and outputs of the patch. So for instance, I can add something here.

And now I can go back to the inputs. And you can see-- there we go. It just works. What we did with that mathematical expression is also the ability for you to indirectly use it anywhere. So I can just enter computations. "In any text field where you enter numerical values and it just works. So that's pretty useful." What's even more useful is the Composition Loader I talked about earlier, and I just want to do a quick overview here. Remember, it's going to be covered in tomorrow's session.

So what I have here is a composition that has two composition loaders. What you do with it is you specify the location of a composition on disk, or URL, something like that. And it will load the composition and render it as if it was just embedded in the original composition here.

So in that example, I'm loading the slide composition, which is one of our examples, and rendering it here. And on top of that, I added a particle system, just to show you that an integration is straightforward. What's better is the input parameters and outputs of the composition you load are also accessible. So here, I have the shape input and the color input of that slide composition that are available as well. So that means I can just change the composition, and it all works.

Composition loader is very powerful because you can now take a monolithic composition and explode it into a master one, kind of an assembly composition, and a number of sub-compositions. And then you can have graphic artists work on sub-composition and so on and they don't break anything in a master composition. Everything's independent.

Another patch of interest is the XML Downloader. A number of people have been doing kind of Ajax kind of things but inside Quartz Composer using RSS and kind of formatting the data using RSS and so on. So what we have now is an easier way to do that. There is XML support so you can have here more XML file. It's purely arbitrary format.

What I just did is I built that as a list of images and for each entry you have a pass to a file. You have the name to display as well as a color to kind of colorize the image with. Pretty simple. Now I build that composition here that reads that same XML. Let me just make that a bit bigger.

That same XML file Using the new XML Downloader Patch and you can see it's produced as a Quartz Composer structure. Then you can do all the usual operations with it. You can parse the structure, extract the members, so in that case the location of the file, the text to display and the color to modulate it with it. And then there is some timing logic but ultimately it displays all that thing on screen. So now you have a Composition whose original dataset is just an XML file.

One of the experimental patches we have right now is GLSL support. So I'm saying experimental because it's not fully finished, but you can already play with it and do pretty powerful things. So I got a couple of GLSL composition here. So that's the usual OpenGL teapot. I'm sure you've seen that before. And what happens here is it's rendered with a GLSL shader on top of it.

The way it works is that the GLSL shader in the inspector, you can display the vertex shader and fragment shader and edit it in real time there and all the changes reflect in the viewer window. The GLSL shader patch is a macro patch, so it loads the shader and then everything that you put inside that macro patch is rendered and affected by the shader. In that case, we have just a trackball to do the rotation thing and we have the teapot object.

The same way you use the Core Image Kernel to write your own kernels, here you could write your own GLSL shaders and any changes you do here regarding the uniforms of the GLSL shader, namely the variables, public variables if you want, will appear automatically as input of the GLSL patch. Let me show you another example. In that case, it's doing a simple environment mapping using that source image. Here we go. Pretty simple as well.

I would like to show you now the Core Animation integration. So here I have a simple Core Animation app that is used to browse your-- in that case, the file hierarchy on disk. And the background is, of course, Composer Composition. The way it's added is just a layer in Core Animation world. You create the layer from the Composition file, insert it at the proper place in the Core Animation tree, and boom, you're done. And you got perfect rendering. It's all synchronized with the display rate and all of that. And the input parameters of the Composition are still accessible.

So that's really powerful when you combine core animation with the layering system and you have real time synthesized content that comes from Quartz Composer, especially considering that Quartz Composer renders at arbitrary sizes. And core animation takes that into account as well to make sure the Quartz Composer layer is always rendered at its optimal size, ensuring maximum quality. Back to slides, please.

So we've seen a lot today, but if you were to remember only a few things, which I hope you're not going to do, but remember more, Quartz Composer is really a visual programming environment and you can use it for two things, primarily. Explore the world of all our graphics technologies on Mac OS X. Remember, we have Core Image, we have GLSL, we have OpenGL, all those things you can play with in a single unified environment.

You can use it to create compositions that have input parameters. So we call them parameterized compositions. And those are very powerful and you can use them in your application, communicate with them and so on. We made it easy for you to integrate those compositions in your applications. A new thing in Leopard is the composition repository.

So you should definitely try to leverage that to add visual effects and animations to your applications and let the user pick those compositions. For simple integration, there is all the interface builder, Cocoa binding support and we have those super simple playback APIs with the QC view. And in Mac OS X Leopard, we have the ability to write custom patches, the ability to add your own effects to photo boost, eye chat and so on. We have animated desktop support and the WebKit support as well.

Remember, we have two Quartz Composer sessions coming tomorrow. The first one at 10:30 in the morning will be ""Creating Your Quartz Composer Composition."" The one in the afternoon will be ""Advanced Quartz Composer,"" which is more programming techniques, how to write your own patches and really use Quartz Composer to the maximum. For more information about the Quartz Composer technology, please refer to Alan Schaffer, our 2D and 3D graphics evangelist. As usual, you can get documentation sample code from the WWDC website.

Three labs of interest: the Late Night Graphics and Media Lab, they will be Quartz Composer forks here. It's tomorrow at 6pm. Then there is a dedicated Quartz Composer Lab which is Thursday at 3:30 and there is also a Core Image Lab at the same time in the same place.