Hello. Welcome to session 415, Mastering the Quartz Composer Editor. As I was just introduced, my name's Nathan Wertman. We've got a bunch of things to cover today, so let's just go ahead and jump right in. There seems to have been a bit of a snafu. This session was marked as a hands-on. The assets weren't available, so I'm just going to go through and demo everything. If you guys could just follow along, that would be great. The first demo is going to be new Quartz Composer Editor features. We've added a whole bunch of things to make content creation easier.

Then one of my coworkers is going to come on stage and show new and improved patches. And a third person is going to come on stage for the last section and talk about the improvements to the programmable patches. So what do you guys care about? The new features that we've added for composition creators include a redesigned editor. We've gone through and made it so easy to create content, it's not even funny. An optimized image pipeline. Most of what you deal with in Quartz Composer are images, so we've made small images more efficient, faster images.

We've added a new feature called "Composers" that allows you to add images to Quartz Composer. So you can add images to Quartz Composer and it will handle it. We've added support for custom patches. So if any of you are developers and you realize that there's just a patch that you don't have that you really wish that you did, you can write it. Or you can chime in on the Quartz Composer mailing list and I'm sure somebody would write it for you. We've added a composition repository.

Since Tiger, a lot of Mac OS X clients have started accepting Quartz Composer as the default way to define your image. So we've added a new feature called "Composers". It's a new feature that allows you to define motion graphics in the system. So we decided to make a centralized location where everybody can go to get it. We also decided to make this public for third party applications. So that way you as content creators can create content and give it to all these applications. You no longer need to carry both torches.

So how do you create repository content? It's a simple three step process. You only need to do two of them. The first step is create compositions which conform to protocols. And if any of you were at Alessandra's session this morning, session 606, you've already seen this. A protocol is just an agreed upon set of inputs and outputs.

Once you've created the composition, just put the composition into Library Compositions. And then finally, the applications then take care of the rest. They ask the composition repository for compositions that conform to a protocol, and then they receive a list back. And then they do what they will with them. So let's just go ahead and jump right into a demo of that, if I could have the demo machine.

So the first thing I'm going to do is I'm going to launch the Quartz Composer Editor. And for those of you who are unfamiliar with it, it is in Developer, Applications, Quartz Composer. On first launch, you see a couple of different windows pop up, including our new template assistant. This is where we list all of the available protocols that are currently in the composition repository. We have music visualizers and image filters.

And down here, you see an explanation of what the inputs and output ports are representing. So for this demo, I'm going to go ahead and create an old Western-themed filter that can be used in image, in iChat or Photo Booth. And I'm going to do something a little bit fun with it, too.

So I'm going to go ahead and choose Image Filter. And you'll see when you first look at it that we have-- A bunch of patches on the workspace. You'll also notice that we, if you're familiar with Quartz Composer, you'll notice that we've removed the patch browser from the left-hand side of the workspace. We've created a patch creator window, which is just a floating window that you use to go ahead and add patches to the workspace.

It can also be closed whenever you don't need it. The main problem in Tiger was that you could never close the patch browser. So I'm going to close that for now. And what exactly does the image filter protocol provide us? Well, the darker patches are required inputs and outputs. So here we have a required input image, the image to process, and we have a required output image, the processed image.

We also have a few optional inputs. We have an X and Y optional input here, which is used to center the image, center the effect appropriately on the image. In the case of the boilerplate we have here, it focuses the zoom blur. And we also have a preview mode, which you can take a lighter weight rendering path down. But I don't need that for this demo, so I'm just going to go ahead and remove it.

So the first thing that we need to do if we're doing anything old western is add a sepia tone. So I'm going to go ahead and remove this process image. And if you were paying attention during the previous session, you'll notice that we've added Undo Support, which is completely supported within the editor now. No longer do you need to remember your last 10 steps.

So we're doing a Western theme. So when you think Western theme, you think sepia. So I'm going to go ahead and add a sepia tone. The patch creator here has a whole bunch of hotkeys associated with it. Well, not a whole bunch of hotkeys, a whole bunch of quick functionality. I can close it with the escape key. I can bring it forward with Enter or Command Enter. When I bring it forward, the search field's already selected so I can type what I want.

And there are a few preferences that you can go ahead and check to customize it more to your work style. So I'm going to go ahead and connect our incoming image, the protocol provided incoming image, to the sepia tone. And I'm going to connect the processed image of the sepia tone to the outgoing image. And look at that. We have sepia tone flowers. It's not terribly exciting. I'm going to go ahead and make a little bit more room for myself.

If you look at any old Western photography, you'll notice it's got a bit of a grain to it. So what's a good way to do that in Quartz Composer? It's to add a dot screen. So I'm going to go ahead and add the dot screen to the workspace and connect it up appropriately.

Now this is, it obviously is applying a bit of a grain to the image, but it looks a bit too intense. So we need to change some of the settings on this dot screen. And to do that, I'm going to bring forward the Patch Inspector. And we can change the settings here. But before we change any of those settings, I want to talk a little bit more about core image filters.

So core image filters have, core image filters can have a set of inputs that are expressed in pixel dimensions, such as width. If we are adding this composition to the composition repository, we have no idea how big the incoming image is. So if we were to just leave this width setting to 5.8 like we have it, then images that are only 20 pixels wide are going to look much different than images that are 200 pixels wide.

So in Tiger, the way you worked around this was by doing a little bit of math and calculating the width you cared about as a percentage of the image's width. Well, for Leopard, we've added the ability to just normalize these pixel-based coordinates. So if you check it, that means then, and you can read here in the description, that pixel dimensions come in as a range of 0 to 1, which means that all images are going to end up looking the same.

So to change the parameters, I'm going to bring forward our new Patch Parameter tab, which allows you to see the parameters for the selected patch, or all the selected patches, as well as the macro patch. This is really useful for things like copying colors from one color well to another, as well as just seeing an overall view of all of the settings at once.

For this demo, I happen to know exactly what I want. Go figure. So I'm going to add a bit of a grain, and I'm going to adjust this to-- I'm going to make it a little bit more intense. Okay, so now we've got grainy flowers. And that would be fine for an Old West style theme.

But like I said, I want to do something a little bit fun. Since this is going in Photo Booth or iChat, What I need to do is have an image that represents what I'm actually going to be applying the effect on. Let's grab an image. So I'm just going to drag a picture of our boss up here, Bertrand.

You're welcome, Peter. And now we see that we have Bertrand. So now we can start kind of getting an idea of how this is actually going to feel within Photo Booth. You know, I always thought that Bertrand needed a mustache. I don't know about you guys, so I'm going to go ahead and add a mustache to him. Close the patch parameter view.

And if you notice, those of you who are familiar with Quartz Composer, I'm sure a little while ago you were like, how is this even drawing anything? There's no renderers on the screen. Well, the Quartz Composer Editor understands the image filter protocol and understands that what needs to be displayed is this outgoing image. So it automatically does that for you.

If we were to add a billboard to this level, which is just a simple way of rendering in Quartz Composer, you'll see that it stops drawing altogether. That's because image filters can't have any renderers, or also known in Quartz Composer as consumers, at their top level. That's because when the patch is being run, we don't know what application is running it, and we can't guarantee that the correct-- that there will even be a display attached to the machine while it's running.

So in order to draw a mustache over a picture of Bertrand, we're going to need to have a patch that will allow us to do some compositing, allow us to do the drawing not at the top level, and then output an image. In Quartz Composer, that's the render an image patch.

Now, render an image can use quite a bit of VRAM if it's misconfigured. So for Leopard, we've added the ability to define the pixels wide and pixels high. And we want to make sure to always set those every time you use a render an image. So what I want to do is I want to have this outcoming image have the same dimensions as our source image. So I'm going to grab an image dimensions patch.

Which is basically just a tool that-- let me make a little bit more room here-- which is basically just a tool that when given an image, gives you information about the image, both the width and height in Quartz Composer coordinate space up here and the pixels high and pixels wide. Obviously, we need to use pixels high and pixels wide here.

And I'm going to go ahead and connect the image out resulting up to the dot screen. And you'll see that everything just goes to garbage. It's nice and western-y garbage, but it's still garbage. That's because we're not drawing anything in this render and image patch. Since this render and image patch is a macro, you can tell by the square corners, double-clicking on it will move in. We've added a lot more powerful zooming to Quartz Composer in Leopard, as well as removing the 2,000 by 2,000 window bounds. So now you can actually use all 30 inches of your real estate.

So we're getting garbage here because we're not drawing anything. So the first thing we should do-- and since we're not at the top level, we can have consumers-- is clear it. And since we want to put a mustache on Bertrand, we're going to need to have Bertrand being drawn at this level. So we're going to go ahead and create a billboard. And I'm going to zoom in a little bit.

So those of you who aren't familiar with Quartz Composer are going to immediately see a problem. How can we connect this incoming image of Bertrand to an image inside the render an image? Those of you who are familiar with Quartz Composer already know the answer. You need to publish the input. And you can do that by right-clicking on the billboard, Publish Inputs, Image.

So now when we move up to the top level, going up to the root macro patch in our new hierarchy browser here, you'll see that we now have an image input that we can connect it to. So I'm going to connect that up now. And you'll see we have Bertrand. Moving back into the render in image, you see Bertrand is small. He's not being drawn the full width and size of our render in image.

In Quartz Composer coordinate space, the left edge is negative 1 and right edge is positive 1. So we have a width of two Quartz Composer coordinates. And you can see by double-clicking on the billboard here that the width is 1. We could just set that to 2, but we've added a new feature for Leopard in the settings under the Patch Inspector.

Go to Settings. You'll see that we have dimensions. And we have a lot of dimensions because we found that people kept doing this over and over by hand. So I'm going to go ahead and set it to real size, which just draws Bertrand at full size for the incoming image. So now we're back where we started. We've got Bertrand applied with an old Western theme, but we now have the ability to do our own drawings. So let's go ahead and put that mustache on him. I'm going to grab a mustache out of the assets.

And we need another billboard to put that on, something that we can put over Bertrand's face. So I'm going to drag that one out and connect it up. You guys can't see this, but rather than having to do drag connections for Leopard, we've added the ability to just do a click to connect. So you click on the source port, you click on the destination port. If you hold down Option, it won't break the connection. You can click on multiple destination ports and you can connect multiple things in one pass.

So now we've got a black box drawing over Bertrand's face. And if he was going for a really boxy mustache, that might make sense. But you can see here in the tool tip that this image actually has some alpha associated with it. Actually, you can't see that in the tool tip because-- anyway. Sorry. That's because this image has alpha associated with it. But we haven't told the billboard to respect this alpha and do the correct blending. In Quartz Composer, you do that with the blending input.

And we choose blending over. Now you can see we have a mustache. But it's not properly centered. And while we could drop this composition just like it is in Photo Booth, everybody would have to cream their neck and try to position their head just right to get the mustache in the appropriate place. As I said earlier, we have x and y values coming in from the host application that typically are going to correspond to where you want the effect centered. I'm going to use those to center the mustache.

So if I jump into the rendering image, it's the same issue as getting Bertrand's image inside the rendering image. We need to publish the input ports. So I'm going to publish X and I'm going to publish Y. And I accidentally did that in the wrong order. So you see I published Y first, followed by X. Well, with these two patches being oriented X and Y, this is just a recipe for disaster. In Tiger, I would have to go unpublish those ports and then republish them again. New for Leopard, we've added the ability to reorder published ports.

You can also rename the ports in there if you would like. So if I wanted to change it to Mustache X, you can do that as well. And then I'm going to connect things up. And you should see that we have the mustache moving on his face. But something's not right. It's not tracking the mouse properly.

And that's because protocols, when their numerical values -- when the host application gives a numerical value to a composition, it's normalized in a range from zero to one. This allows an app to know that zero is, in this case, for X, zero is the left corner, Y is zero is the bottom.

So 00 is the lower left corner, 11 is the upper right corner. And they know that if they want it centered, they can just do .5, .5. They don't need to know how the composition works at all. So in order to offset this appropriately, we need to do some math to translate between these two scales.

In Tiger, what we would have had to do is break the connections, insert the math, then connect everything back up to the X position. For Leopard, we've added the ability to just insert an input splitter, which will maintain the published state of the port as well as the connections.

So now we have room in here to do the math without breaking all of these other connections and having to remember how to connect everything back up. So I'm going to go ahead and publish it for the Y. Let me bring forward a new patch called Mathematic Expression. The Math Patch was really powerful, and the Mathematic Expression doesn't do anything that the Math Patch doesn't, but it makes it a lot easier. You can just type an arbitrary equation in here and it'll automatically propagate the inputs and outputs, or the input ports appropriately.

So I'm going to go ahead and type 2 times a minus 1. And you'll notice that it removed the b input on this particular Mathematic Expression patch. You'll also notice the title was automatically updated. There are a certain set of patches where the settings really represent the titles. So we have set up automatic titles for those particular patches.

Things like splitters and the Mathematic Expression. So now we have the mustache centered on the mouse. And that's exactly what we want. So step one is done. Step one is creating the composition that conforms to a protocol. So what I'm going to do now is save it to the appropriate location.

I'm going to save it to-- My home directory, library, compositions. I'm going to overwrite the one that was previously there. Then when I launch Photo Booth and go to the effects, After you get through all of these great background removal effects, you'll see that we have the mustache effect. And I can go ahead and orient things appropriately and take my picture.

So that's really great that it just works in Photo Booth, but if you noticed, one of our other clients is Automator. I've got a picture of Bertrand now with a mustache, but what if I want all of the executives to have a mustache? Bertrand is the, or Bertrand, the Automator is the perfect tool for that. So I'm opening the Automator workflow, and I'm going to drag in my directory of pictures.

Basically, all this is doing is, it's basically just getting finder items, copying the directory someplace, and then opening them up in preview. So here we have all of our executives with a bit of an old Western theme applied to them. Tony's my favorite. Yeah. So if I could go back to slides, please.

So here's a list. I mean, we've got a whole bunch of new features, but here's a few of them that I couldn't quite work into the demo. Occasionally, you know the exact aspect ratio or pixel dimensions you're dealing with for your application, and you don't want to have to worry about resizing the viewer appropriately by hand. For Leopard, we've added a pop-up menu in the viewer, which allows you to quickly choose which either aspect ratio or pixel dimensions you care about, as well as customizing them in the preferences for the application.

For those of you familiar with Quartz Composer, you'll know what clips are. Clips, for those of you who aren't familiar, in Tiger were sort of persistent copies of small functional units of the composition. They weren't really first-class citizens. In Leopard, we've moved them to be first-class citizens. They now show up in the patch creator when you search for a patch. You can create them on the fly just by this Create Clip button in the toolbar. And you can manage them within the Preferences window.

With all these changes, there's going to be some compatibility issues, and we tried to make it as easy on you guys as possible. So what we have for compatibility with 10.4 is we've added the ability to have badges displayed on the patches for whichever patches have either warnings or errors.

This makes it easy to know right away if something isn't behaving correctly. I don't have a slide for it, but we also have the ability on Leopard to run the composition using the Tiger runtime, which allows you to see what it would look like in Tiger right next to the Leopard version. That's under the File menu.

And Quartz Composer, in some environments, runs in safe mode. Safe mode basically restricts some patches from running for security reasons, such as WebKit, Dashboard, et cetera. We've added another badge as well to indicate whether or not a patch is running. It will be available in safe mode. And to talk about all of the new patches, here's Troy.

Thank you, Nate, for showing us a look at the new Quartz Composer Editor. And now, my name is Troy Koelling, and I'm going to show you a little bit deeper look at the Quartz Composer world. We're going to talk about some of the new patches that you might be interested in when mastering the Quartz Composer Editor. So let's go ahead and start with a demo.

Now, as a developer, say I'm a game developer, I spend a long time perfecting my game, whether it's in Quartz 2D or if it's in OpenGL. And now the last thing I need to do is create a high score screen to display the user's high scores. Now, I don't really want to take a whole lot of time to develop it, but I do want something that looks great. So Quartz Composer is my first choice.

Here's kind of what I came up with to display these high scores. Because my game is a pinball game, I came up with the theme of a kind of dive bar to display the high scores, keeping the user in the environment they're embedded in while they're playing the game.

You can see the most prominent aspect of this high score screen is, of course, the scores on the menu. But I also have an image in the background and a movie playing to show to you how Quartz Composer can be interactive as well. This video could be your game trailer or it could be a replay of the current game.

I think if Ducati was actually advertising, they probably wouldn't choose to advertise in this bar. But you get the idea. Now, the most important part of this composition is, of course, the scores. So I'm going to dive in and show you a really popular design pattern, how to get data from your application into a composition.

The goal state we're going to look for is this, just a list of names. Of course, you could have the scores as well. This list comes from your application as a structure. It could be on a published port or it could be as a file on disk with the new XML Downloader patch.

I'm going to close this down and start directly from scratch. Now, because I don't have an actual game to show you, I'm going to use some fake data in order to get the structure that I want into the composition. I'm also going to go ahead and right away clear the rendering view.

Now, whenever you have a list, the first thing you should think is iterator, because an iterator simply takes its contents and renders them to screen. It has a number of iterations, so every iteration, it renders whatever it contains to screen. The number of times we want to compute the contents of the iterator is defined by the number of items in our data set. So we can get a structure count out and set the number of iterations. Now I'd like to send the structure inside the iterator, but there's no port for that. So just like Nathan did, I'm going to go inside and publish a port for that.

Now you can go back to the top level and get that structure inside. Now you can see what the structure looks like. It's got a list of key value pairs. Now, fundamentally, each one of those key value pairs is a string. So naively, we could just try to display that as you would any string. And of course, you use the image with string patch for that.

I'm going to set the blending just as if you saw Alessandra's example. You want to make sure that the alpha component of the string is respected on the billboard. I'm also going to set the width so that it's always rendered as sharp as possible. Now if we take our structure and put it on the string, you'll see that Quartz Composer knows there's a structure in there, it just doesn't know how to display the contents.

What we need to do is split out each of those items in the structure and display them individually, one per iteration as we go. So it makes more space for us. And the two items that we can use to split out that structure is the structure index member and the structure key member. Let's go ahead and get those out.

Now as we flow the structure through there, we can split off each item based on index and key. I'm going to set the key to name. And you can see that I can set the index here and get different items out of the structure. Now really what I want to do is set that index depending on which iteration we're on. Well, the iterator patch has a convenience patch called the iterator variables, which allows you to know which iteration you're currently rendering.

Thus I can set the current index on the structure index member and display all the names. But you'll notice they're all displayed on top of each other. This is a common mistake just because we don't have a differentiation in the position of the billboards. You'll also notice that there's a current position on the iterator variables, which I can connect from here to the Y position in order to spread them out.

That's great and all, but they look a little off-centered. The current position is a number between 0 and 1 because it's simply taking the current index and divide it by the number of iterations. What I want to do is get between negative 0.5 and 0.5 on the rendering destination. So let's go ahead and just subtract 0.5.

Now you can see that the names are properly displayed on the composition and we can go ahead and modify those as we want in order to put them in our pinball demonstration. So let's go ahead and see how that fits in with the broader demonstration. So we have names and scores on the menu. Let's take a look at how this all fits in. First thing we do-- is we're clearing the workspace.

We're just doing that so that we ensure that wherever the image doesn't cover, we're not going to get garbage in the application that we're running us in. After that, I import an image of a bar. This is a bar down by Apple Campus that I like to go to. And we apply a bloom effect so that it kind of gives a kind of dreamlike state.

And that's just set on this sprite here. Next thing we do is we draw the television. The movie playing on it is just loaded from a file on disk, and I've made it a macro so that I could display both the image of the movie and sort of a frame around the image.

I'd like to stress the importance of using kind of real assets because it really gives you a good composition rather than trying to construct everything with core image filters from scratch. Lastly, we have the menu. This is a 3D transformation in order to give us control over where it is positioned in space. You'll notice here the fake data that I had earlier. And this, of course, would be a published port to your application or an XML downloader, as I mentioned earlier.

Inside the menu, we have a lighting patch. It gives it kind of the red-blue glow on the pages and also gives it some shading. Inside that, we have the menu left and right, which displays the image on some sprites that are rotated in space so that they give a kind of folded effect. I'm using a render an image here to composite the high score title with the list and also the texture of the paper so that it has an easy way to get that all together.

You can see here the image with string is displaying the high of high scores and this is the actual paper that it's drawing on. Here, of course, is the Structure Iterator demo that I just completed. You'll see this in many compositions because it's a very useful way to get data into your application.

Now, this composition that I'm showing you is completely compatible with Tiger. Everything you're seeing can be done with patches that will work on Tiger. But let's see what we can do with new patches in Leopard in order to spruce it up a little bit. First thing, this JavaScript I've created is sorted in order to make my point clear.

But if I change this data, You'll get unsorted values and this may be closer to what is happening in your application. Because say you save the data in the high scores data file chronologically or just randomly, you may want to be able to sort that data. So, new in Leopard, we have a structure sort.

The interesting thing about the structure sort, of course, is the sorting descriptor, which allows you to define what item in the structure you're going to be sorting on. Because structures may have many types of numerical data, or you may want to sort on the alphanumeric value of a string, you have the ability to set the sorting descriptor based on a dotted path into the structure.

Now let's talk about something that most people just try to ignore, and that is how this menu has kind of got jaggy edges. If you're an OpenGL programmer, you know the reason behind this is because OpenGL works on discrete pixels. But for you as developers and your users, don't care about those discrete pixels.

They just want smooth edges. Well, new in Leopard, we've added the ability to make sprites anti-aliased on the edges. Let's go ahead and dive down to where the sprites are rendered. You'll see here, In the settings panel of the inspector, we have the ability to give it some anti-aliased edges.

This only works on the sprite patch because we wanted to make sure that you are using this properly. It can use some extra resources. It's also important to note that the sprites that have the anti-aliasing setting on need to have a blending of over or add because the sprites are actually compositing with the layers below them.

Next thing I'd like to do, this Ducati video looks pretty nice, but what if I wanted to apply an effect on it? I may or may not have time to actually think about a core image kernel to write and apply an effect to the movie, but wouldn't it be great if I could farm that work off to somebody else? If I could, say, create a composition which applied that effect and then just load that in on the fly in my composition. Well, new in Leopard, we have the new Composition Loader.

This is a very powerful concept because it allows you to modularize your compositions and load them on the fly. But so as an extension to that, we also have the Composition Repository, which provides a set of filters which you can use in your application. It includes-- we haven't really talked about some of the other things that Composition Repository provides for you, but image filters are one of the most useful.

We also have animations for those of you who have-- want to display backgrounds, or we have music visualizers, as Alessandro talked about this morning. So in order to configure this composition loader to work as an image filter, we can go to the settings and auto-configure from a protocol.

In this case, we're going to use Image Filter as a protocol we're going to use. Now, you could use the input ports here and create those ports on your own manually. But the auto-configure from protocol is a convenience feature to allow you to get the required input ports to the composition loader without any work. Now when we route the image through here and set the location, let's talk about the location first.

Location is, of course, going to be a URL on disk. You can load any kind of composition and speak to the inputs of that composition using these inputs here as long as they're named the same. But as a special case for the composition repository, you can use the identifier of a composition with a forward slash at the beginning in order to load a composition directly from the repository. So I'm going to use a film stock.

And there instantly you can see the video playing is black and white. It's got some changed aspect ratio and a little bit of grain to give it kind of the right effect that we want. But let's say we wanted to mimic a bar closer to San Francisco. I could change this. Right now we're on the fly to something you might be familiar with in Photo Booth, and that's the Pop Art effect.

All right, let's see what else we can do to this composition to make it just a little bit better. Actually, let's take a look at some compositions, some patches that we, I didn't have a chance to fit into this demonstration, but we're really excited about them and we think that you will enjoy them as well. I'm going to save this as a new file on the desktop. Now I can demonstrate the Compare Compositions feature, which is a window allowing you to see the differences between two compositions.

If I load up the original composition on the right side here, You can see the differences between these two compositions are demonstrated in red here. And this allows me to remember what I had forgotten, and that is I actually want to add the ability to move the menu out of the way. So let's go ahead and open this back up.

Now, the menu is a great item here in the way, but... I'd really like to be able to move it out of the way, just kind of on the fly in my application. Maybe the user wants to see what's going on in the background. Well, since this menu is situated in a 3D environment, I can move that simply by showing the patch parameters and scaling around this number here.

But I really know what values I want. I just want to go between two presets. So the best way to do that is to use a multiplexer to set those presets. I have two presets, so I can go ahead and... Change the number of inputs to two and make them number inputs. And then what I can do is set those presets here on the input ports.

go between two positions of the menu just by sliding this slider. Of course, I could also set this zero and one here. But I don't really want to remember zero and one, and the slider doesn't give me any clue about what position the menu is in. So, new in Leopard, I have the ability to add labels to that menu position published input. I can simply insert an input splitter, which automatically creates an index splitter, and change the type of that to use labeled indices.

If I label this menu up and menu down, I You'll see that the input parameters allow me to switch directly between the up and down position. You'll also, new in Leopard, you'll also find that this pop-down menu and any published ports are provided to you as a view that you can put in your application called the QC parameter view, QC patch parameter view. This allows you to give your composition a nice interface without having to actually query the ports and see what type they are and all that jazz.

Now what if I wanted to change the behavior of the menu so that it slides in a more graceful fashion? It would be great if I could just smooth it out. So with NU in Leopard, we have a smooth patch. Which goes between one value and another value in a smooth way, with a settable speed here on the input ports. Now when I change this, the menu glides into place.

So now I can really go and show you some of those new patches that didn't fit in that presentation. One idea you have in Quartz Composer is the idea that you need to save a state in your composition. So I have this Pin the Tail on the Donkey composition, which allows me to click and hold the position of the tail.

If you saw Alessandro's session this morning, you'll know that's done by the sample on hold. What the sample on hold does is has three inputs. One is what to remember, the second one is when to remember it, and the third one is when to forget. In this case, we want to remember the X position of the mouse and the Y position on the second one.

We also want to remember it when the left mouse button is pushed in both of those sample and hold patches. And when to forget, well, we don't really need to forget any time soon. So the output value is just set on the X and Y position of the sprite.

The values compatible with the sample and whole patch are anything that Quartz Composer handles. That could be images, it could be numbers, it could be structures, or it could be booleans or strings. The sample and hold idea can be extended to another new patch called the queue, which not only remembers one item, it can remember a whole series of values to store.

The queue patch is just like the sample on hold. It has what to remember is the value, when to remember it is the filling value, and the reset signal is still when to forget. We also have an additional parameter, and it's the queue size. In this case, it's four, because I want to remember the last four clicks, four tries for that pin the tail on the donkey.

The output of this is a queue, and that's, of course, just a structure, a list. So what do we do when we have a structure? We use the iterator. And it's just like you would imagine. You have the X positions as a structure. You split the items of that structure using the structure index member and place them on the billboard where this tail is being drawn.

You'll also notice that I'm using a pulse patch. Because the cue can remember more than one value, you want to make sure that it only remembers the instant the mouse button is pushed. So because the mouse button can have a kind of square input where it instantly changes from zero to one, but then stays there for a long time, the pulse patch can detect when that value goes from false to true and send out a single pulse frame to that filling parameter of the cue patch.

The next composition I have displays something that will be familiar to you if you've used more traditional rendering packages, and that is the timelines. If you want to be able to define arbitrary motion in a composition, then it's useful to be able to use timelines to do that. So let's see how that works.

Here is the definition of those timelines. You have keyframes, you have control points, you can move these around and come up with all sorts of crazy motion. You can add as many timelines as you want and name them. and you have all sorts of power with that. Now, if you remember Tiger, any time you click off the Timelines patch, that goes away. But new in Leopard, we have the ability to go to show patch settings, and that'll give you a solid window that doesn't go away when you choose different patches.

Next, I'd like to show you this noise patch. In Quartz Composer, it's always been possible to get high-quality random values. But because Quartz Composer is a 3D environment, it's important to get those random values in multiple dimensions. Since the random patch is essentially just a one-dimensional stream of values, now you can insert a position in space and get 1D, 2D or 3D noise. As you can see, this rotating cube is simply the 3D noise displayed on screen.

Next I have two patches that work in tandem that we think are really neat. These are two separate compositions, but they're synchronized using the network broadcaster and network receiver patches. Of course, this is just running on one machine, but you can imagine this happening in a lab or something. It communicates over UDP on the local area network.

We want to stress the fact that this is not for mission-critical synchronization across the world. It's simply a string transmitted over the network. The string can't be more than about 200 characters. As you can see, the data generators get the data here and output it on the broadcaster sprite. It also packs it into a sort of datagram and transmits it over the network using this network broadcaster patch.

On the other side, we receive the data, which you can see here with the colon-separated numerical values. It's a string. We separate it out and then put it on the sprite. If we close the Broadcaster Renderer, you'll notice that the glow ball stops on the receiver end. So that's how you know that the data is in fact transmitting.

I'm going to go back to slides and just talk about some patches you might want to explore on your own time. There's some advanced patches for those of you who are image processing mavens. Image Histogram and Image Pixel are very powerful patches and I hope that those of you who know kind of the difference between them will take a minute to explore what they do.

We also have open sound control client and server patches for communicating with outside devices, as well as patches geared for communicating with hardware, such as the Apple Remote and HID-compliant hardware devices. We have some numeric patches, some more string patches such as grep patch, and of course the OpenGL Shading Language is the newest programming patch for you to explore just what you can do with GLSL. And to talk about some of the research applications that Quartz Composer is capable of, I'd like to invite Kevin up on stage to talk about.

All right. Hello, everybody. So I'm going to talk to you about the redesigned Core Image Filter patch that we have now in Leopard. So this guy used to be called Core Image Kernel Patch in Tiger, and we did not only change one word. For those of you who are not familiar with this, a core image kernel is a little bit of software that runs on the GPU and that does image processing in parallel using the power of the GPU.

It's pretty fast and it's pretty cool. What you used to do in Tiger is that you type your little kernel and you add the input ports of your patch that reflected the input arguments of your kernel. So here you have the simple case. It takes an image, returns the image. It always returns an image only, but it can take multiple images, floats and so on.

So, So why did we move from kernel to filter? Well, it's because in the Core Image world, a Core Image filter can be a lot more than just a kernel. It can be two kernels, three kernels, five kernels, or lots of kernels. And domain of definition, lots of other options that you can set.

So what we did is that we added all, we brought all what Core Image can do at the QC level. And what it means is that QC is now the pretty much perfect environment for you to prototype your Core Image filters. And the reason why is because you have immediate feedback once you're getting QC.

So you type something and you see immediately the result on screen, you change the parameter, you change the kernel, you change something, and you see what's happening. So you don't have to set up your Xcode project to build, to compile, and to do your test images and that kind of thing.

You can do all these things to see the effect of what you do. You can see it immediately. And the result is that you can be more efficient and so you can go far away in a faster manner. And I'm going to show you in just a little bit some examples of the possibilities and the location you can go to with this redesigned Core Image filter patch. I'm going to show you some image processing and some for scientific research. Okay.

So, one example I'm going to show you is going to be a WaveLay transform. It's pretty cool math for those of you who like math. And it's at the core of the JPEG 2000 high-quality image compression. And so that's what it looks like, it's level 4. And that's pretty complex, lots of things.

And that's a little -- that's too bad because you see here you have four times the same functional block. So you would like to be able to just put that in a function and on the loop and something like iterate this. But this one works just by chaining one kernel per patch together.

So it's possible with one kernel per patch. But, you know, so we would like to improve this. And if you look a little closer, you see you have lots of times the same patch. So we would like to have a better way to deal with all these kernel duplications.

So now in the core image filter patch, you can define more than one kernel. So that's one thing. So then you need to say, well, you need to say to your filter which kernel is going to go first, and then which kernel is going to go after, and so these things. So to do this, we added a second editable text view that is JavaScript environment in which you apply your kernel. You call the apply function on each kernel variable that becomes available in this JavaScript environment.

So I just want to say something now real quick, is that this stuff is for advanced users. So if you create a new core image filter patch in Leopard, it's going to behave exactly just like it used to in Tiger. But for those of you who might want to do advanced users, advanced image processing, and so on, well, you have the option of adding the extra power of core image.

So what you see now is that this kernel, and you have all this apply function is encompassed in between inside a main function of JavaScript. And so in this advanced mode, it's now the input arguments of this main function that becomes the input port of your core image filter.

And so here, for instance, I just added an index. And I can use this index to preprocess and preprocess data before fitting into my kernels. Or I can also change the kernel suite in itself. For instance, here I applied a number of times iteratively regarding the count input parameter. input parameter.

So that's it. So that's one thing that we added. So you can define multiple kernels. Another thing is the possibility to define domain of definition, or DoD. So domain of definition is pretty important for this kind of filter that transform, that explode, you know, for which, so domain of definition, I'm sorry, is this region in which the result image of your kernel is going to be defined.

And for some filters, kernels just like this, a zoom blur, the region where your filter is defined is bigger than the domain of definition of the input image. So you need to say to your kernel that this is going to be the case, and to do this, you use the first argument of the apply function.

So it's a rectangle, and it can be unions of rectangle, intersection, rotation, and so on, lots of funky things. And we wrap the CI filter shape for those of you who are familiar with this to do all these operational rectangles. So another thing is region of interest, which is, if I can help you, the domain of definition in the other direction.

So a region of interest is which, given a destination rectangle, what is the source rectangle that I need to process this destination rectangle? And in the case of sub-sampling, for instance, the default, which is identity, doesn't work. You need to say that. So you say that the source rectangle is actually bigger, and for this, you set the ARI handler property on the kernel variable to a function, which is JavaScript, so all of this is in the JavaScript environment, and this function is going to take the destination rectangle and return the region of interest, so the source rectangle.

One last thing is that you can access from within the JavaScript all the built-in core image filters that are shipped in the system. So you don't have to redo the color transformation, geometry, compositing and all the stuff that CIS already does where you blur and so on. So it's very easy to do.

You just write the name of the kernel of the filter, core image filter, built-in core image filter, and you apply it. You give it a name and it works fine. So now let me show you an example of the Core Image filter patch in action for advanced users.

So now -- let me go there. Here we go. So that's this fabulous advertisement that we absolutely all adore, which is the Ducati motorbike something ad. So you even have sound. So it doesn't look like a lot is happening. You have the advertisement full quality, so you're very happy. It's in HD, you see.

So here you have, if you look at the composition, you have a movie patch that feeds images and that goes through a core image filter. So actually something seems to be happening. So what this core image filter does is does as much as what I said before, it's doing a full-fledged wave-led decomposition and recomposition.

And so this stuff is used in JPEG 2000. For those of you who are image processing people, you know that wave-led waveform is less at compression level zero. So here that's what we have. We have zero. But we can in real time, I'm going to start, we can in real time adjust the compression parameter and see the effect of compression in the input image.

And in real time, a lot of stuff is happening over there. Lots of math. All the kernel that you saw before, that's what is going on there. And it's in real time happening on your video. You can also stop. So yeah, so what I want to say, so all this stuff is happening on the GPU, the JavaScript from the CPU, but it's just there to build the suite of kernels. So the kernel function, all this kernel junction stuff, kernel one after the other.

So the kernel recipe, the filter recipe, but the evolution of the kernel function, the evolution of this kernel recipe, so the core image filters is actually evaluated on the GPU. So that's where the expensive stuff is going on because you have to process all these pixels of these bits.

So, you can also use the same patch to show you the wavelet transform at the decomposition level. So, you have all the multi edges, multi resolution thing, which is, I mean, you know, it's pretty exciting for people, like this kind of image processing stuff. You can change, you know, you can see the effect of compression on the wavelet coefficient, change the wavelet level.

So, you have full compression, then you have more. You can also save that as an image filter to take a photo booth picture of decomposition of your face to your friends and family. You know, that's a secret feature of WebViewer. Secret secret. Nobody knew about it. Of course.

So here we go. So I hope all the image processing gurus over here are going to have a lot of fun with this. It's very practical. And I'm just going to show you how the code looks like. so we have it here. We have some kernels, so we have maybe a dozen, maybe a little more, convolution, sub-sampling, so yeah, you can check it out. You have the source available on the WC website.

Here, that's a JavaScript, so you have a little bit, but the -- so, but, okay, so this is a bunch of DODRIs, so we use all the parts, so here that's just the code for the recomposition, which is the main thing, and here that the code for the decomposition, which is just the hopper, so it's, you know, it's not insurmountable, and if you had to do that in a CGA, it would be bad. It would be worse.

So now a last example for computer -- computer vision research, so stereo depth computation from a stereo pair of camera. So you have left and right image, and there is an algorithm up there -- so two cameras looking at the same scene, a little distant one to the other.

So there's one algorithm up there that say that, you know, we're doing some translation distance and minimum. You can actually find the depth. So we just did it in a couple of hours' read in Teclon. And that's the result. So, you know, it's not too bad. White means closer, and dark means far away. So, you know, pretty much get the idea. The lamp is closer and so on. But what is cool is that now you have this depth map.

So that's a depth map. But now you're in QC, so you can use it to -- you can use all the other patches to visualize, to interpret better this data. That's what we did. So you can just see -- explore that in space using Sprite and visualize your depth map and projecting on the HITPUS sprite.

Thank Thank you. So you see that's nice. And then you can see even better that you have a bunch of noise. I mean, that's really -- I'm not -- you know, it's not a SIGGRAPH paper of next year. Just to show you what we can do and to point out. So what we can do is actually you can reuse what we just did, which is a JPEG 2000 thing.

So we're just -- it's used for denoising, so I can just insert the guy, have a noise threshold here. So here's the left and right image. I'm sorry, left and right image going through the core image for the patch that returns the depth map over here. So I'm just going to put that through the JPEG 2000 patch with a noise threshold with a threshold with a coefficient of 0.6. Let's go. Let's see. Let's insert that.

And you see we're doing denoising. So you can play with that and you can experiment. And you know it's very easy to reuse the work you're doing. It's very fast. And the code, real quick. So you see this is easier than we have transformed some kernels and the JavaScript. So the JavaScript, you know, it fits in one page, so it's not too hard.

Okay. Let me quit that. Oops, I'm sorry. Let me go back to slides, please. Thank you. Okay, yes, I need to change this line. So here we go. So just briefly, check out the new JavaScript patch. It's improved. We did a lot of work over there. Now you can have typed input and output ports. And globals are fully supported. So you can conserve state along the execution of your composition.

It's very powerful. OK, check it out. Thank you. All right, I'm going to talk to you guys really quickly about composition optimization. Now that we've made all these compositions, we want to make them fast on all the hardware that Apple ships. And there are a few simple things that you guys can do. First one, only execute the necessary patches. It sounds obvious, but a lot of people continue to execute patches that are no longer on screen. Use the enable input port on consumers to disable all of the patches that they're pulling data from.

Since most of what you deal with in Quartz Composer is images, make sure that you optimize your images. There is no point in applying very large, expensive effects on a large image if you're only going to resize it down later. So make sure that your image dimensions are what they need to be, and to make that more convenient, we added an image resize patch in Leopard. Also factorize common images. If you need an image in multiple different macro patches within Quartz Composer, publish things up to the top level and have one copy of that image rather than having multiple copies in all those macro patches. They're just taking up extra space.

Like I said during my demo, the render in image patch is incredibly powerful. You're going to run into it if you want to do anything complicated. But it can also be very expensive. It can use up a lot of VRAM. So make sure you configure the pixels wide and pixels high. Also make sure that you configure the settings appropriately. There are some internal settings on it that you can use to tweak the amount of VRAM it uses.

Core Image is an incredibly powerful technology. Just make sure you use it carefully. It can do some amazing things that really can bog down even the best systems out there. So make sure that you sort of chain them together in your composition so you're not bouncing from the GPU to the CPU and back and forth.

And this one's one that most people don't consider. Adapt your compositions dynamically to the hardware capabilities. We provide a couple patches such as Host Info and OpenGL Info that allow you to tell what the machine you're on is capable of. So for instance, if Core Image isn't supported, go down a different path rather than, you know, if Core Image isn't supported in hardware, I mean, go down a different path as opposed to continuing to fall back to software.

So that's a couple of things that we're going to talk about. And those are really the five things that help with optimizing compositions. All of this information, plus everything that we didn't have time to cover, is in the release notes. It's under the Help menu. There's also a Getting Started guide there if you are new to Quartz Composer.

For more information, contact Alan Schaefer. He's our graphic evangelist. We have a very active mailing list with almost 1,000 members on it. All the members of the Quartz Composer team are on it pretty much every day as well, answering your questions. We have a lab tomorrow. We're going to be here pretty much all afternoon, so feel free to come in and ask us any questions you want. We also have a session tomorrow on writing custom patches and integrating within your application. That's session number 421. It's at 1030 tomorrow morning.

And in summary-- We've changed the editor and removed a bunch of the limitations that were in 10.4. We've made it easier to create compelling content. With the Composition Repository, we've made it easier for you to get that compelling content into applications. So what we want you to do is we want you to take some time and explore.

You know, poke around. Things that you didn't think were possible with Quartz Composer, like a lot of the stuff Kevin was showing, try them. See if you're able to do it now. And most of all, have fun, because that's why we really want to be in software anyway.