Hello everyone, my name is Troy Koelling. I'm an Engineer in Quartz Composer. And welcome to Quartz Composer Advances in Snow Leopard. Today we're going to talk about Quartz Composer advances in Snow Leopard. But first, I'd like to talk about Quartz Composer advances as they've come in the other releases of OS X. So Quartz Composer was introduced with OS X Tiger 10.4 and what was-- what Quartz Composer is, is a framework which allows you to do graphic visualizations which is embedded into the operating system.

So you don't have ship any plug-ins or any special applications for your users to use with your applications or if you want to just ship the compositions, the users will be able to open those in QuickTime. Quartz Composer is also an editor which also ships with the developer tools and that ship with Tiger and it allows you to create these files which we call compositions. And compositions are a set of patches or notes that are connected together to define how your graphic visualization is going to work. In Leopard we really matured the technology and we introduced a whole slew of new user interface elements such as the QC pick review.

We have the parameter view. We also introduced a way to share compositions among the entire operating system with the QC composition repository. And what this allows you to do is to create a composition which will work not only for Photo Booth, iChat and your own application. And it allows people to really have an open community and really utilizes compositions in a powerful way. Of course you also have been making a lot of really great plug-ins with the QC plug-in API which were introduced in Leopard as well.

And we're really proud of all the work you guys are doing and it's really great work that we're seeing out of you guys. So today, we're going to talk about the advances that we're making in Snow Leopard. 'Cause as I said, Leopard was really the maturity of Quartz Composer, now we're going to take you a couple steps further in some very specific applications.

And also we're strengthening the baseline with performance improvements and workflow improvements as well. So what are you going to learn today? You're going to learn about the editor, the performance, improvements that we've been making, and then we're going to talk about OpenCL and the geometry pipeline. And it's important to note that the things we're talking about today are really passive instructions that we've been working on.

And I want to challenge each of you to kind of take a call to action to think about what we're talking about, how that can apply to you and how you are going to go out and use this in your application right away. Because I can talk about how we've made the editor better but what that's going to mean to you is, now you can make your compositions faster, you can create them and develop, deploy them faster.

Or I'm going to talk about performance improvements and that's going to really impact your application and allow users to use your application on lower range computers or you can integrate more complexity into your compositions. So let's get started with the editor. We have a brand new user interface in Snow Leopard. We've been working really hard on this. We're really proud of it.

It's supposed to both help you work faster and more-- get you down to business quicker and also supposed to look a little more professional so that you know that you're not working on a toy, this thing that's been taking you a week or a month to develop. It's not just some kind of bubbly little user interface application. So what you're seeing here on the left hand sign, I believe that's your left hand, yes, is the editor.

This is where the nodes are lined up. For those of you that are new users, Quartz Composer is a technology which takes nodes and puts them on a workspace and you connect little wires which we call connections between them to define how the data flows through your composition. So for example, this might be a macro patch which is taking in a list of items and those-- each of those items is a point with an XYZ position.

So for example, this composition, it takes the-- an item at an index member which is that structure index member patch and it splits it out into three different numbers. And you can see those three different patches in my gray box which actually do the splitting out. Each of those will have a different index number for each of those patches. So you get the XYZ position and then you were setting that on a sprite.

So what have we improved in the Quartz Composer Editor? Well the big thing, one of the big things we've been working on is the new patch library. This patch library shares some user interface elements with some other applications on the system you're not use, used to, such as the interface builder and Dashcode. But it retains a lot of the same behaviors that we had in Leopard that were really great. So for example, you can still search and filter these patches by just typing out a couple letters in the search bar.

We actually improve the search as well. We've added alias terms to some of the patches. So for example, you may not understand if you're coming from a DSP background that in Quartz Composer nomenclature, multiplexer is what you might think of as a switch. So if you type switch now, you're gonna get multiplexer. And that should make it easier for you to discover which patches you need to be using in your compositions.

Next thing that's the same as Leopard is inserting. You can still just hit Return when you have a patch selected in the list or you can drag and drop from the list. That's no different than was in Leopard. We've also added custom groups Now this is a-- could be a really powerful paradigm.

We're not sure 'cause we haven't seen anybody actually use it. I haven't seen you guys, what you're going to do with it. But basically, what you can do is pop-up the dialogue like this, create a filter, a search term or there's other properties that you can filter on there. You can name it and it's going to create that folder icon there in the Patch Library.

And then you can quickly get to those patches easily. We've also added examples, it's going to be really powerful but for those of you who are new users, because you can just click a link in the patch description, open up a composition which uses that patch and find out how it's supposed to be used. And the Patch Library is the place to go to manage your Virtual Macros.

And Virtual Macros are very important. I use them all the time and I don't get to do that much playing around with Quartz Composer, only the Fridays that I kind of take off and work on my own stuff. So if I can use them in my daily work, then I know you guys can use them in the cool compositions that you're creating.

So I'm going to talk a bit about that today and go into detail on how that works. Virtual Macros, what is a Virtual Macro? Whenever I say Virtual Macro, you can just think of an external composition. A composition which has the functionality of a certain macro, a certain subsection of your composition, and it's saved externally on the disc and loaded at run time.

So what does that mean? That means you can modulize your compositions, you can reuse aspects of them, and you can share them with your coworkers, or you can deploy them on the web, that kind of thing. The Quartz Composer Editor has one button to create Virtual Macros. There is one button to edit them.

There is one button to share a Virtual Macro and one button to install them. It's one button for-- one composition to bring them all and in the darkness rule them.

So, here's the Quartz Composer Editor. There's the Patch Library as I mentioned. What we're going to do is select the patches. The new UI represents selected patches with an orange band around them and I hope you can see there-- that up there on the screen.

All three of the patches in the gray square are selected. Now, what we're going to do is click the Add to Library button in the toolbar. And this is going to present for you a dialogue where you're going to be able to name your Virtual Macro. And really important, it's going to also allow you to create a description, this is really important because this is how your users are going to know what to do with your Virtual Macro and also for yourself in the future whenever you're going to use this.

When you click Done, those three patches are going to be wrapped up into a Virtual Macro and saved to disc. And then you're going to get this new patch which is inserted into the library, into the workspace for you with all the connections recreated. And as you can see here, the name of the patch is added to your Patch Library and you could begin using it right away.

This is a really powerful thing, for let's say you have a bunch of different components in your composition but you have repeated functionality. This way you can have one place to edit that functionality and have it affect all the different places where it's used. Now it's in the Patch Library and we can continue to manage them with the action menu, that's the little gear on the lower left hand corner, or you can right-click on the patch. So you can edit a Virtual Macro.

What this does is open up the Virtual Macro as a composition, you can make all the changes you want and you can press Save. Then the edited Virtual Macro, the edited composition that is, will be saved to disc, of course, and then repopulate all your open compositions that use it.

That's make it really easy to streamline your workflow because you can make a small tweak and you don't have to close down Quartz Composer as you did in Leopard, reopen it in order to reload these patches.

We also have the ability to export a Virtual Macro. This means that you can take the Virtual Macro which is on disc and save it off to say your desktop and then you can email it to somebody. What's important to note that you don't have to do this to deploy your compositions.

In Snow Leopard, we've implemented a feature which will serialize all your Virtual Macros that you use and add them as metadata in your composition. So it's really important. Let me say it again, the Virtual Macros are going to be serialized in your composition, meaning that you're not going to have these nasty dependencies as if you were using, like, real QC plug-ins because the Virtual Macros are installed with your composition and they can be used in a read-only fashion. So exporting, what use does exporting have? Well that's how you get them to your coworkers so that they can have read-write functionality over those Virtual Macros. And that's how can you really collaborate on these modular components.

When you export a Virtual Macro, it's going to be saved to disc as a composition because these Virtual Macros, as I mentioned, are just compositions. But we do have some metadata on there to know that they are compositions. So that when you double-click on one, it's going to open up this dialogue that says, "Do you want to install this? Because it's a Virtual Micro." Otherwise, you're going to open up a composition and it's going to have nothing rendering.

It's not going to seem like a complete thing. So we also put some logic in there to make sure that if you already have a Virtual Macro, it's going to upgrade it and all that stuff. So we take care of all the heavy lifting. You never really need to go to the finder, into the library, and figure out that silly directory where they're saved.

You just can do everything you need to do from within Quartz Composer. So now let's stop messing around the slides and actually go to a demo and see the new Quartz Composer Editor in action. So let's go ahead and open up the Quartz Composer Editor. First thing you're going to notice is our brand-new welcome sheet.

When you first open up Quartz Composer, it's going to be on this getting started pane which is going to provide an introduction to Quartz Composer and also these four links to the very important documentation and link to the external developer list. Also examples, as they are on the web. And this will be your jumping-off point for learning how to use Quartz Composer for those of you who are new users.

All these are live links and they'll open up in Safari. We also have this new tips panel. These are the tips that the Quartz Composer team have collected and decided these are the really important things we want to share with you, that don't really fit in a release note and they don't really fit in a tech note. These are just short little things that will help you get your work done faster. And finally, we have the templates view. This is similar to how it was in Leopard.

All of these templates are live updating. This is actually using a composition loader to load the composition into this sheet that comes down and show you what you're going to get when you click Choose. We also have examples linked from here so you can have an idea of what that template is going to do.

You know, go ahead and close this for now and open up an example that will help us learn about Virtual Macros a little bit better. So not only do we provide the feature of Virtual Macros, we also provided a bunch a of Virtual Macros that we use every day that seemed to be useful in most compositions.

This is like us introducing a whole bunch of new patches but these Virtual Macros, these patches you can go in and edit if you need to. So what I've done here, is created a composition which is a small movie player. [Noise] And as you can see, I've created a little hud floating controller on this composition and it floats over the movie and allows you to play and pause using our new interaction model in Quartz Composer.

And although these buttons aren't very complicated, you could imagine that they were created in Photoshop and they have they have specific sizes and I always want them to display at that size so they don't get stretched out or anything. So I've done everything right with-- I've used the anchor patch to pin it to the side, I've used the billboard's actual pixel size to make sure that the image never stretches or shrinks.

And these are all things that you can find out about by viewing previous years' demos. Unfortunately, we don't have time to go into those in-depth but I do invite you to check those out if you have time. But there's one thing that a lot of people have trouble with and that is, if I want to make this bigger, I want the hud to stay to the same size and the same aspect ratios. But because it's using Quartz Composer units and I've intermixed Quartz Composer units and units and pixels, these buttons tend to move around.

And that's no good. So what we need to do is also position the user interface elements, using the same methods that we use to size them and place them. And so for that you need to convert from Quartz Composer units to pixels, actually the other way round 'cause we want to set pixel values and translate that to Quartz Composer units. So Quartz Composer now provides that for you. We have the units-to-pixels patch and the pixels-to-units.

So for example, I can just drop this pixels-to-units patch in here and I can type in a number of pixels, say 100, and it's going to output the Quartz Composer conversion of that. And this is a really easy conversion to do. It's totally trivial, it's just two patches. Actually, let's go ahead and show that. It's just the rendering destination dimensions.

So I'm going to edit this pixel-to-units patch. And you can see it's this rendering destination dimensions and we just get the ratio of the width to the pixels wide. So the pixels wide is the number of pixels on the screen. So in this case, I'm guessing that's 800. And in this case, the width is always going to be 2 because Quartz Composer units are two.

So, there are two and let's see how close I was, 512, I guess I was little off. So these two patches can now be used inside this Virtual Macro. Now setting up all those pixel values for each of these offset values is kind of tedious. So I went ahead and prepared that before the show. But let's go ahead and see what happens when I connect each of these up.

And that's all there is to it. [Noise] You see the interaction is still tied to the sprite in question or the billboard in question in this case. And [noise] my little movie player is now properly resizing just as I wanted it to. I can go all the way to full screen.

I can drag this anywhere I want. And so that's a brief demonstration of how Quartz Composer now uses Virtual Macros. So that concludes our editor session. If you have any other questions, please feel free to visit us in the lab tomorrow. Now I'd like to talk about performance and what we've been doing to make Quartz Composer even faster in Snow Leopard.

So Quartz Composer is a node-based graphics language and what that means is there's a graph. Now I'm not talking about a spreadsheet graph like you'd see in, numbers or something. I'm talking about your CS 101, a tree is a graph kind of graph. And so here's an example of a graph, because we have an image node it connects to the billboard and we also have a separate kind of entity which is this interaction patch which moves around a mesh which has been imported.

This is a 3D model. So because this is a graph, we know certain things about it. We know, for example, how to find a sub-graph and when a sub-graph needs to be executed. We can find that corresponding region onscreen and we can skip rendering whenever possible. And this is really great for performance because the best-- the best performance is doing nothing. In fact, with new idle state optimization, if your composition doesn't have any changes and the animation is going on, we're going to stop rendering completely. So you have a perfectly idle composition.

That's really great for all sorts of practice-- all sorts of-- all sorts of implementations because especially for interaction you do something, your composition reacts, and then it goes back to a known state. So let's talk about the performance-- let's go back to a demo of a performance. I mentioned earlier that when you open up Quartz Composer you get this template sheet.

What I didn't mention is that this template sheet is actually implemented in Quartz Composer, just to prove that we could do something so complicated. It's not something we recommend that everybody do, obviously, 'cause there are much better ways to display a view like, let's say, WebKit or even Interface Builder. But because we want you guys to have the most freedom possible, we had to prove to ourselves that we could create a composition that could be used as a interactive interface.

So all these things are being done through Quartz Composer. So to prove that, I can go ahead and open up the composition which you can find in the resources of the editor if you want to muck around with that. And you can see how this welcome sheet was created.

It's not a simple composition obviously. There are a lot of patches going on, a lot of things. It's not something I expect new users to understand right away. But for those of you who've been around for a while, you probably can get a gist of what's going just by opening it up.

And you can see that it's live-- it's completely live in the editor. That means that when we are prototyping this welcome sheet, we were able to play with it and all of that stuff. And then we skip the phase between prototyping and deployment because the prototype is the deployment. So one of the things you do when you're prototyping is you try to figure out the performance in the performance bottlenecks.

And we wanted to make sure that what was rendering onscreen is as small as possible. So in the debug feature of this viewer, there's a little bug here and that will turn on the performance painting attribute. And that will allow us to paint a red square, not only on these patches and see when they're executing in the workspace but also in this viewer. Now you notice that these patches are executing every frame, that's only because it's checking whether it needs to execute. It's not actually doing anything.

So patches down here like this, they're not executing because they've already discovered that this renderer is-- has declined an update for this frame. Let's go ahead and close that so you guys don't get sick. And I said that we're going to paint red squares over the areas in the viewer that have been updated but there's nothing in red squares here. That's because we have 0 FES. Nothing is updating. And this composition is completely idle.

But if we go to an animating composite-- animating template, you can see the red square painted over these clouds. So that's how we know that all the buttons don't need to change, only this little square needs to change in the viewer. We're also using this implementation of finding what needs to update on a screen for our interaction model. So we know when-- what area the renderers on screen take up. So this allows us to do interaction with 3D objects, interaction with 2D objects, and pretty much anything that renders on screen.

We also worked on a bunch of other performance things. This is Snow Leopard, so it's a better, faster, lighter cat and we have a whole bunch of other performance improvements. Movie playback is faster because we're using the latest QT APIs. QT, meaning QuickTime, obviously. Iteration is much faster because we're minimizing state changes between each iteration of the iterator. We are much faster under garbage collection.

And I know many of you graphics programmers may cringe at the thought of garbage collection in a rendering application. But this is important for your screensavers, for example. Because they switched over to garbage collection in screensavers, so your compositions are going to run, actually, faster than they did on Leopard even though garbage collection has been turned on. We have better support for multiple GPUs. This is the common scenario where you drag a composition from one display to another.

And we're using FBOs all over the place. And FBOs means Frame Buffer Object. And these are vastly superior to Pixel Buffer Objects because Frame Buffer Objects are on the GPU. This is especially important for your render an image compositions because those will run much faster. And JavaScript has been made faster by us in addition to the WebKit performance improvements that have been made in the Snow Leopard timeframe.

So that concludes my section of the Quartz Composer talk. I want to challenge you again to go out there and open up the editor, play around with it, and have a lot of fun. But now I'm going to invite Alessandro up to talk about some of the great new improvements we've made on these two very important technologies.

Alright, thank you Troy. Again, my name is Alessandro Sabatelli, and I'm an Artist here at Apple. And it is both an honor and a privilege to get to talk you about OpenCL on the geometry pipeline. So OpenCL, many of you are probably familiar with OpenCL at this point. But if you're not, it's a technology which is really suited towards processing large amounts of data. So being that we are a graphics technology, this is going to be things like geometry components. So vertices, normals, indices, colors, et cetera.

This also means components of images. So things like pixels. Though where it really starts to shine, particularly inside of Quartz Composer, is for the processing of arbitrary types of data. So previously within Quartz Composer, if you wanted to process generic data, this would be done using JavaScript. Or if you were writing Objective-C plug-ins, they'd be using your plug-ins, or of course you can process them within your applications and then pipe that into Quartz Composer. But if you're processing, say, 2 million items per frame, none of these would have suited.

And that's again where OpenCL really starts to shine. So as a practical example of arbitrary data processing within Quartz Composer using OpenCL, we have the app wall. So I'm sure many of you have already seen it, downstairs on the second floor. So in this case we're using Quartz Composer to render out all of the apps. But we're using OpenCL to do all of our generic data processing. So OpenCL is doing the heavy lifting in this composition by generating an animation time or literally for each of one of these 20,000 elements.

And then of course, we use traditional or typical Quartz Composer methodologies or paradigms for drawing the actual icons. And that really allowed us to split up the work and leverage everybody's talents that worked on the project. So being that this is the last session of the day, we figured we'd show you guys a bunch of demos. And these demos are going to show you some of the other things that you can do with OpenCL inside of Quartz Composer.

All right. So this first example, you can clearly say, says Quartz Composer and OpenCL, oops, [laughter] or not. How 'bout now? OK. So perhaps it's a little bit clearer this time. But there's also a little note that I left for myself at the bottom, says press Space.

And this is an example of a Virtual Macro that we've included in the patch library, it's called "Instructions." And you'll see it time and time again within many of the examples that we've included. And it's just nice and we're starting to homogenize these things and we encourage you to use it within your own examples.

So in this case, if I do press Space, you'll start to see some crazy psychedelic smoke [laughs] up here all over the screen. And this is using OpenCL to do a computational fluid dynamics simulation employing the Navier-Stokes equations. So what's happening here is we have the text, Quartz Composer OpenCL which is overlaid on top of some other text where when I hold down the Space button, the alpha becomes 1 and the text is seeded into a set of kernels that are doing this CFD. The color is random per frame. And we're interpreting the color as the direction in which to push the smoke over time. So if it's red, it's going to right.

If it's greenish blue, it's going to go down. So the best part about this demo, for me in particular, besides all the crazy colors and the fun you can have just holding down the spacebar [laughter], is the fact that you don't even have to care about the fact that these are Navier-Stokes equations or this is computational fluid dynamics.

We've included this as a Virtual Macro, it's called 2D Smoke, within the patch library. So if you want to add smoke to your compositions it's pretty much as easy as searching for it in the patch library and adding it to your composition. Although we of course encourage you to then go in and edit if you are so inclined and play around with the CFD.

Perhaps you can make it even cooler and share it with the rest.

So this next example clearly says "snow."

So, we're doing here is we're using OpenCL to process, again, these arbitrary types of data. In this case, these are particle positions. It's a very simple particle simulation.

What we're doing here is we're generating a bunch of random points at the top of the screen and we're doing that using JavaScript, because JavaScript does this awesome function that you guys have probably heard of, called "Random." [Laughter] And basically, every frame, we have the snow kind of just drifting down little by little.

What's really nice here aside from the fact that we can use JavaScript and all these other technologies, is that I'm also integrating an image into the OpenCL calculation.

So at every frame, I check to see if the pixel underneath the snow happens to have an alpha of 0, in this case. So you'll see it's black and we're drawing the word snow.

And if its does-- and literarily just 0 out that velocity vector so that the next time it comes back through the simulation, it won't move. And as a result, it accumulates on the top of the letters. So I'm sure you can imagine the wind blowing, maybe the snow moving, but again, this is just a really simple example. It's been posted online. You guys can start playing with particle systems inside of OpenCL and Quartz Composer.

All right, so this next example shows us some procedural geometry creation using OpenCL. You'll see on your left, at the top left hand corner of the screen, it says "bonjour." And what we were doing is we're taking that image, feeding that into an OpenCL kernel. For each pixel, we're generating a vertex, and the vertices' color is based on the pixel color of that input image. So for instance, in the image at the top left, it's black, so that's an alpha of 0.

And where it's white, we're also adjusting, say, the Z-depth. So you get this 3D text that kind of sticks out. So generally, this is probably not how you'd want to create 3D text because it's overkill. I don't know-- I don't even know how many vertices we have here, but it's probably 200,000 or something like that. Of course, this is all live.

So we can type, say "hello," you'll see it's pretty fast. The best part about this is, because the source is an image, we can actually go in and use the typical, say, image processing using Core Image to affect that input image and have the result be interpreted by the OpenCL kernel to create our geometry. So here we have a crystallize filter. I can blur that out to kind of melt it.

This is actually quite a bit of fun [laughter]. Here we have a kaleidoscope filter. If we can blur that out, and of course, you know, I mean you can just chain together your Core Image filters, whatever. I mean, really the point here of this demo is the fact that Quartz Composer brings together all these different technologies and allows you to play with them and work with them together. So here, we have-- here's a zoom blur. So it can actually move the zoom blur overtime. This one's pretty cool, this is the up tile. So again, the point here is that we're bringing together all these different technologies.

All right, so how do you actually go about working with OpenCL inside a Quartz Composer? We showed you a little bit of that last year, but this time I'm going to show you through the mesh filter. So create a new composition. And here you have the mesh filter template.

It's a new protocol within Snow Leopard. Again, the protocol is just really being a contract between your composition and whoever is using it. So in this case, the mesh filter says I take a mesh in and I output a mesh. Of course, you can have any other inputs and either outputs, but as long as you have a mesh in and a mesh out, you have a mesh filter. And you can see here, we have this kind of sphere and it's being undulated in a sinusoidal fashion.

So here we have our sphere. Here is our OpenCL kernel, which is dynamically named "sine-wave," it's in quotes. Of course you have the typical Quartz Composer interaction. So I can-- I have a bunch of published parameters. I can change those live. We take a look at the actual kernel.

It's pretty straightforward. We're getting the index, which happens to be the vertex, or the ID of the vertex. So here, we actually get the vertex. And then we're affecting the vertex's Y position based on the sign of the vertex's X position. So as it goes across an X, you'll get the sinusoidal motion in Y. And again, the thing that's really cool about this is that you can go ahead and change the function, you can change whatever you want in the kernel. We'll go ahead and compile that in the background and update it live for you.

So does anybody have a favorite trig function?

Yeah, 10?

So here's the 10 function [laughter], notice the infinity [laughter]. So you get the idea. So it makes it really to just focus on your kernel and not really have to worry about anything else.

So we took a look at the OpenCL kernel. It's really easy. You just drop it into your composition. And again, it allows you to focus on writing the kernel. We do a bunch of things for you automatically. So we deal with things like resource management, o you don't have to worry about memory and stuff like that. We also do a bunch of caching, so you don't have to worry about any of that.

We also have a bunch of heuristics that are built into the OpenCL kernel patch which automatically configure the kernel based on the data coming in and the kernel itself. So typically, you'd have to do these things manually. And we actually allow you to override these automatic settings. There's an advanced settings within the kernel patch.

And also worth noting within the advanced settings, we have some metrics, so you can sample the time that the kernel is actually taking to execute. So you can evaluate performance and things like that. And the times when you'd want to override this and set these manually are when the data coming into your kernel is of different sizes. So if you had, say, a bunch of different arrays, one of them is half the size of the other one, you'd want to configure this manually.

We also deal with outline transformations, color space if you have images coming into your kernel, you don't have to worry about any of that. So if you wanted to integrate OpenCL into your application, there's a couple of different ways you can do that. So of course, you can do it from scratch.

And there's quite a bit of things that you're going to need to do that. I encourage you to go to the OpenCL sessions tomorrow if you're interested in doing that from scratch. But as an alternative, if you built your kernel within Quartz Composer, it's a great first to prototype it.

You can actually also deploy your composition within your application simply by-- you have it wrapping your kernel within a composition. So you have a kernel, it's in a composition. You have a published input, a published output, you use the QC render API to execute that, say, once. Get the data from the output port, and you're pretty much good to go. And of course, you can keep it on whatever device you're working on.

So if you're on a GPU, you can keep it there. All right, so next, I will talk a little bit about the geometry pipeline. And for me, this is something I'm really excited about. This is in many ways why I came to work at Apple. And finally, we have a geometry pipeline within Quartz Composer. And the geometry pipeline allows you to import 3D geometry, create procedural geometry, it also give you accessors to the components.

So again, the components being things like vertices, indices, et cetera. We also do things to say, make working with geometry in Quartz Composer expect-- or behave as you just expect it to. So for instance, if you were to transform your geometry, if you had an OpenCL kernel that was processing that geometry, if you were transforming your geometry on the consumer, that mesh renderer, which is one of these new patches, we would update the vertices going into your kernel such that it reflects that transformation. So these are things that you just kind of expect to happen. And these are really what makes it a pipeline and not just, say, geometry import.

Also if you were to hover over a mesh port, you're going to see a quick little preview of what that mesh looks like. So here's our wind tunnel, and we're using again the 2D smoke that we showed you before. But in this case, we've enhanced it a little bit. We've added some different advection at the end. So you can see it kind of swirling around back behind the car there, and the car is fully 3D.

So we can transform the camera, et cetera.

But for me, what's really nice is that the geometry fully integrates into Quartz Composer. In this case, we can use, oops [laughter], we can use OpenCL to actually transform that geometry and we can see those changes updated live, hopefully.

Let's try that again.

So maybe not as fast on this machine but you'll notice that we're running on a laptop. It's pretty impressive. And so I can do things like deform the car, and you'll see that the smoke adjusts automatically. So maybe not prettier than the original design, but you get the idea.

So this next example we're showing you a bunch of mesh filters that we've included in the composition repository for you guys to use and leverage within your own applications. So in this case, we have a mesh twist so we're taking that 3D imported geometry and twisting it along the X axis.

One thing I failed to mention was that we've also included shadows. So our lighting patch now has a toggle per light so you can have multiple lights and you can enable or disable shadows on a per light basis, you can control the colors of the shadows, the quality, et cetera. And this is really important for 3D geometry because it allows you to see the relief of the geometry and actually get a sense of the fact that it is 3D. So in this case I have shadows.

I also have just a very simple GLSL shader to do per pixel lighting. Here, we have a bend filter. So you can imagine text bending around objects. Or--

-- in this case, here we have a wave, wave is kind of fun. So if you wanted to do 3D ripples, the bulge.

Oh, this one is great. This is a jiggle. So the jiggle is a temporal filter. We showed you a little bit about how the geometry, or actually how OpenCL worked last year. So in this case, we are taking the geometry and we're using literally our queue p atch in queuing that geometry overtime and then selecting from that queue based on the distance from, say, the point on screen that I'm grabbing on. So you get this kind of jiggly text. So you can imagine text kind of flying in or bouncing on the ground.

And then we've also included Perlin noise, which many of you are probably familiar with in some form or another. It's just a basic graphic primitive that allows you then build more complicated and interesting effects. So here, we're just looking at the raw output of the Perlin noise filter.

So another thing that makes this a pipeline is the fact that you can take these OpenCL kernels and, say, chain them together. And this is very similar to our image processing pipeline. So typically, if you're working in Core Image you building up complicated effects by chaining together different Core Image filters or writing your own.

And you can do the same thing using OpenCL within the geometry pipeline.

So here I have my set of mesh filters and we're looking at the mesh twist. So I can, say, plug the output of the mesh bend into the mesh twist. And now we have a mesh bend and twist.

And you can do this all day. You can, say, [laughter ] plug the wave into the bend. Wave, bend, twist, or twist, bend, wave [laughter]. How about a bulge, bulge, wave, twist, bend, bulge. You get the idea and it's still pretty performing. And again, this is the laptop.

All right, this next example I'll say more practical application of applying mesh filters. So here we have a PDF document. I have a 3D spine on the left hand side, and this is, I think it's the image filter tutorial for Core Image, and I've added some-- some shadowing on the ground and also some interaction which allows me to grab the individual pages and turn them.

And so what you're seeing here is 2 mesh filters. This is the mesh bend which is giving us the page curl and that's coupled with the mesh jiggle so that gives us that kind of nice physical reaction.

Let's see maybe one more time.

Oh, well. All right.

OK.

All right.

So we took a look a little bit, a look at OpenCL and the geometry pipeline that you can see that they work really well together and they work hand-in-hand.

We took a little look at geometry import and creation, so you saw some geometry there, and some procedurally generated geometry. We took a look at what's it like to work inside the OpenCL Kernel Patch and that's used for geometry processing, image processing, so for instance, when we're looking at the snow image, determining that the alpha was zero and killing our velocity vectors. We took a look at generic data processing as well for our particle system, rendering geometry, so in this case you can render your geometry.

We also have the ability to display that geometry in wire frame point sprites, also volume rendering. Also worth noting is that we do automatic conversions for you, and this again is something that makes it really a full-fledged pipeline. So what this means is, for instance, if you had an array of float 3s, we would automatically convert that to an array of float 4s if your kernel required an array of float 4s.

And we actually do this using Quartz Composer in the way that I described earlier with regards to the QCRender API. So we have literally a composition that contains a kernel. The input to the kernel is these float arrays and then we output the float 4 arrays. We also do that for, say, color.

So for instance if you were to change the color on the mesh render, you'd expect the mesh to change color. And the way that we do that is again we literally have the composition which contains a kernel and in that kernel we've literally just multiplied the color for each vector-- for each vertex by the color that you've specified. As a result, the mesh changes color. So you'd expect this maybe this wouldn't be fast, but in reality it's quite performing. So of course we have other things that we've added. I've mentioned interaction which we showed you a little bit last year.

And again, you saw it in the page curl demo. The inertia and force physics, which is key to that jiggle mesh filter, I encourage you to go in and check it out. We've added Multi-Touch trackpad supports so you can do things like pinch, to zoom, swipe, rotate, et cetera. We also now have a new 3D Sound Player patch which is pretty awesome, so you can add little sounds to your compositions and really start bringing them to life.

This of course works within the iterator as well so you can make all kinds of crazy things. We have new controls for the Movie Player patch, so Troy talked a little bit about performance enhancements of the Movie Player patch, but in this case, we've made it a little bit more livable so you could do things like loop, et cetera. And then we've made a bunch of patches iterator aware. So, for instance, your smooth patch, your counter patch, anything that has stayed now will work properly within the iterator. So for more information, you can of course contact Allan Schaffer.