So what we're going to do is sort of take a look at sort of a 60,000-foot level, and we're going to look at, you know, what Mac OS X's graphics sort of feel. It's a sort of a very immersive environment where you can have, you know, different windows, different levels, all integrated together with a dock in your picture that you see there. You know, you have video, you have 3D, you have normal 2D content.

Over time, different features have been available, or we've made different features available based on primarily some of the architectural decisions that we made. So it doesn't really matter whether it's a window. It doesn't matter whether it's video or 3D. It just seamlessly sort of integrates and flows together, and Expose was basically an example of that.

The other feature that sort of evolved over time was sort of dashboard, where we have more rich content coming in and overlaid and dimmed in the background and really sort of very rich sort of graphic environment from the, you know, from the top of the screen. From the user's perspective, when they look at their desktop. Another feature that was recently added was Spaces. It's more of a usability, right? So Windows, the same thing, just moving them around, and, you know, we can get all these different features for you.

So let's take a look at what architecturally actually happens. So we have these buffers, right? You have an application that's either a 2D application or a 3D application or a video application. It deposits its data into its backing buffers, and then the core, its compositor, basically blends them all together and puts the frame inside the frame buffer, and the display sort of syncs out at VBL. And that's kind of how all the information flows.

So obviously with the advent of GPU, we definitely have video on the GPU. We have 3D on the GPU. And the rest of the stuff back in maybe Mac OS 10.1, 2 was done totally in software. So then came Quartz Extreme, where Quartz Extreme basically said, take the compositor and put it on the GPU. So everything now in terms of video content coming through, as well as your backing buffers, they get uploaded onto the video card, composited together with a Quartz compositor, and display to the screen. So that's cool.

But as time went on, a lot of these graphics architectures and hardware acceleration started to show up in different frameworks, whether it be Core Animation that you've heard about or whether it's Core Image. So a lot of these different architectures for your 2D drawing, whether it be image processing or just animation, they're all done on the GPU also.

Over time, we talk about multi-core. All of these frameworks effectively use multi-core in one form or the other, whether it's via GCD or whether it's basically creating separate threads to supply video on one thread decompression and sending it out the frames. You know, multi-threaded GL is also another example of taking advantage of multi-core.

So basically over time, we've got this sort of synergistic conclusion where we have these resources, whether they're CPU, whether they're multi-core, whether they're GPU, and we sort of at the framework levels, you know, whether it's video or 3D or 2D or animation or whatever, we get to take advantage of all these resources to bring this sort of, you know, rich user experience to one, your apps, and two, to users.

So while we went off and did all of this stuff, along the way, we got to take advantage of all of these resources. So while we went off and did all of this stuff, along the way, we decided that, hey, you know, all of these different features are really cool, and we would like to somehow expose them to applications, developers, so you can take advantage of these features that we develop with inside of the windowing system and the graphics system, and you can simply put it inside of your app. So a lot of what you're learning about in this year or even in last year was about this process of moving all these rich technologies forward for you to take advantage of directly.

So we're going to talk a little bit from a different perspective. Now you know how stuff gets to the screen. What we're going to talk about is how stuff actually gets rendered within your application and what the stacking and the levels are. So obviously at the bottom, there's the CPU and the GPU.

And then further up from that, there's the Quartz family of technologies. Those are sort of the low-level graphics technologies. Above that is the application kit, Cocoa. It's how you form your applications, how you create your user interfaces, how you fetch files, how you do preferences, how you interact with the system as an application as a whole.

So going back to the bottom, we obviously have OpenGL at the lowest level, and we have OpenCL, which we just introduced first in the leper. Further up in the graphics stack, we have Core Graphics, Core Image, and Core Video. Names speak for themselves. One for graphics, one for images, and one for video. Further up from that is Core Animation. Speaks for itself. Animation Engine. And then Quartz Composer. Slightly different kind of beast. It is really something that takes advantage of basically all the technologies and allows you to do some visually rich programming.

Further up the stack, the App Kit. This is the infrastructure, basically, of an application. And on top of that, there's a bunch of other graphic type kits that we have that allow you to do various different things to handle different types of media. So let's sort of peel away the onion and take a look at the top. What we're really looking for is the juicy part in the middle, but we'll start from the top.

So here we are at the application level, and we have Image Kit. Image Kit is a really great application framework for managing lots of images, managing different types of images, managing browsing, managing editing. All these simple features we use inside of our applications to do image browsing. So it doesn't really matter whether or not you're simply trying to do an edit of a single image, or whether or not you're dealing with hundreds of images or browsing hundreds of files.

Image Kit is definitely the kit you want to use to do sort of image handling, rather than going off and writing all this code and doing caching and doing all this stuff. So you can do a lot of different things with this. You can do color matching and color matching, and just use Image Kit.

PDFKit. PDFKit is also pretty complicated, right? In terms of, I shouldn't say PDFKit, but PDFs are also pretty complicated. There are annotations, there's viewing, browsing, scrolling, all these different things are all handled. You just simply take your PDF view, stick it in your application, and away you go, you have full PDF support.

WebKit, HTML is complicated. You don't want to have to deal with that. Safari uses it, WebMail uses it for their HTML content. You can simply just create these, use WebKit to sort of deal with all these, the web content, as well as it takes advantages of advances that we make, whether it's the more complicated cascading style sheet with all the animations, you just get this all for free by using this kit.

Then there's QtKit. If you went to the State of Union and saw the latter part of it that had to do with media handling, we've been encouraging developers to move to PDFKit because it's primarily geared, it's cross-platform, it's the way to deal with mid-video playback. And 99% of the time, that's what you're actually doing. So instead of going and mucking around with all the low-level frameworks to do video handling, simply use QtKit and you'll get exactly what you wanted.

So that's sort of the hardware abstraction layer. So now we're gonna sort of focus a little bit more on the middle, the juicy part. So now, let's talk a little bit about the fundamental guy, the core graphics guy. How do you do 2D? Basically, what's Core Graphics? Core Graphics is a 2D rendering engine.

It has a rich set of primitives for both shape processing as well as image process, sorry, shape processing in the forms of lines, rectangles, path, text, masks. You can simply set those up, fill some color through it, as well as there's a rich set of sort of colorized primitives, which is images, patterns, gradients, and shadings. It's device independent. It doesn't matter whether or not you're drawing to a window or a printer, we will take your content, and we will faithfully reproduce it on the device that you've targeted, bitmaps, PDF, all the same thing.

In addition to it being device dependent, it's also, when we say device independent, we go into a little bit more detail of talking about resolution, depth, and color. So if you're talking about your content going to a PDF, sorry, a printer, whether it's raster-based or ProScript, whether it's 100 DPI or 16,000 DPI, your content would be faithfully represented on that device. Windows, they have a smaller DPI, ranging from 72 to maybe 150. Bitmaps, any. You just set it up, and you can draw it, and it will just record that data and fill the data appropriately.

We're also depth independent. Doesn't matter whether or not the destination is 8 bits or 16 bits or 32 bit flow per component. As well as we're color independent. You could be drawing onto gray destination, RGB destination, CMYK. We'll do all the correct color management to reproduce on the device that you're talking about or talking to.

Features associated with Quartz is definitely anti-aliasing. A lot of people like it, and transparency support. Transparency support on any device, whether it be a printer or not. Transparency on a printer may not have been supported maybe, I don't know, five, ten years ago, but it's supported with Quartz or Core Graphics.

Also, when it comes to UI drawing and a lot of graphic rich content, images are really the king. A lot of stuff is actually done with images. And there are a lot of image formats out there. And what we provide is the ability to read and write to many different formats. Now, these different formats have different features. It might be that they're deeper in depth in terms of the ability to represent different things. They might have transparency. And we might have thumbnails.

And we might have metadata associated with the image. All of these sort of features are part and parcel of doing proper image handling regardless of the format. So we have support for metadata, thumbnails, and with incremental loading where people can read data incrementally and supply it to their devices.

So for instance, when you're on the web, Safari uses Image.io to do this incremental loading as the data comes in off of the network and we can decompress the data on the fly, streaming down, and load it. So we provide lots of great features with Image.io and handling of many different image formats for you. It's platform independent. It doesn't really matter whether it's Mac OS X or whether it's the phone or Safari and Windows. And of course, it leverages the GPU and CPU capabilities, whether it's multicore or vectorized instructions for SIMD or the GPU itself.

So not only can you do sort of rich user interfaces, whether it be mail or some other video app or just the graphics applications, you can do business graphics, right? All pretty good quality, and they're faithfully represented on different types of devices, print or display. Not only can you do that, you can do more comprehensive sort of richer environments for pre-press types in situations, doing the magazine. It's all device independent. Doesn't matter whether your printer is huge DPI or whether you're just running to display, you get a faithful reproduction on that device.

Applications that use it, Pages, that's a good one, that's sort of pre-priced. Keynote, this presentation was done using Core Graphics to do the representation. And Safari, so you know it's fast, it has to go fast too. Okay, so let's, so that was Core Graphics. Hopefully you have a sort of understanding. At this point, I'm really telling you about what's on the menu. We start to talk about pairing things later.

Core Image. Core Image, basically it's an image processing engine, right? You have your input data, you send it through some filter, and it comes out the other side. It doesn't necessarily have to be image data. It could be a gradient field. It could be just some arbitrary data that you can have. And you can run it through different filters of your design to produce your output, physics simulations, et cetera, et cetera. So what we do is we basically say we can chain the filters together.

You have a kernel operation that is supposed to manipulate something from one thing to the next, and you can then do another operation that takes that thing to something else. So here we have an example of an image that was applied with CPTone, and then it was hue adjusted.

So conceptually, it looks like that. But what we actually do is something slightly different. We actually chain the filters together. So you have your input image, and you then run it through two filters, and you get your output image. In the first example, you'd be actually writing data twice and reading data twice. In this example, you'd be reading data once. And you'd be reading data twice and writing it once.

So it's a much better sort of result. So fundamentally, It's really just a pixel engine, right? Conceptually, you think of the data, which is your source image data, being applied through the filters and getting into your destination. But what's actually happening is that the destination is requesting the pixels from the source.

And it's at a pixel level. So you can do these things more or less in parallel. Doesn't matter whether or not you're making fetches for one pixel or 50 pixels. They all can be done in parallel. And because this sort of pixel chain happens from the destination all the way through the source, running inside of this kernel, we can have sort of a minimal loss of information where the actual depth of the pipeline, in terms of the resolution of the pipeline, is actually the data, the resolution, or the precision, I should say, of your calculations.

So basically Core Image has a very large set of filters. It's about 150, I think, the last time I checked. And they fall into different types of categories. You can have sort of pixel and color, which is sort of the diagram that you see there with the blur. It's a blur and actually hue being applied to it.

As well as you have sort of geometric transformations or distortions. Here's a page curl filter that you can do. And then we have sort of stylized filters where you can do, in this particular case, it's pixelate. There's crystallize and pointalize. And there's a whole bunch of stylized things to give you different types of effects.

But not only can you use the built-in set of filters, just basically as building blocks, you can create your own custom filters, right? And a custom filter consists basically of a little kernel program, which basically says what you're doing to the pixel. And it's an OpenGL shader-based language subset. And you then also specify some sort of parameter marshaling, where you're saying, "Okay, here's my inputs. I'm gonna take my inputs, and I'm going to put them and feed my kernel in that particular way." Another interesting point is the separation between executable and non-executable filters.

Executable filters are CPU-based code that will actually run. It's code that will run. And sometimes it's not the thing that you trust the most. So we have this other concept of non-executable filters. What you do is you simply describe a sort of restricted set of a kernel, as well as a declarative meaning or understanding of what the parameter set is, and how the parameters get passed to the kernel. Those are secure. You can put them in any application. You don't have to worry. The right thing will get done. You don't have to worry about somebody doing RM in your kernel.

Feature-wise, this is all standard stuff. We use multi-core when possible, threading. As we talk about pixel pipeline, it doesn't matter how many pixels we're processing. We can all do it in parallel. We have, you know, SIMD sort of support, you know, runtime-generated code that's all vectorized. And we have full-fledged GPU support. So definitely it leverages both the resources of the CPU and the GPU effectively. Examples of that are the menu bar.

Some people like it, some people don't. But it's a pretty complicated filter, that menu. It's got about eight different stages on it, all sort of manipulating the background, doing dodges, and moving stuff around. It's a perfect example of Core Image just right in the windowing system, used to give you a very rich sort of visual appearance. Dashboard is another example with a ripple effect. This is actually sort of a simulation, but you get my point. You've seen it. Sheets and menus, if you look really closely behind sheets and menus, you see that the background is actually blurred. That's the Core Image filter.

So that's Core Image. So now we're gonna talk a little bit about Core Video, but not so much, but merely so that to give you sort of, as I said, you know, to show you what's on the menu. So let's take a look at Core Video. What is Core Video? It's basically a digital video pipeline model. It assumes that there's going to be some source of frames, which is your video source, and there's going to be some sort of filter processing that has to happen, i.e., for instance, color matching, and then there's going to be a final output stage, which is rendering.

You can provide your own filter, sorry, video source. You can provide your own filtering, and you can deal with the frames and produce it whether or not you're writing out bitmaps to something or sending it across the web or something or network. Not only does it have this sort of model, it also handles display synchronization. So if your output is actually a display, we actually feed it back into the pipeline to make sure that frames are delivered in a timely fashion, all synchronized and whatever.

In addition to that, as you know, with video, it's just tons and tons of buffers flying around the place. So we have efficient buffer management. If something needs to be on the card, the video card, then we try to ensure that the frames, since we know what the destination is, the data is actually put in the right place such that the video card can gain access to it.

So this infrastructure is there for you to use. You just put in your pieces wherever you want. You can use it just to move data around, or you can move it to supply a source or add your own filter processing. but you can be part of this sort of video chain.

So that was Core Video, and if you want to learn a lot more about this stuff, you can go up to the QT sessions and find out more about it. So let's take a look at Core Animation. Core Animation is the animator guy. He combines a lot of different technologies, and he's a little bit above that level that we just talked about with graphics and imaging. And it's more about animation and user interactivity.

So at the basic level, it's really a compositing engine. It's just a bunch of stacked layers that all can just be composited together. It also has 3D transforms, 2.5D transforms. It allows you to do perspective, move stuff in space, in 3D space, move them around. You've seen lots of examples of it in some of the demos in previous WWDCs and recently.

It also has different types of animation types. You can set stuff explicitly or implicitly, and all you do simply say is, "I want this to do this," and you forget about it. It'll actually do all the animations. Not only does it do that, it has a rich set of content types. Here you see a QC, - I'm a Quartz Composer media, as well as a movie that just flew by. And then there's also different types of, ooh.

But the whole point of that last slide was that you can set a distortion filter on your movie as it's playing when it went by. So yes, so basically feature-wise, it's an implicit and explicit animation model. You can either implicitly just set things and it'll happen for you, or you can explicitly control exactly how it happens.

It's got a good layer and constraint sort of management, pins and struts kind of, so things are void. If you move stuff around, things move away. And it's got sort of transaction management that allows you to do both transactions and modifying several things all at the same time.

As it says, there's rich media support, whether it be Quartz Drawing, or whether it's a PDF document, or whether it's OpenGL content, or whether it's QuickTime or Quartz Composer. Just deals with them all and animates them all seamlessly, except for the problem. It's platform-independent in the sense that it runs on the phone, and it runs on Mac OS X, and all those smooth animations that you see on the phone that we all love, all the scrolling, it's all done with Core Animation. You know, really easy to use.

It's all GPU and CPU capable. It takes advantage of the GPU in a sense that, you know, your rendering is done on the GPU. It takes advantage of multi-core because you're actually making changes to the layer tree, and it's actually doing the work on a separate thread. And, you know, in cases when we need to, we have full support for all the SIME-type things, all codes vectorized. Examples of it in the OS, the Dock. The Dock for Leopard was actually a Core Animation application.

Preview. Preview uses Core Animation primarily for the display, so you get this asynchronous display when files are slow or long or whatever or big, as well as does the multi-level sort of handling, where you go from zoom one, zoom level to the next. The fetching of the data at the different resolution for the zoomed-in content is actually done asynchronously by Core Animation, and it shows up seamlessly. Other examples, CoverFlow on the phone, as I said. If you use it by the phone, all the flicking around and all the smooth animations, all Core Animation. Time Machine is our example. Time Machine is also using Core Animation.

So Core Animation is definitely a technology that is extremely useful for any type of sort of immersive visual user feedback and just general graphics. And it's a great compositing engine, something that you should definitely consider using if you're going to be going down the route of doing sort of this visual feedback user interactivity.

Quartz Composer. Quartz Composer is a little bit of a Swiss Army knife. It basically does anything to anything and handles anything. But primarily, it's a very good visual programming environment. It's really simple, easy to use, and the programming model is very intuitive. Basically, it just consists of a bunch of patches, and those patches represent some unit of work. And you supply inputs to the patches, and you accept outputs from the patches. Here we have an example of Around. That's one of the patches that's available.

Other things are more, you can make more complicated patches called compositions by simply connecting different patches together. This particular example is basically taking an image and putting a crystallized effect and putting it to the display, and what it's doing is basically chasing the mouse around and locking an image to a grid. So there we have an example of interacting with a human interface device, doing some graphic manipulation, and presenting it to the screen with some logic in the middle.

So there's a rich set of patches that QC supports, and they fall into different types of categories. There's source, whether it's image or QuickTime, audio, all of those different things. There's controller type and network type patches, which have to do with human interface devices, MIDI, stuff like that, Bonjour, RSS speeds, stuff like that.

And processor and filter is sort of math, logic's sort of a mux, if you want to say on this thing, start doing this instead of that. There's all those different things. And it's all in a sort of a visual programming sort of thing. You just simply build these blocks together, and you get the output, set up the conditions, et cetera, et cetera.

As well as there's sort of more extensive programability in a sense that you can actually write code. If you can't satisfy your particular logic request, you can actually write code. Or even if you wanted to make a filter, a different type of filter. You have GLSL or OpenCL. You can make all of those things, patches that you can integrate within your compositions.

Now, not only can you sort of use the built-in set of patches, you can make your own, right? You can have a composition effectively become a patch. So you may, one developer may have done this patch that says, you know, I'm going to do some physics simulation or whatever and say, I want to export that as a macro patch. And that macro patch can be used in anybody else's application as long as they sort of provide the inputs and get the outputs. It's just this way of having this extensibility.

Not only can you do that in terms of making a patch as a, consisting of a building blocks of other patches, which is just looked at from the outside as just a black box where you put in inputs and get inputs, you can actually write custom patches, like executable code, right? Where you supply your inputs, you supply your outputs, and you do your little logic inside of your little module, and that module becomes a patch that people can use within Quartz Composer and within their different applications. Of course, the mantra today, we use the GPU, we use SIMD, we use multi-core, we use OpenCL in this example. You can create patches that are OpenCL patches and OpenGL.

Examples of it, iTunes Visualizer. Clearly something that somebody wouldn't really want to write by themselves. It takes the input, this input is audio, and it does something and creates graphics for the output. Perfect example of Quartz Composer. Sorry, Quartz Composition. PhotoBoot. PhotoBoot, examples of different features, and where picture is actually MC, MC Graphics.

The screen saver, RSS screen saver. Now the point here is not that it's a screen saver. The point is that it's getting RSS feeds and actually representing that on the screen as a screen saver. So here you have, you're interacting not only with some device that is a user sort of keyboard thing, but it's actually fetching stuff from the network in order to add to your composition and make it richer.

So hopefully now you sort of have a little taste of sort of where things are and how they are sort of constructed and where they're used. And what we're going to sort of take a look at is how they sort of interact and interoperate with each other. This is where we actually start after I've told you what's on the menu. Now we're beginning to pair wine with what you're going to order. So now we're going to talk a little bit more.

You look at that graph and you say, oh, that's pretty complicated. In fact, it kind of is. It's just basically a cycle. Everything depends on everything else. But what we're going to do is sort of decompose it into some smaller units and take nibbles at a time. So let's focus a little bit on core graphics.

So Core Graphics basically has a bunch of fundamental types, and they are the fundamental medium of exchange between all of the other frameworks. One type is the context. Another one is an image, which is we spoke about images, and layers. So what's a context? Well, it represents a thing by which you can draw.

It's your drawing context. It's like the pen and paper analogy. It's your pen. You have your pen with your ink, and you can tell it to do things. It's got state. It's got features inside of it to set up masks, tell the pen to move from one place to the other, put down a spot. You can dash lines. All these different things you can do.

So great, we've got our pen. So how do you make one? It's pretty simple. If you have a pen and you want to point it at some raster data, you say, okay, great, I got a bitmap. It's got wide times high times color space. You set it up.

You know, the parameters are not important here. The point is that these functions exist, and they're really easy to use. If you are sort of inside of a view method, you're responding to a view callback to draw, you can obtain the current graphics context by simply asking the current graphics context for its graphics board. You've got your CG context, and you can start drawing into it. At Carbon, slightly different model, you would have to ask the event for its CG context. All very simple, easy to use to get at these contexts.

Images. Well, we all know what images are. They're just a big bucket of pixel data. They've got color and they've got alpha. What if you wanted to make one? Well, you make one by simply specifying its width, height, color space, and the data pointer, and say, "Here you go. I want you to make me an image, and I've described it fully." If you had a context, for instance, the bitmap context that you had, you can simply ask the bitmap context.

After you've done your drawing with your CG context, you can ask your context, "Hey, I'd like an image of what I just drew." With Image.io, we spoke about understanding different types of formats, file formats. So previously, we were talking about bits. Now we're actually talking about, well, you didn't manufacture the bits. You have to fetch the bits from the disk. Fundamental types are basically an image source and an image destination.

They merely represent references or proxies for the repository. You can create an image source or destination with just a block of data, or you can create one with a URL. That's fine. You can get the reference to it, but you can't really do anything. You actually have to start fetching stuff out of it.

So here you have an example of how to create an image or a thumbnail for a particular image at some index inside of your image source. Simply ask the image source, I'd like to get an image or a thumbnail, and this is the index frame number six. Give me that image.

Similarly, if you have an image, whether it was one that you got from something else, let's say you got it from a TIFF file and you wanted to put it into a PNG file, you simply say, create my image destination and add this image that I just got from my TIFF file, and I write it to my image destination. I add that image to the destination, and then the image will get written out inside of the ping file. Pretty easy to use.

So let's talk a little bit about layers. So that was images, now we're talking about layers. Layers are sort of a hybrid in a way that they really kind of are images in a sense that they sometimes can be device dependent, right? Or they can be device independent. You will create a layer by specifying some context. You had a bitmap context, you say layer create with context, and you give it a bitmap context.

And what will actually happen is that we will end up creating a layer of a user-specified size that matches completely in terms of resolution and depth and format. So you may have gotten a CG context, but it was drawing to 128-bit pixel. And when you ask for its layer, you'll actually get back a layer that's compatible with that destination.

So once you've constructed your layer, you then need to say, I've got a layer, which is basically a sheet of acetate. It's clean. It has got nothing in it. And what you want to do is now provide drawing. So you need to get a context from the layer so that you can provide your drawing.

So once you do that, you can draw whatever you want to draw on. Then you have a layer. You now have a reference to a sort of a representation of your drawing. Now you can take that and pass that to the system and say, here, you can use that.

Now, similarly, too, once you've finished drawing your content, you actually want to draw the layer itself. You can simply say, hey, layer, draw this layer to the context at a point or a destination. All very simple to use. So in that way, you are provided with an ability to, whether it be a bitmap context, do some drawing and then have that drawing, ask for the bitmap context for an image and draw it. Or you can use the layer model where you don't have to worry about what the depth should be or how compatible it is with the context. You simply create the layer, draw your content into it. Then you can take that layer and draw it. it anywhere.

So let's talk a little bit about Quartz GL. Quartz GL is basically hardware acceleration for all your Quartz operations. It offloads all of the burden of doing the rendering onto the GPU. And in a lot of cases, it makes for a better story when integrating with other GPU-based technologies. But more on that later.

How do you enable it in your application? Well, you can enable it application-wide. Basically fill in the plist entry list and say, hey, I want to be enabled, and your entire application is going to start being QuartzGL, every single window. That's a great way of doing it, I guess, but sometimes it's not the best. But it does give you a very easy way of turning it on, seeing how it performs right away. The preferred method is to actually sort of analyze and understand exactly what you want.

If it is that you want your main window, which is typically where it should be enabled, you can actually explicitly say, I prefer to have my main window in video memory to take advantage of QuartzGL, but my little tooltip or my color wheel, I don't really think I care about that.

The menus, I don't think, sheets, no, maybe I don't. All I care about is my main contents because that's where all of my intensive drawing is going to be done. So the preferred method is to explicitly say which window you want to actually turn it on, is also quite fine too.

So that was a little bit about core graphics. And you realize that it's sort of the underpinnings of this medium type of exchange, where you have these primitives and you exchange these primitives with other people. So now we're going to talk a little bit about core image. As you see, core image depends on core graphics from this level of it. Fundamental types in Core Image is a CI context, kind of the same model as a CG context. It represents a pen. It's got a CI image, which is basically a proxy for an image data, and it's got filters.

How do you create a CI context? If you have a CG context, let's say the CG context that you got back from your view method, or even the CG context that you were rendering into as a bitmap context, you can simply say, CI context, create with CG context, and away you go, you have your CI context, you can now start drawing CI drawing.

And as I said before, alluded to earlier, if it's QuartzGL enabled, then that context gets complete hardware pass-through. So if your filters and your images were all residing in hardware, and your context, destination context, was QuartzGL enabled, all of that stuff would all happen. All the image processing will happen on the GPU.

Similarly too, if you actually have an OpenGL context, and that's the mode in which you are running in, and you wanted to create a CI context with it, that's the call you would kind of use. CI Images, same representations as same kind of constructors. We just basically, you can create it with data, give it its pitch and its size and the color space, or you can create it with a URL. In that case, what actually happens is we go behind the scenes and go ask Image.io to open the URL.

You can create with the CG image that you may have been constructed. Whether or not that image was constructed from bits you made, or whether or not you fetched them from a bitmap context by that previous call that I showed you earlier, you can get back a CI image that represents a CG image. And similarly too with a layer.

If you have an OpenGL texture, let's say you were an OpenGL application and you wanted to use CI to do some sort of filtering on it, you could simply say, "Here, give me a CI image with this that represents this OpenGL texture." That's great. You've got your image now, and you want to draw it. Pretty simple, too. You simply say, CI context, draw this image in this rect.

If you wanted to actually render bits, whether or not the CI filter was running inside of the CPU or on the CPU or on the GPU, and you wanted to get a raster representation to do something with it, you know, like, whatever, write it out to disk, then you simply make those calls to get the image and draw it into your raster. As well as you can ask the CI context to make a... CG image from your CI image. Pretty simple APIs to use.

So that's great, but that's just for images. What about what we really want to use Core Image for? We want to actually change and modify filters, or change and modify images by applying filters. Asking for filters, you simply say, "I want you to create a filter with a particular name, "whether it's KCI, Crystallize, or Pointalize, "or HueAdjust, or whatever." You specify your filter name, and it'll load the filter. You get back your filter. The next thing you do after that is you need to initialize a filter. So you set it defaults and you can supply inputs for your filters.

Once you've constructed the filter, the filter is just the recipe for the modification. It's a hue adjustment. It has parameters of how it should adjust the hue, but it doesn't actually have any inputs or outputs. You need to supply those inputs. So you simply say, I want you to set my input value to be my CI image, and I want to then take my filter and ask my CI filter for its output image.

And then you have a CI image that really represents the transformed image from the source through the filter to the output image. is an example of creating a filter and simply setting its input image and asking for its output image. You get back your CI image and you can now draw your CI image using the methods we spoke about earlier on the CI context.

An example. Have a CI context, CI image, you want to draw it inside of some rect, and you want to apply a filter. You simply set the filter's input image, and you ask the filter for its output image, and then you say, CI context, draw image in rect, and you're done. You get your hue-adjusted image.

If you want to apply multiple filters, you have an array of filters, and you want to apply them and chain together. You simply step through a loop. You say, first filter, set its input, ask for its output. Second filter, set its input to be the output of the previous filter, ask for its output. Third filter, set its input filter to be the previous output, and so on and so on. You get back your CI image at the end of your chain, and you simply draw your image into the context. Pretty simple.

When should we use it? Definitely if you have transmogrifications of your images. You want to do some hue adjustment and a whole bunch of different things to it. Or if you have just basically some processing data that actually needs to be rerun on your-- you have your own custom filter.

But if you're doing things like just an affine transform, or you're trying to draw an image color matched, you don't necessarily need to use CI. So definitely you should figure out whether or not it is something that you really want to do in terms of really manipulating filters and trying to manipulate your data with complicated filters, or whether it's just something simple.

So you can also use it whenever you want if you're dealing with sort of arrays and streams of data. You have some gradient feeler, you're doing some physics simulation, you can also do it then. And it's important to note that CI is more about sort of a building block approach, in terms of there are all these things that you have, and you can chain them together and build blocks, and you can create whatever you want to create out of them. You can add different types of filters.

So it's really a cookie cutter kind of, oh, I got these blocks, and I'll assemble them together, and I get my result. Whereas in contrast, OpenCL is slightly different. OpenCL allows you to actually do everything explicitly. So you might make a choice for OpenCL, or you might make a choice for Core Image, whichever one is simpler. If the blocks that are available in Core Image, you can definitely use those blocks together, build them up, and create the effect you want on your image. Another interesting point is about sort of explicitly making data.

Example, you can create a CI filter, I'm sorry. You can create a CI filter that represents your, let's say, a professional photo manipulation application. And you want it to sort of say, look, the user is going to modify and do certain hue adjustments and whatever, because he needs the image that I've just taken with my camera, and I want to actually distribute it to something else, but I want to do some modifications. You don't actually have to run the filter and make the data.

And store that as a separate image and then export that image. You can simply just say, "I've got my input image, and I can create my filter chain," which is the modifications that the user did, and you can actually store the modifications, just the filter chain. That's what you store on disk. You leave the original image alone. When the user wants to see it again, you simply apply those filters onto the image, and boom, you get the result. So that's a way of doing it, rather than explicitly running it through and actually physically making the data.

So that's Core Image. And now we know how to sort of make images and make graphics, and what we really want to take a look at now is how do we animate all of this stuff together. This is what Core Animation is for. The fundamental types are basically a layer. They're sort of layers floating in space.

As well as there's a basic animation. It's the thing that allows you to move your layer properties or change your layer properties around. And then there are transactions. Every layer has properties to do with contents, whether it's background, foreground. They have geometry, perspective transforms, rotations, and they have filters and styles. You can add opacity, shadows, or you can set a CI filter.

So let's talk about layers. Create a layer. You can create a layer. I got a layer. If I wanted to now take that layer, the layer has nothing in it, it's just an empty layer, and you want to now set it on your view and say, "I want this view in my application to have this layer," you can set the view's layer to be the layer you just created. And then you tell the view that the layer, that the view wants a layer. So you can set your layer up, and you can check it into your view.

You can also make layers of other layers, or sub-layers of other layers. So in the same way you can inject your root layer into a view, you can inject a layer into a root layer, and make chains of layers, and build them up hierarchically, where a layer has sub-layers.

Useful method for layers, setting a content to the image. Lots of the times, that's exactly what you're gonna wanna do. I got an image, I wanna stick it inside of a layer. I can just simply set its contents, and that would be it. If you want to provide contents by drawing context, what you actually have to do is actually subclass the layer and implement the drawing context.

And what will happen is, as it's animating, and the Quartz Core Animation requires data from the layer, it simply gives you this callback and says, I want you to draw contents. And you simply draw your contents. So whether the layer is scaled down to a thumbnail or whether it's large, you don't have to change anything. You simply provide the data, and it will be captured correctly.

Animations, how do you perform those? A lot of properties on layers are implicitly animated. You simply set the value. You set its new position, and it'll automatically animate. Sometimes you actually might want to do explicit animations where you say, I'd like to create an animation, and I want you specify some property path, which is basically the path to the attribute you want modified.

You set its from value, you set its to value, and you set its duration. And then you take that basic animation and you add it to the layer. And lo and behold, it will happen for you explicitly. Once you're done, you can simply remove the animation by removing the animation that you added by that particular key. And that's how you can perform explicit animations. Now, you can also set filters. We talked about, uh... The image, the movie that stopped.

That was really basically a layer with a filter on it. I had a pinch filter. And you can do the same thing with your layers. You can add any type of filter on it, whether it be a filter that is a content filter, meaning you want to actually distort the content or pixelate the content or do something, as well as you can add something like a blur to the background, meaning that when the layer is drawn, it's going to try and ask the background, and once it fetches the background, it's gonna say, "Okay, well, I want you to do sort of a glass distortion on the background." All you have to do is simply set the filters to the appropriate attribute that you want modified.

Transactions. Basically, every sort of implicit animation is more or less a transaction. It gets generated for you. You don't have to worry about it. But if you actually wanted to perform an explicit transaction, meaning you wanted to modify several objects all at the same time, you can simply say, transaction begin, make your mods, and done. And by the way, they're nested.

Subclasses of sort of layer. There's text layer, simply plain or attributed text, really easy. Tile layer, when we spoke about preview earlier, we talked about multi-level of detail. The tile layer allows you to do that seamlessly, where you simply provide your callback and your data is drawn at the specific resolutions, and Core Animation will just do the rest from there. Scroll layer, similarly there, if you're talking about scrolling content, Core Animation will go and ask for the newly fetched data, evict the old data, and so all the scrolling stuff that you see on the phone, stuff like that, is all done with scroll layers.

OpenGL. That's a typo. OpenGL, if you have OpenGL content and you want to provide OpenGL content to a layer, you use that class. It's CA OpenGL layer, no space. And what you do is you really basically subclass it and you implement the draw method. Draw in GL context. So when the layer, when Core Animation needs your content, it will simply call your subclass and say, provide the content now, and gives you the context to draw into. Pretty simple.

We've talked about Qt Kit. You can provide Qt movies by simply saying, create a Qt layer or a capture layer. Pretty simple. Once you have that, you have your layer, you can basically add it into your tree, insert it as a sub-layer, or add it as a root of your view. Done.

Quartz Composer. Simply, you can basically say, "I want a layer that has a composition in it." create a Quartz composition layer, and you supply the Quartz composition. There you got your layer, and you can insert it into your layer tree. All very simple to use. When should you use it? Definitely if you have layering to do and compositing to do.

Animations to do. There's an alternate. You can actually use the AppKit's implicit animations. They also have a lot of features for animations, and you may not actually need to dive down to Core Animation because the animation support inside of the kit might be, you know, just as good.

If you're doing things that are really simple, potentially you may not want to use Core Animation. If it is that you just want to fade something, and that's the only animation inside your view, you may not need to basically start using Core Animation. It does come with a resource, and you have to be willing to understand what that resource is. And if it is that you're just trying to pulse a button or something like that, it may not be appropriate for you to have Core Animation running just to pulse that button.

So that's Core Animation and its dependencies, and how you interact and how you can use it. So now we're going to talk about the Swiss Army knife of the Quartz technologies. Fundamental types basically has a composition. Remember, we spoke about patches and connecting patches and supplying inputs. That result is a composition. QC view is the view that you inject inside of your user interface. And a QC renderer basically allows you to control the actual rendering of a composition more explicitly.

So you can create a composition off of disk or a file, specify the URL, and/or specify the data. That way you have now this representation of this composition. Compositions by itself only describes, like filters, a description of what needs to be done. So you can query the composition for its attributes. You get back a dictionary and you can query the attributes, who the author is, what the name is, and you can query for its input keys. Oh, it wants the input to be a float. It wants, and its output is a rounded float, et cetera, et cetera.

Now you can understand a little bit about what the composition will do when you supply the inputs. But the composition by itself doesn't really do anything just yet. You actually have to attach it to something in order to do that. So let's say you created your QC view, and you put it inside of your user interface, and now you want it to load the composition that's associated with the view.

Once you load the composition associated with you, the view is now attached, it has an output destination, and we know what's going to be happening. So you ask the view at that point to get its input keys and its output keys. You supply those, and your composition is now set up for rendering. And then you just say call, call start. Then the composition runs, and it shows up in your application.

Similarly to with a QC renderer, right? It's not a composition, it's not a view. It's an execution engine. It's a renderer. You also have to create it and load the composition into it. These particular methods allow you to create it for a particular type of output destination. If it is that you're just doing a processing filter, in a sense you just want to -- it's calculating some math. It doesn't really do anything but give you back a spit out the output. Here's the average of these numbers, for example. You can just create a pro-processing renderer.

If it is that you want to do output to OpenGL context, you will create it with the OpenGL context, and you load it, you pass in the composition parameter. Or you can create it for an off-screen. Let's say you wanted to grab frames from the composition. You create a QC renderer, and you would initialize it with the composition, set it all up, and you'd actually be able to start rendering off-screen frames so you capture them and write them out to a video file or something.

So once you've gotten the composition and you've loaded it up, just like a view, which is an actual connection between the composition and the output, similarly with a renderer, you have an output, you have the composition, you now have made an executable thing. So now you have to supply its inputs. Same type of API. Values for input keys and values for output keys. You can set the values for its input keys.

And with a renderer, the QC renderer, you can actually start stepping the renderer at particular frames. Say I want frame one, frame two, frame three, which is different than the view. It just runs. So in that way, if you wanted to actually use compositions and actually get individual frames and tightly control, this is an example of how you would do it in the APIs that we use.

Not only can you create compositions, you can do a lot of interesting things with them. Basically, Quartz Composer is used to experiment with CI filters. If you wanted to make your own little CI filter, and you have your little understanding of what you want to do, you can simply just say, "I want to make a Core Image filter, and here is the filter that I want it to do." And you can make a complete experiment. You can change it. I don't like the color of that red. You change the code. You modify it. It's all live. You sit there, I mean it's miraculous sometimes what you can do with it.

Here's an example of something that I just built. It was pretty easy to do. As I said, I was kind of, it was kind of mind blowing. So I got this movie. It's a roller coaster movie on the developer examples. And great, I got a movie. So, you know, in the graphics world, things are kind of problematic, and it's like a little bit of a roller coaster. So I got this movie, and I put it in, and I said great.

So now what I want to do is I want to sort of blur it, because this is what really roller coasters look like when you're flying down them. Down the roller coaster, it's all very blurred. So I simply just added a blur filter to it. That's all I had to do. Just take that, connect the inputs, put it to the output. That was it.

So now I said, OK, great. This is the graphics world that I'm accustomed to. It's a roller coaster. So I said, welcome to my world. It's a PDF document that has text, welcome to my world. All I did was then took that movie, which is blurred in the background, masked it, and that was my output. All I need to do is add those two patches, all live.

So then we say, okay, great, that's cool, but let's actually put a pinch on it. So all I did was then just added a pinch to that output, and that went to the... And then, this is great, even though my world is a completely blurred roller coaster, I do think about the sunset and the vacation that I want to go. So I basically sobered the text, threw the text with the pinch, and then put it on top of a movie that was a sunset, my favorite beach. And that was it.

So an example of the QC renderer, for example, is the video wall. You all saw that last year. What's actually happening there is you have one composition, and that composition is being farmed over multiple GPUs that

Appropriateness. Well, anytime you want to do sort of immersive visualization, anytime you want to do experimentations with the Quartz technologies, whether it is you want to have an image or a movie and see what kind of effect it is, it's great for rapid prototyping of CI filters.

It's great for rapid prototyping of just saying, well, does this GL slang thing work? I mean, it's a great tool that allows you to sort of efficiently go through, prototype things, find out what the final product is, and it's not just for prototyping. You can actually, once you've created this composition, you can then sort of say, this is how I want to deal with my workflow.

I then want to put it inside my application, and that's how I process my data. And you don't need to write code, really, except if you're doing a GL filter, a GL slang filter or a JavaScript. You can just basically, there's tons of little patches that you can connect together and do. logic so you may not ever need to write code.

So that was the Swiss Army Nice of a Quartz technology. That was Quartz Composer. So hopefully, you know, hopefully the goal of this session is to have you leave here and say, I think this stuff is really cool. I want to go to session X, session Y, session Z, because this is the area that I want to look at, instead of going blindly and say, well, when do I use Core Image? When do I use Core Animation? Do I use V-Image? Do I, you know, this is kind of the information that we're supposed to give you. You have to then order the thing on the menu. So I told you what was on the menu.

I told you how to pair your menu with your wines, and you have to figure out what you want to actually do with it and what sessions you need to go and find out more information. So the 2D graphics session, where you learn about Core Image and a little bit of Core Animation is on Wednesday at 9:00. High Performance Image Processing, which is about Core Image, that's at 9:00 a.m. also on Wednesday.

Oh, wow. And Core Animation Techniques. Learning about Core Animation. Go to that. That's at Wednesday at 2. No conflict there. And then Integrating with Quartz Composer. That's on Thursday. Labs, open hours, in the graphics and media labs, but specifically to do with the Quartz 2D lab. That's definitely a Tuesday or two. So hopefully you have gotten a sort of an appetite, your appetite has been wet for the different types of graphics technologies, and you'd like to learn about them all. So what I'd like to do is...