Let me introduce myself. I'm Ralph Bruhner. I manage the Quartz Composer Core Image and Core Video projects, and Core Image is, well, what this session here is all about. So let me give you a brief overview of what's going to happen. So yeah, there's a short description of what core image is, just in case you have missed last year's session or didn't get around to read the documentation.

But that's going to be very brief, so I kind of assumed that you have heard about it in the meantime. Then I will do a short description of some of the sample code you find on the TigerDVD. And essentially just pointing out what -- where to look if you're having a specific certain type of problem. And then we go to the star of the day, which is the CI annotation app, which is the actual hands-on part of this session, which Frank Dubke will do that part.

And then Mark Zimmer will come up and demonstrate to you how to build two Image Units from scratch, and then we're going to show you how to use them, in this case in motion. And after a bunch of demos, depending on how much time we have, there will be pointers to arrays of sessions to where to go to learn more.

So, about Core Image. Well, Core Image is a new image processing framework. It was introduced in Tiger, and the goal of it is to do image processing on the GPU. It has a full floating-point pipeline, and it's fully color managed. It comes with a set of 100 filters, and there is this concept of Image Units, which is essentially a plug-in architecture so that developers can extend the base set. If an adequate GPU is not available so that the Core Image cannot do the processing on the GPU, it will compile on the fly to the CPU, and it will actually produce AlteVec code, or it will produce SSE3 code to execute your filter.

Okay, so what are you going to do with it? I'd like to encourage you to use Core Image wherever you can because, well, that's my agenda. If you have applications that do in-place, would have a place for in-place image editing, then you should consider doing that. So, for example, if you do a word processor, images are probably not your primary type of data you deal with, but somebody can drag in an image and you should do something decent with it.

So, one option is to give the user an inspector where they can change the properties of this image, like brightness, contrast, the usual stuff, so that the user doesn't have to switch to another app to get that functionality for simple edits. And the key thing to remember is... Core Image is built around the idea of doing everything in real time. So you do your image processing in the display pipeline.

The opposite of that is essentially doing it offline. So you take your image, you apply a filter, and you write out bits, and you keep these bits around for, well, however long your application lives. This is not the right model if you want real-time feedback. So the right model is actually when you draw the rest of your document, you also draw the image through the filter and have the filter applied just at the time, well, at draw time.

And that essentially means there is no apply button and there is no progress bar that comes up that lets you wait until it's done. Plus it makes things like multi-level undo really easy because you're no longer having to store megabytes of bits, you're just storing these three parameters for this filter.

Anyway, if your application works with video, then, well, think about putting up color controls, again, brightness, saturation, the bread and butter stuff. And you get a good example of that in the QuickTime player where you can go to the menu, get the color controls, and do essentially color adjustment in real time, which, in fact, is using Core Image.

Further, if it makes sense for your application to host image units, then please do so. And that might not be a totally obvious thing in some cases. Clearly, it's an obvious thing if all you're doing is image processing, then you probably want to have some kind of plugin architecture.

But let's say you have a very specialized app. Say you're looking at satellite imagery and you're probably grabbing something from Google Maps or what do I know? And you have basic image processing abilities in there, maybe false coloring and stuff like that. It might not be obvious that you should host all image units because you look at what comes with Tiger and see things like bump distortion and twirl effects and then you say, well, that doesn't really make sense in my app, so I don't do it. Well, by hosting image units, you give an opportunity to an other third-party developer to actually extend your app.

So if somebody builds an image unit that can, say, detect roads and do some skinning. . And, yeah, if you write your own custom filters, consider writing them as an Image Unit so that other Image Unit hosts can actually make use of them. And last but not least, you can also use Core Image to do UI effects.

You can think of the most wacky things like putting particles on buttons and spotlights and whatever. Please apply some taste when doing that. But, I mean, having a smooth, rich user interface is definitely for a consumer app a plus, So you should consider looking into this as well.

So how does this all work? Well, us in the Core Image team, we are essentially a bunch of lazy bastards. So whenever... Whenever you ask Core Image to do something, chances are it will actually not do it. It will write a little note saying, "Well, at one point I'll have to do this because somebody asked me to." And that's a good thing. Sorry about that.

So the only time that Core Image actually does work is when you render the image. So when you take an image and you apply a filter, what's really happening is there is a note being added, this filter needs to be applied. And then you apply the next filter and you just append to that, okay, after this, do this as well. And when you go to draw, there is a just-in-time compiler that will take these different pieces that you requested, compile them into a single program, and does all the processing in one step.

I think I skipped a point here. Okay. So, and why is this a good thing? So, if you take your top-of-the-line G5 today, the time it takes to get a single floating-point value from memory into the CPU

So using Core Image in this way by actually letting it to concatenate filters is usually a huge performance gain.

I have to talk about color management as well a bit. So as I mentioned before, Core Image, the full pipeline is color managed, which means when an image comes in, it has a certain ICC profile attached to it. It gets mapped into a working color space. All the different filters are actually applied in that working color space.

And then when you go to the destination, this another color matching operation is happening where the working color space gets matched to the display, for example, if you draw to display or whatever the target is. And there is a working color space is something you can set when you create the context, but there is a default one which tends to work pretty well, which is we chose generic RGB HDR.

So what that means is it's the primaries of generic RGB. So it's very close to what you usually be used to in terms of full red is still full red in that. So full red is one zero zero in this, this working color space, which is useful, but it is light linear and has an infinite gamut. So infinite gamut means we actually allow values that are above one or negative. So that way you don't have any gamut clipping by doing these color space conversions. Okay, so with that, I will go to the demo.

and give you a little walkthrough through some of the sample code. So the simplest Sample code we have is the CI exposure sample. Let me just run this. Make this a bit bigger. This is kind of the hello world of Core Image. It's probably about 20 lines of code, and all it does, it gives you a slider to adjust the exposure value on this image. And I will show you in a minute how this works.

So this is the source code for CI ExposureView with a really large font. And everything magic is happening in the draw-rect method when we actually draw the image. So the first thing we do, we get the CI context. In this case, we use the one that AppKit allocates for you.

And then there's a bunch of initialization happening here, which I will go into in a minute. And then the actual filter is being applied at the bottom. So first we take the value from the slider and set it to that input EV input of the filter. And then we go to the context, say draw image, we ask the filter for its output image, and then there's essentially a bunch of coordinates, which piece from the source needs to go to which piece in the destination. And that's pretty much all there is. So the piece that I didn't explain is the actual initialization part. So here, these two lines load the image from disk, in this case, the ROSE.jpg.

And this line creates the actual filter. It uses the exposure adjust filter here. And it presets one of the inputs right on creation time. In this case, it sets the input image to the rows image we just loaded. Because during moving the slider, the image doesn't change, so we can just do that during initialization time.

Okay, so this is essentially the simplest filter you can have and simplest use of Core Image, and this is a good starting point. So let me show you a second example called Micropaint. MicroPoint is similarly something like 30 lines of code, and it's a small paint program. It's really, really simple. Looks like this. You can set the color.

And a brush size. And clearly I'm drawing a masterpiece here. So let me show you the code, how this actually works. Okay, there's again a bunch of initialization that is not terribly interesting, so let's go to the interesting part. We create a brush filter. A brush filter is, we use here, is a radial gradient. So we put the opaque color in the center, we start off with black, and we let it fade out to the rim of the brush to a translucent color. So we allocate radial gradient, and it has this input color one, input radius.

Some of these pieces get pre-initialized and then overwritten slightly later, so let me show you that. How the brush, the DAB, is actually put into the canvas, we use a composite filter, so that is this piece here. So we use a source over compositing filter to take the current state of the canvas and the brush and merge it together.

So the key ingredient for the micropaint example is that CI accumulator, this guy here. So an eCMI accumulator, think of it as a buffer. You can take the state out of the eCMI accumulator and you get an image. You apply a filter to that image and you can write that result back into the eCMI accumulator. So this gives you that feedback loop so you can accumulate change. So creating a eCMI accumulator is essentially this call here. You give it a size and you give it a format.

And we initialize it using a constant color generator. We take a constant color generator with white, and this is the first image we stash into our accumulator. So the canvas starts out with white, which seems like a reasonable default. And then yeah, exactly here, this sets the image back into the accumulator, the white image.

Okay, so the actual drawing happens in this mousetrap method here. So we take the values from the UI, in this case we take the brush size from the slider and set that to our brush filter. We allocate a color, again from the UI, and set that on the brush filter. And last but not least, we set the brush filter center to the XY coordinate that we get from the mouse.

And then there's a bunch of concatenation happening here. So first we take the output image of the brush filter. So this is the DAP and set it as the input image of the composite filter. And the second image that goes into composite image, the background image, is the state of the accumulator. So we take the state of the canvas, we take the DAP, and the composite filter puts all of these on top of each other. And at the very end, that resulting image needs to go back into the accumulator. So this is this set image call here.

Take the output image of the composite, put it back in. And there's an interesting optimization going on down here when we do This set image call is actually in the super class, which I haven't showed here yet, which essentially just draws a CI image. This guy actually passes a dirty rack, so we only update the part that has changed by putting down the DAP, so we are a bit more efficient in terms of updating the screen.

Okay, so let me point out something here in this little example. One flaw of this is if you move the mouse fast enough, you get individual dabs and you don't get a connected line, what you probably would expect. And a bit later in this session, there will be an example how to do a proper connected drawing.

Another example I have is the CI transition example. And this is essentially running nine different transition effects at the same time.

You can explore different types of UI doing stuff like this. So the idea is if you have to put a UI up to let the user select a different type of transition effect for, say, I don't know, a presentation app perhaps, one way to do it is have a little pop-up that lists wipe left, wipe right, ripple, and so on. Or you can just show the user by making little thumbnails of the actual transition going. So naturally it wouldn't make them that big, but it makes a better demo, so I did it that way. Okay.

So, additional source code, there is also a WebKit problem, a WebKit plugin. So, we have a new version of Core Image available, which shows you how to put an image into a web page and have filters on that one, so you can get that from the ADC website.

And with that, we're going to start with the actual... Ladies and gentlemen, this is Frank Dupke, who will guide you through this exciting-- Can we go back to slides? Thank you. Thanks all for coming. My name is Frank Döpke, and I will show now the CI Annotation app, which is our hands-on part of the session.

So let me tell you what we'll actually be looking at. The CI Notation app, we looked at like satellite images, and they have, these days, a little bit of a problem. They're really, really big, and they offer lots of little details. So to show them, you either have to scale them down to see everything, but then you don't see the juicy details anymore, or if you blow it up, then you kind of lose your track.

So this is the one part that we wanted to show in this application, how we can get around this. And the other part is also that you want to put out like, you know, you remember like on the old postcards, this was my hotel room where we stayed. You want to do some markings on these images, so that's what we want to do here as well. But we don't want to touch the original image, so we will show actually how to do stuff on top of these images.

And with this, I would like to go to the demo machine. Demo, hello. Thank you. Okay, so the image that we will be looking at is actually an image from Washington, DC. So I hope that you all filled out the security form that's under your seat for the department. OK, we don't have to do that. Good.

So this image will take a little moment to load because it's like 10,000 by 5,000 pixels. So what we have here is now our view of Washington, D.C., and we can color correct this. I would say this could be a little bit brighter, a little bit darker. We can change the saturation, make this a little bit more nicer green. It looks a little bit juicier this way. So now I can look at this image. I can actually scroll it around. This is also a nice part that we have. So I have a good overview of everything.

Then you see in the middle, we have this big, big circle. And this is actually our lens in which we now can actually look at the more detailed part. As you can see, I can move this around. And see, oh, what's going on there? Okay, there's like a big traffic jam. This is like one of the things. Now I can use my marker here and say, whoa, this is There we have a big kind of traffic jam. We can also put some text on top of this.

Okay, what else do we see in this image here? Let me finish up the text. Aha, there's something down here, so let's see. A little pointy thingy here. Anybody knows what that is? Very good. Okay, then we can look a little bit further. Then we have this building here. There's a very famous garden around it. So what is that, of course? That is, of course, Busch Gardens.

So let's have a look at this incredible rooftop. It's very smooth. I wonder how that happened. So you can see, that's like the nice part, you can see all the juicy little details, but we still have a good overview of at least this section of Washington. Now I said that I can also do this without actually really destroying the original image, and for that part, actually we can peel this whole thing off.

And you see actually there's a little bug that I've done here. So it's like the whole image turns a little bit gray. So this is not because I didn't sacrifice a sheep to the demo gods. There's actually a bug in the code, which I'll show how to fix that later.

So this is what this application can do. And now let's have a look into the code, what we actually did here. And this is all available on the WWC website. You should also have all access to the source code. And let me go first to the slides to explain a little bit how this whole thing looks like. So we have, can we go back to slides? Thank you.

Okay. So we have our application. It has a single document Cocoa application. We have a controller, some documents, and we have a view that just handles our mouse handling part. And then as I showed already, there is some stuff like stacked on top of each other. So we work with layers, and these layers have a CI layer, which is our abstract base class. We have one layer for the painting part. That was my red marker. We have one for the text. Well, I could write stupid comments onto the image.

And then we have the actual image layer in which we really handle the real image. Then you also find some utility functions in the sample code. And there's especially interesting the part, the sample-CL scroll view. For those who have looked at our original sample code, I want to show you what the difference is there.

When we look a little bit closer, you saw that I was able to scroll. For those who played around with original sets of sample code that is on the Tiger disk, it was always the question, OK, we have this. This is an OpenGL view, but how can I do scrolling when I have a bigger image? And this is exactly what we've done here. I will show you in the code what actually made this view scroll.

Then you see actually that we've done some text editing on top of the OpenGL, so for that I don't want to go into too many details here as we're not a Cocoa session, really. There's just some child window that we had to put on top to do the text editing, for those who are curious. Then we look at how we did the image import and export, because we can actually save the image, I will show that in a moment as well. And then we look a little bit further into the compositing part.

As I mentioned already, there's something special about this view. So far we already kind of propagated the sample CI view. And you saw that, for instance, in the transition sample that Ralf showed. It is very easy for you to use, so you don't have to deal with the OpenGL underpinnings of it.

And it supports that you can just do a selective drawing. And that helps you actually quite a bit on this part. Now if you look at the sample CI scroll view, it has the same bells and whistles as the sample CI view, but it allows you scrolling. And we do this by actually setting the autographic projection matrix of OpenGL to really move your view around. So this is one of the things that we will be looking at into the code.

And then we look at the compositing part. So I again use the CI source over compositing, which helps me to put this different stuff on top of each other. And this is important to use this rather than just drawing the images on top of each other, because as Ralph mentioned, it gives you the chance that these filters actually can collapse into one pipeline and therefore be faster executed. And the other part is where we go a little bit more creative. You saw that I used the transition, actually the peel-off transition to do also compositing. So this is just a little bit different way of doing some compositing, giving some spice to it.

The layers that we want to look at actually is the real interesting part. So we have a paint layer where we again use the annotation part. And I will show you a little bit difference with the filter. Then I'll show you actually that we scaled down the image and we applied a filter that again scaled it up. This is the next important thing.

And then we look in last but not least at the document. This is actually what brings everything together, brings it onto the screen. And it actually does also like the output when we save out the file. And with this, I would like to go back to the demo machine.

Okay, so as promised, we look now at some code stuff. So the CI layer, that is our first part. This is just as I said the abstract base class. We init, it was a delegate. The delegate is actually our document, which gives us the chance that we can really talk back to the document, making it redraw. It does the mouse handling.

It returns an image. So this is like the result of our layer. And it has a rectangle, which gives us the size. This is kind of the part that we wanted to look at. Okay, then let's have a look into the text layers, our first real layer here.

So how do I do the text rendering? The interesting part first is if we create a text layer, I have two ways of this, and this is like for the demo purposes. To render text, we need some kind of a CG context. We use CG for the actual text drawing.

I can do this by either using a CG layer, which gives us an advantage because I can use the CG layer coming from the CI context. It has the proper color space attached to it, and therefore is faster with the rendering as we don't have to do any color transformations there.

But if you have already lots of like rendering in your code, and you don't always have access to the CI view, you can also create a bitmap context, and this is what I'm doing here in the second part. That allows me, again, I create a CG bitmap context, and I will then transform that into a CI image later on. So I can render into this bitmap context for those who are familiar with CG drawing.

OK. Then when we look into the real rendering part, there is again the same piece that we have here. So right now I actually can use the CI layer part, which makes the code much, much easier. After I've drawn my text, which is actually happening here, all the little text objects draw themselves into this layer.

And all what I have to do is I create a CI image out of it. And that is the one that I want to return then as the result of this layer. If I'm working with a bitmap context, then it's slightly more complicated. As I first have to create a CG image from this bitmap context and then a CI image out of this again.

So this is how text draws in our application. Now we have a look at how painting works. Let me show you what filters are. You saw already we used an accumulator. We have a brush filter. and a composite filter, those are the main pieces that bring everything together. I used a static CI color here and also the static brush size, feel free to play around with it, it just makes the code a little bit easier for me. So how did we do it again? We set up a bunch of stuff in our initialization.

So as I said, we have fixed color and fixed brush size. Now the brush filter, you see there's a paint filter. This is actually an Image Unit that I'm loading. This is a special filter that we set up. And we will show a little bit later how this filter really works. Then again, I set up my composite filter and my accumulator, clean my accumulator, and I'm ready to draw.

And then it's very much the same thing. When we look into our mouse down handling, I just, the only difference here is like in comparison to the paint part, I'm not just getting one point, I'm remembering the last coordinate from the mouse and just a new one, and therefore just can draw basically a line, so which gives me a continuous brush stroke. That's why I have to actually set two values on the brush filter, it's like for my last point, and the new point, and then just add this together into my accumulator, and then I'm just telling the document something has changed, please redraw. So that's the painting part.

Now let's have a look at what the image does. On the image layer, as I said, we need to scale it down. We have this big, big image, and we want to scale that down so that we have a scale filter. Then you saw the lens, which was our little magnifier that we had on top. And you saw that I did some color corrections. That is actually the color filter. And we have the resulting image, so I'm having this as here as well. Okay, so how does this code look like? Again, going into the initialization.

The CI Color Controls is your absolute standard for Setting up the color values on this. And then I use the CI Affine Transform. This is actually where I scale the image. The important part is actually that I apply the Affine Transform after the color controls and right before I use my next filter, which is the Lens filter. You can actually see from the name again, this is not a standard filter. This is again an Image Unit that we provided and we'll show later on how that will work.

And I just have to set up my lens field. And you see, I'm not setting every single parameter. The lens field actually has a few different parameters that I can set, and Mark Zimmer will talk about this in a moment. So, but I'm not setting everyone specifically, so it's a good idea in that moment to call setDefaults, so that the filter is initialized with good values. So, unless you set every parameter of your filter, please call setDefaults. Here I'm now after that setting the sum parameters of this filter and making sure, okay, how big is really my image that I want to bring up, and that's actually the size of my document.

So, and again, when we come into the mouse down part, all I have to do here is that I actually find out, like, where was the old lens, and now I move it with the mouse to a different position, calculate the unified rectangle from it, and then I can let the layer redraw with this part. So that is all we needed to do for our filter. And now we can look into the document part. This is a little small.

You can see my eyesight is really bad. I need a big font to read this stuff. Okay, so what I'm doing in the setting up part is like I create a whole bunch of filters. So I create my layers, which is first the background image. That actually tells me how big my document will be. I create my text layer, my paint layer. And then I said I use the page curl transition. This is for the peeling off part.

I make sure, okay, that it's initialized. And I use actually like a transfer the coordinates of my mouse later on to the time. Normally transitions, you'd think in terms of time that passes by. So I have like my peel time, but this is like just transferred from the mouse coordinates. So there's a whole bunch of parameters that I have to set on this filter to make this look correctly.

And then I need a couple of composite filters. And the first one is there to actually composite my painting together with the text, so that I have one image layer coming out of that. As that goes on one side of my filter. And then we have a second part, and that's actually a little bit also where my bug sits.

We have this parchment backing image. My idea was if I peel off this transparent layer, which just had the text, it looked a little bit goofy if there was nothing. So the idea was that on the back side we have a slightly transparent image that I want to use. And that's actually where my little problem was, that I forgot to do one thing. So I'm actually using here a color generator as my filter. And when I do that, that basically has an infinite extent.

So this image is huge. And that is causing our little problem in the part where we actually now want to correct. And actually, since I don't have to type so much, I prepared this a little bit So we will crop the image, and therefore make sure that it has the same size actually as the front side of our transition. So let me actually try to fix this here. So bear with me and hope that our demo gods are with us this time.

So I insert a crop filter, set it to be, so the input of this filter is again the output of my color filter. Then I set the size of the crop filter to be the rectangle of my document. And all I have to do now is actually, instead of the output of the color filter, we use the output of my crop filter. That should fix my problem. Let's have a look quickly at the application, if it does know what I'm expecting it to do.

"Anytime, there we go. Load our image. This is what our image is about. And now you can see the peels off much nicer. Debugging right in front of your eyes. Thank you." So that was the first part that we wanted to look at in the document. Now, how does the document actually put everything together? And for that part, we actually have to look at how we create our output image.

So as I said, I'm putting together the text and the paint layers into one image. So for that, I use a composite filter. And what I've done here to a special case, like when I'm doing all the interaction with it, I actually don't use the peel-off filter. So I have the special case if there's no peeling off, then I just skip that part.

Otherwise, and it's the more interesting part, so I'm using actually then the output of the layers, put those as the input images of my peel-off filter, together with the backing image of it. And as a result, I use the output of the peel-off filter right here. That is the image that I really want to bring on screen.

And this is again where we just use the set image part on the, well, our little subclass of the CI sample view. And that is the next part that we can just have a look at. And that is the next part that we can just have a look at.

So the interesting part here is again, this is our sample CI scrollable view, which is very, very similar to the original one. You do a set image on it. The only difference is if you look at it and compare those, it's in the update matrices. So here what we actually do is we look at the mapped rectangle that you get from the enclosing scrollable view. And we set up the viewport and the glAuto for this to actually map now to this moved rectangle. And that's how our drawing is simply then mapped, and and therefore we can scroll.

Okay, so the last part that we want to look at here in the document again, Now we've done everything on screen. I can actually save this image because this was also a question that came up very often. And to do that, actually, all I have to do is I create a URL. So if somebody is telling me, okay, I want to save it in this place. So I have a URL which tells me, like, the destination of it.

And now Image.io exposes a new set of APIs in Tiger that we actually have available through CG. And what I'm creating is actually an image destination. This is the part here. And I'm setting it up to be public JPEG in this moment. I want to write on a JPEG file, but there are many other file formats that you can use.

So I have now my destination, and the next part is, so I'm from my context, I'm getting a CG image. So I simply ask CI, okay, give me an output image in form of a CG that I get from your context. I add it to my destination, and with finalized advice on the file, I'm done. That's all I have to do.

And we can have a look at this a little bit later. So, that is how I can save out the file. There's one more thing that kind of bothered me a little bit in this application. When you saw the text actually did not show that clearly on the satellite image. So, let's actually have another look in what we can do with the text part. So, let me bring up my text layer. And my little notes.

So what I wanted to do for the text layer is actually that I need a few things. I wanted to actually make a little glow behind the text. The way I actually tried that out is I used our Funhaus application, which you also find on the TigerDisc, so I can play around with the parameters. And I found that if I actually put a white glow behind the text, then it better distinguishes itself from the image.

So what I did is actually I take this black text that I'm writing, invert it, so I get a white text, and then I'm blurring it out by using a Gaussian blur. And then I composite again my black text on top of it. So I need three filters: an invert filter, a blur filter, and last but not least, again, I need to composite it. So we have three filters.

So as I said, I'm doing an invert. That's my first part. I need a blur filter and I use CI Gaussian Blur. And as I said, I played around with the Funnels app to find out the other three is a good radius for what I wanted to achieve. And I need a composite filter. So I put this into my initialization. Of course I want to save it.

Just in the end of my layer, I add these three filters. And of course, since I retain those, remember to release those in the dialog part as well. Wanna be good citizens here. And now comes the part in the rendering. We need to change this. So what I'm doing is in the rendering of the text, let me scroll down here. As a result, I normally just have this text image. Oh no, it gets really dark there. I actually put this... Lost the screen. Okay. Sorry about that. I just thought everybody was getting dark here.

Okay, so I put this text image into my invert filter, this gives me the white text. I run my blur on it, then I composite those two together. That's basically my effect. I'm putting that at the end of my rendering part. And then last but not least, The function that returns the layer image, instead of just returning the text image, I need to return the result of our composite filter. So we need to change this.

Looks much better, right? So that's how simple it is to change something. Thank you. and I promise you this application can save. It actually can print, but that's so 20th century. So let me actually save this image out and just put this here on the desktop. Always good to clutter the desktop. It opens it up in preview, and there's our regular JPEG image. There we go.

So now to go a little bit more into the depths of the filters, I actually would like to bring Mark Zimmer up on stage to talk about the Image Units that we have here. Thanks, Frank. Can we go to the slides? All right. Okay, so no matter what you need in image processing, you may be able to take existing Core Image filters and put them together, Image Units, put them together, or you may be required to make new ones, as the two Image Units, in this case, the one for painting that produces continuous strokes, and the lens filter were produced as Image Units. Okay, in the Paint Image Unit, the first thing is that when you saw micro paint, we're employing CI Image Accumulator so that you can accumulate the results of your painting.

But the problem that we had is that they weren't continuous. In fact, in order to make them continuous, you have to do something special. Basically, rendering a lozenge instead of a circle is a quick way to do that. So let's talk about how we do that. Well, instead of one point, we input two points to make it continuous.

Then, what we do is we make sure that it has a given width and width on the end. We use circular end caps to ensure that. So we have a variable input width that controls that. What we do to actually make this work is we need to evaluate a composite distance function that has a different value in the three different half spaces that you see.

So we'll use a 2D dot product to get the half spaces on either side, on the side of input point one and input point two. And we'll use a Euclidean distance in those half spaces to make the circular distance function. A 2D cross product is a simple technique to get distance from a line. We'll employ that in the space nearest the line. Once we have our distance function, we just threshold it to make a lozenge alpha, and then we apply color to it and we're done.

Okay, so let's look first at the, oh I hate when this happens. Let's look first at the paint filter. You can see we've got the two points, a width and a color. Just really not a lot of details to this. Turns out it's pretty simple. When we're actually putting it together, the main thing is all you need to do is just bring in the kernel like you do in every other image unit. So if I had the CI demo image unit, I could just take that and modify that to make this, and all I really would be changing was paint.cikernel and paintfilter here.

The custom attributes are straightforward. Again, two points, a width, and a color. And here we're giving them default values. So if you happen to bring this into Core Image Funhouse or into Motion, you can actually see the parameters with the appropriate values. Okay, now in output image, there's only one trick. Because this is a generator, it inputs no images. All you really do, all you really need to know is the domain of definition.

In other words, the area that you're rendering. And to calculate the area that you're rendering, all I really do is put a rectangle around the two input points, outsize it by the radius. That's the opposite of insetting, I suppose. And probably this section of code could be rewritten to be like two lines of code with a CG rect inset, sorry.

And then I evaluate the vector here so that I can end up doing dot products to calculate inside or outside of the two half spaces. And I pass that in as v01. And then I call the kernel. Okay, so let's have a look at the kernel itself. It's being passed 0.1, 0.2, the radius, the vector for the dot products, and the color.

The kernel is also quite simple. So, remember how I said to evaluate a composite distance function? Well, what that means is you'll evaluate three distance functions and then you'll composite them together using compares. So, why are we doing that? Because the kernel is evaluated for every pixel in every working space pixel in the output. And so what that means is you have to decide what's going to happen in that, evaluate all the possibilities, and then select from them. That's how to get that information.

The absolute value of the Z of the cross product gives us the distance to the line. If I didn't do absolute value, it would be positive on one side of the line, negative on the other. The space closest to point zero is evaluated here, and it also will take that distance and decide whether or not the dot product is less than zero and choose one distance over the other. So that gives you one level of compositing, that gives you one half space and the mid space of the line. And then we do the same with P1, exactly the same kind of code.

You'll notice that Dist 2 and Dist 1 are switched. That's because the dot products on either half space point the opposite directions. And then, this is a trick that we pull several times in this demo. Once I have a distance function, I can subtract radius minus that distance function and clamp it to 0, 1.

And what that'll do is it'll give me a function that's between 0 at exactly radius and 1 at radius minus 1. And that gives me the anti-aliased edge. I'll multiply that by the color. Remember, we use pre-multiplied color in Core Image. And that is the result that we're returning for the paint. Okay, so now let's go back to the slides.

Slides, please. Thank you. Okay, the Lens Image Unit was a bit more complicated. With the Lens Image Unit, we have to do various things. We have to magnify without blurring. We'll talk about how we actually achieve that. I'm embedding it in a ring to give it a little bit of weight so it looks like a real object.

It needs a shine on it, too. There's other things that you could do that would extend this and make it look more interesting. If you have an app that you need to use something like this in, you may want to pull some of these things out of the hat. Chromatic aberration is something that you could do. These are exercises for you, the user. Reader. Watcher.

Some distortion inside the magnifier. Yeah, there's, whenever I use a magnifier, I notice there's some distortion, especially near the edge. That's a trick that you could make it do as well. You could put a caustic under the lens. That's just like you're trying to magnify or burn those ants. Remember, I used to do that when I was a kid. I was a bad boy.

And there's other things you could do. Like you put a little shadow behind it, too, with a Gaussian blur. You saw exactly how to do that, only in reverse, with the text before. So now let's talk about the details of how it really works. The first parameter is input image, and no, it's not capitalized. Sorry about that.

All right. So in order to magnify inside, there's a trick that you have to play. The image, remember he told you it was like 5,000 by 10,000 pixels? Well, what you do is you do use a view transform using a CIFI transform to make it smaller on screen. And then what happens is I will use a counteracting transform to zoom it back in so I can see inside the lens at full resolution.

Okay, so I'll use an additional CI sampler with its own affine transform for the magnified data. That's the trick. So, adjust it to counteract the view transform and get to the individual pixels. In other words, if it's exactly the opposite scale, then you get one-to-one pixels on screen. Exactly the visual trick we want.

To define the lens, we're going to need input center and input width, just two parameters that clearly just define a circle. That's the area in which we do our magnification. What we do with this is we overlay a shine with it using what we call a shine map.

Shine map is basically a map that's circular, that has alpha in it, and the alpha will tell it the parts that are opaque and the parts that are transparent, the parts that overlay 100%, and the parts that only overlay a little bit to make it lighter or darker. And you can see there, it does make the lens lighter and darker as well. And input roundness is used to control how much of that gets loaded into the circle of the lens.

Premultiplied alpha, of course, is the rule for Core Image. Now, when we're putting the lens on around it, it's a ring, we actually have to render that ring, and we're going to fill it with an environment map, in this case, a material, which is the reflections of a globe.

So I captured that with a digital camera, and then I resized and blurred it so it was the right size for use inside of Core Image. The right answer is to really use a sharp map and then blur it inside using map width over fillet width, using CI Gaussian blur. That would have been much better, but I didn't want to waste the time to do that, sorry. Okay, to create the ring is a little bit challenging. So what I did with the ring is we have an inside fillet and an outside fillet.

Okay, so here we have the inside fillet and the outside fillet. The outside fillet goes from outer radius minus fillet radius to outer radius, etc. And what we want to do with this, specifically, is to create a 2D normal vector for every pixel on the ring. It's actually pretty easy to compute. Normally, it's easy to compute just like the distance field.

You subtract the center from the current point, and then you normalize it to give it a unit vector. Okay, but it typically points outwards. So what I'll do is I'll scale the normal vector inward for the inner fillet. Interesting language there. In the ring interior, the normal should be scaled to zero length so that you just get the center pixel of the map.

And here's the surface shape function that I'm going to use. It goes from minus one on the inner radius to zero in the middle of the ring, and then back up to one in the outer radius. And that's, I'm just going to multiply that normalized vector by this value, and that will give me the appropriate two-dimensional normal vector for every point on the ring.

Okay. Now, the environment map is just a file, just a texture. So it has a given height and width, and what we want to do is we want to take the 2D normal vectors, which always go from minus one, plus one, minus one, plus one in both axes, to zero dot width to zero dot h. And so we end up doing a little bit of scaling inside there, and you'll see when we get into the program exactly what it does. We're going to need a specific width for the ring.

The fillet radius, typically when you use it, is going to be half the ring width. The edges of the ring have to be anti-alias, and we'll pull that same basic trick out of our hat that we used for the paint. Okay, let's go back to the demo machine.

Okay, so as you can see, the Lens Image Unit is a bit more complicated. It has, it inputs, it's the image that we're magnifying, the center of the lens, the width of the lens, The width of the ring, etc., etc. And then we've got various things, magnification, fillet radius of the ring, the roundness, and the shine opacity, which allows to control the various elements of its appearance.

Inside of here, probably the only difference between this and an init procedure you'll see in CI Demo Image Unit is the fact that we're bringing in images as well. And also the fact that our kernel, we bring in two kernels instead of one. So this is an example of a multi-pass Image Unit.

So here we're just getting our path, we're getting a URL, we're bringing in the ring material image and the lens shine image, the two ancillary images that are required by this filter. The custom attributes, we have a lot of attributes, so I won't bore you with the details of that. You can see you just need to set up their parameters.

For the lens, you need a region of interest for this. We'll talk about that a little later, but let's just comfort ourselves with the concept that we need to provide the entire extent of the shine image because it's something that you never know which pixel you're going to load, so this is a trick that you end up playing.

And then the area of the original image that we use inside the magnifier, really all you have to do is subtract off the center, multiply it by the magnification, and then create a rectangle based on the width of the lens, and you have the area of the original image that gets used. So the region of interest really is the amount of the sampler that you're using in order to render your result.

Okay, there's also an ROI for the ring, which is only provided because the material map with the Chrome needs an extent. It needs to know the entire extent of the material map. Okay, sort of the complex part, Output Image, is the way you define how the filters and the kernels get strung together in order to produce your image result.

And in this case here, What we're going to do is we're going to first set up that sampler that's the non-magnified, first the source is the non-magnified sampler for the image. This is the image under transform that was done outside. But because it's Core Image, we can apply a transform to that and they'll cancel each other out because of lazy evaluation. Remember, we're lazy bastards. We are.

Okay. Seems like a lot of code though. Okay. "It's just the way I write code. Alright, so here's the magnified source. Again, all I did is create a, I input the magnification float value, created a transform associated with it, and all it does is translate to the center, magnify, and translate back. It's really, really no complexity to it at all. Then I create the source, and it's ready to use.

Now, you'll need to calculate a rectangle for these effects because the domain of definition is required for these at any rate. The lens actually is partly a distortion effect and partly a generator. So that complicates matters slightly, but not a huge amount. So all I have to do is calculate a rectangle for the lens effect, and that rectangle ends up getting used with the source definition as the domain of definition for the lensed image.

So here the lens kernel is applied with the various parameters. Now the ring is applied as a separate item, and then I use our old friend CI source over compositing to put them together. And here's where the ring is applied. So I'll get the material map for the ring. I'll memorize the ring width, which is used in calculating the domain of definition here for the ring. The domain of definition is passed in here as part of the apply for the ring, and then the ring kernel is called.

Now one thing that we're doing also is if we have an ROI procedure, we have to assign an ROI selector here before the kernel. And in fact, you pass the kernel in to that to bind them. And then later we'll do in the apply, we'll say apply option user info and then pass whatever object you want. This can be an array of other objects. Turns out it's important to do that because you never know if your instance variables are really still going to be around and set to the same values when the filter is actually evaluated.

Okay, and that's why we passed the user, the information for the ROI in with the user info. Okay, so there it applies the ring kernel and we're pretty much done. All we need to do is have a look at the kernels and you'll see what it's about. So, the lens kernel is the first thing. It's what does the magnification.

It doesn't really have a lot to it. The destination coordinate here, this is our current coordinate and working coordinates. We subtract the center, take length, and normalize, which is our standard breakout for calculating a distance field with a unit vector. Then, I do a little extra work to decide how much of the map, in other words the shine map, is going to get applied to this. And then I'll evaluate the shine map pixel.

Over here I take highlight size, which is the width and height of the highlight shine map, and multiply it by the unit vector times this map distance here plus 1 divided by 2. And that's how you get from -1, +1, -1, +1 to 0. . . width and 0. . . h of the map. So there's the trick that's used.

Okay, then I get the two pixels, the undistorted and the distorted pixels. Remember, all the work for Magnify has been done by the sampler and using the counteracting transform. Then I apply the highlight, and this is very much like what you would do in CI Source over compositing, but because I'm a lazy mm-hmm, I decided to do it here instead. And then I mix the results together. And here, I'm creating the alpha for the lensed area by taking radius minus distance once again and clamping it to 01. This is exactly the same trick I used in the paint filter.

All right. For the ring, which is a bit more complicated, it seems like it's shorter, it shouldn't be more complicated, but it is. We do the same thing with the distance field at the beginning. So I create distance and V0. V0 normal vector, distance, distance from the center. Then I create that function for the surface shape lookup, and I evaluate some linear functions here. And then I mix them together using this comparison. And so that gives me the graph that you were looking for, looking at before.

Then I multiply the unit vector by that number, and that gives me a two-dimensional normal for lookup into the material map. Then I have to, now I do a little digression to create the alpha for the ring. It uses the same basic trick that we used before, except it's kind of one is positive and the other is negative. Then I multiply them together and that gives me the correct alpha for the ring.

Finally, I do the lookup into the material map. So here's the completely undistorted view of how you take that normal and turn it into a material map, lookup coordinate. And then you do the lookup here. Finally, we multiply the color of the material map by the alpha, and that's it for the ring. That's all we have to do, because we're pre-multiplied alpha. Okay, so now I'd like to turn it back over to Ralph, and thanks very much. Okay, back to the slides please.

Okay, so I'd like to encourage you again, write Image Units. As Mark was explaining, it's really easy, and if not, well, something to learn. If you're using Core Image, consider being a host for Image Units. And there's actually a logo program, so if there is a validation tool on the ADC website, where you can run your Image Unit through, and it will tell you whether it conforms to some reasonable standards. And if you conform to that, then you can actually use this Image Unit logo on your packaging and tell the world that you're a good citizen when it comes to Image Units.

So with that, I just use the rest of the time with a bunch of more demos because, well, we have some time. Okay, so the first demo I'd like to show is, remember the Image Unit that Mark did, that essentially was a two-pass Image Unit? Demo machine, please. Okay.

So this is Motion, and Motion is a host for Image Units. So I took the Image Unit that Mark did and put it into a little Motion project. So let me deselect. Looks like this, and there is a lens zipping across. So this is, there is no additional code once you have the image unit in there, so you can actually go look at the lens.

Here is the lens layer. You can inspect it, filters. So you see in motion you have now the ring width, parameter, magnification, and so on. So everything that Mark was talking about. So, for the people in the audience that have motion on their laptops right now, you have to be careful. There is actually a bug that prevents that Image Unit from running with the version of motion you have. So there will be a bug-fixed release coming shortly so that it actually works well.

Okay, so the next example I have is actually something I borrowed from another session. There is an advanced new techniques with AppKit session which uses Core Image as well. And I would like to use that app. There is a long discussion about how that app works and what it all does in that session, which is on Friday. So go there if you're interested. The reason I'm having it here is they use an Image Unit as an effect in UI. So for example, when they switch between the inspector panes here, you see there's a little ripple effect going on.

It turns out it's actually really easy to write and go to that session to learn how to do it. And remember what I mentioned about applying taste when doing these things?

Everybody likes Google Maps, so I have a short example of this. So this is going out to the... Come on. It's going out to the network, which apparently stalls every now and then.

Okay, let me talk about what's going on while we're waiting for a server. So this is a little CI app which creates an image, and it's an image provider, which essentially means you're creating an image without providing the backing bits, and then you get a callback where you can provide tiles of your image data on demand. And that callback, in this case, goes out to the Google Maps server and requests the appropriate tiles.

So if I zoom in here... So you can go look at things, look at details. Now, the really cool thing about that is the code for this is extremely short. Essentially what you do, and it sounds really cool, is you create an image the size of the continent, and... Remember using double precision for the coordinates. And then when you get the callback which tile is actually needed, you go essentially build a URL, request the JPEG file, and show it.

And that's pretty much all. So the stuttering you see is every now and then it goes out to the network and requests a new tile as I'm moving through. And as I mentioned, the network isn't too fast at the moment. So this is really cool, but it's actually more of a demo for Google Maps than for XIV Core Image, so we decided we have to put something else on top of it.

So, let me zoom out a little bit more. So, Frank had the idea of taking a database of California earthquakes with the locations and give you a little bar you see on the side there. So, in each date and the magnitude is in there. It's probably hard to read. It's a bit small. So, for example, there was an earthquake in -- Okay, we are going to the network again. Oops.

So we put a little ripple effect and a red glow, which is really comforting. On top, so this is an earthquake in 1986, so if you zoom out a little bit more, you actually see it's very close to the Bay Area, and had quite significant magnitude. Okay, so again, the app here is really simple.

It takes an image, which is the continent, and it zooms it up and renders it at whatever display resolution we have. And then we append the ripple filter and put a Gaussian dot on top with some red glow to make it stand out a bit more because the ripple alone wasn't too visible. Okay. So with that, I'm pretty much done, so let's go back to the slides.

Where to go from here? So there's a lab starting at noon developing image units. If you go to the Tiger lab, you will be able to talk to us and bring your code and we can look at it if you have problems. Tomorrow morning, there is the Core Video and QuickTime High Performance Video in Mac OS X Tiger session.

So, essentially, if you're interested in integrating Core Image with video, then that's the session where you have to go to. There is also a session about FXPlug, which will talk a little bit about how to use Image Units in motion. The general story is it will just work, but there are definitely some guidelines to obey that to make it work well.

For UI effects and AppKit integration with Core Image, you can go to the session on Friday, Advanced View Techniques in Cocoa, which was a set that Reducer App I was showing you before will be explained in great detail. There's definitely a lot more Core Image in there than I was showing.

And there's another Image Units in Motion FX plug in depth, if you want to go really deep. That's on Friday at 10. Okay, so that's pretty much it for this session. If you have a question, contact Travis Brown, Graphics and Image Evangelist, or come to the lab right after this session.