So my name is David Hayward, and we’ll be talking about creating image processing effects and filters using Core Image. This is what we’ll be talking about today. Give a brief introduction to Core Image for those of you who may be unfamiliar with the technology, and show how easy it is to get started using Core Image in your application. After that, we’ll get to the heart of our conversation today which is to show you how to write CIFilters. We have several great examples to show you how to do this. And then we’ll conclude with some final thoughts on debugging and other resources to consult.

So first, a quick introduction to Core Image. So what is Core Image? So first and foremost, Core Image is a framework that allows you to do ultra-ast image processing, and to apply the effects for fun or profit. It works in realtime with other parts of the system such as Core Video on video, or also on the user interface using Core Animation.

Runs optimized in all the supported Snow Leopard hardware, and it has several key features, not the least of which are floating point accuracy and a full color managed pipeline. In addition to the image processing framework, Core Image also includes a base set of 125 filters to get you started.

And in addition to those 125 filters you can extend that filter set with your own custom filters. And that we’ll be talking about considerably later on in the presentation today. So here’s how we use Core Image today on our operating system. It’s used throughout Mac OS X, our applications, and in third party applications as well. We use it for user interface candy, such as the Dashboard ripple effect, we use it for video effects in iChat and in Photo Booth for fun.

We also use it for screen savers and iTunes visualizers. And then we also use it in our professional and, you know, consumer type-applications like iPhoto and aperture to do image processing. So that’s where we use it. Let me show you a little bit about how it works as part of our overall framework. At the top, the key part of Core Image are its filters.

And these filters are written in an architecture-independent language that’s C-like, that’sbeen extended to have vector types, and is a subset of the OpenGL shading language. These kernels are then executed through Core Image, combined optimally, and then executed either using OpenGL or Open CL use -- either the CPU or the GPU, depending on what’s best for your application.

So here’s a brief illustration of how Core Image works in the most trivial example. Start out with an original image, and you want to apply one of our filters to it. In this simple example we want to apply a sepia tone filter, and the end result of this filter is a new image which is outputted, which is a sepia tone image. Obviously, this is a very simple example, we can start to contrive more complex examples by combining filters. And this is where the flexibility of Core Image really starts to grow.

And this simple example we now have 2 filters. An original image goes through a sepia tone filter, and then the output of that goes into a hue rotation filter. And that gives us a look at the output which looks kind of like a blue tone rather than a sepia tone image. And obviously these are just 2 filters, but if you can combine more you can see how you can get much more complex effects.

One key aspect of Core Image is that it will concatenate filters whenever possible. The idea behind this is even though we’re applying 2 filters we don’t necessarily want to have an intermediate image in between them. And so by concatenating filters we’re reducing the number of images, and that improves performance.

And of course, as I alluded to earlier, complex graphs of filters are supported. It doesn’t have to be a linear chain. You can have tree-type graphs with multiple input images producing in the end a final output image. And this again allows for much more complicated filter effects, including spacial effects and distortion effects, and so forth. And no matter how complex your graph is, Core Image will work to optimize that graph to give the best possible performance.

So as I mentioned, Core Image includes 125 base filters. So let me just give you a quick demonstration of what some of those filters look like. We have an original image here. We include four different types of geometry filters for doing A fine scales, or rotations of an image. We have a bunch of distortion effects, such as this glass effect that we can apply to an image.

We have a whole bunch of different blur operations. We’ll be talking about that a little bit more later on in the show. We have typically Gaussian blurs, but we also have something called a disblur, which is used for more photographic effect. We have several different types of sharpening an image, several different color adjustments, and color effects to produce different distortions on an image. More fun, stylized filters.

These are some of the filters that you see in Photo Booth. Halftone effects, such as a cmyk halftone effect. Different tilings of images, which will be useful. We also have 7 generators. A generator is a special class of filter that produces an image but doesn’t take an input image.

So for example, this one produces a star burst pattern. We also have nine different filters which are of a class of filters we call transition filters. And transition filters are very useful for video. They take 2 images and a transition time, which will allow you to do wipes between a source and a destination. We have lots of composite operations, all the standard porter-duff composite modes and others. And then we have several reduction type filters. These are filters that allow you to get information about an image and reduce it down. The most typical example of that is a histogram.

So we have histogram filters as well. So that’s our filter set, or a brief tour of it at least. Let me talk to you a bit more about how Core Image optimizes filter graphs, because this is one of the key features that I think Core Image provides to your application.

So Core Image contains a runtime just-in-time Compiler that is designed to optimize your filter graph. The optimization is deferred until the image is actually drawn. And this allows us to only evaluate just the portion of your image as needed to render. That means, for example, if you’re zooming in on a very small portion of your image, Core Image will only execute the portion of the kernel that’s needed to render that sub-wrapped.

Another thing about Core Image and its optimization is it will concatenate and prune the tree to remove any redundant operations in order to improve performance. Once all this is concatenated we have Redo passes over the filter graph to do global optimizations, such as traditional Compiler optimizations such as peephole and sub-expression elimination.

And then another real important aspect is that we perform several optimizations that are unique to the domain of image processing. Let me talk about that, because that’s where we get some of our key benefits. But the great thing is that in Snow Leopard we now have -- because of these optimizations, we have 20% improvement on average in our filters.

This is great news. Some are even more. So let me talk to you about -- now you’ve got an introduction of what Core Image is capable of, let me give you an illustration of how easy it is to add Core Image to your application. So there’s 3 key data types you need to know about when working with Core Image.

The first is a CIImage. The CIImage is an immutable object which represents either a pixilated image that comes from disc or from memory, or it represents the output of a filter, which we’ll talk about next. A CIFilter is a mutable object that represents the recipe to produce the CIImage.

And this CIFilter has input parameters which are based on Key/Value coding, and these input parameters, such as the input image, will affect the output image. The last key data type is the CIContext. The CIContext is a destination where Core Image will render its results. And a CIContext image is based on either an OpenGL context or a Core Graphics context. So to walk you through the process we have five steps for how to add Core Image to your application. The first is to create a CIImage. And there’s 3 easy ways to create a CIImage.

We can create a CIImage from a CGImageRef, we can create it from a URL that references a file, or we can reference it from data that your application has produced using some other means. Next step is to create a CIFilter. CIFilters are represented or instantiated in Core Image by name. So typical code to create a CIFilter would be to call CIFilter, filter with the name Sepia Tone. Or CIFilter, filter with name Hue Adjust.” If you add your own custom filters in your application, these too can be created by name.

So for example, if you have a custom filter you can reference that by name. So we have 125 of these filters. In order to know which one you want to put in your code you may want to discover which ones we have available in more detail. So we have a utility that’s part of the developer install package which is a Dashboard widget, which allows you to explore and see the parameters and all the filters that we include.

So in order to get this, you go to your Dashboard, bring up the Core Image filter browser, and it brings up this blue pane which allows you to scroll through all the filters we install, search through them, see which parameters they take as inputs. There’s another kind of secret feature of this browser is it supports copy and paste. Which allows you to paste -- once you find the filter you want, paste the template for it into your application.

Once we have the CIFilter, the next thing we need to do is set the input parameters on it. So filter parameters are set via Key/Value coding. One of the most common input parameters on a filter is the input image. You set that by calling filter, set value forKey and the value is the image we want to set as the input, and the key that we pass in is KCI InputImageKey.

Filters may have additional parameters. For example, the sepia tone filter has a parameter to set the intensity of the amount of sepia tone. So that’s set again via Key/Value coding. This time the value we pass in is an NSNumber, and the key that we specify is KCI InputIntensityKey.

So now we have an image, and its filter. Next thing we need is context. We can create a Cox text from a CG using context from a CGContext. Or we can create it from an OpenGL context, using create with OpenGL context. There’s a convenience method for Cocoa programmers, which is that if you have an NSView custom drawing proc, within that draw proc you can call AppKit to get the current CIContext for that view.

One important thing that you might want to consider is that if you’re using a GL -- I’m sorry, a Core Graphics base context, to get the best performance out of that context you want to have your application opt into QuartzGL acceleration. And you can do that by adding the QuartzGL Enable key to your applications info P-list. This greatly improves of the performance of Core Image because it allows Core Image in conjunction with Core Graphics to be fully accelerated on the GPU. It avoids the read back from the graphics card into Core Graphics backing store.

So this is a great way to get optimal performance when using a CGContext. Next and final step is to draw the result. So first thing we want to do is we have our last filter. We want to get the output image of that last filter. And we do that by again using Key/Value accessors.

In this case, we access the value KCI output image key. And then we tell the context to draw that image, which is a very simple call to tell your context to draw the image at a point in the wrap. So a common question we often get is “Well, what if I want the results of a filter for other purposes?” And there’s 2 calls you may want to consider if that’s what your application needs.

One is once you have a context you can create a CGImage from it, and another call - with CGImage you can then call image I/O to write the image to disc. So that’s useful. Another option is you can call -- you can ask the context to render to a bitmap, and that allows you to get the raw bits that came out of the filter. So just to summarize those steps, we can put everything we talked about on these preceding slides into one screen here. And this screen shows us how little code it takes to do an image processing effect using Core Image.

Again, what we’re doing here is we’re creating a CIImage from a URL. We’re going to apply 2 filters in this example. We’re going to first apply the sepia tone filter, and then we’ll get the output value from that filter, and then we pass the output image from that filter to our next filter, which is the hue adjustment filter. And we get the output image from that. And then we tell the context to draw this. So what we see here is all the code that’s needed to do the example I showed a little earlier, which is the concatenation of 2 filters.

So as I mentioned before, Core Image is built based upon other technologies in the system, such as OpenGL and OpenCL. I just want to give you an idea of all the work that we’re doing behind the scenes which is -- this is the resulting OpenGL calls that will result from concatenating these filters. There’s lots of stuff that’s going on here, as you saw briefly there was some orange code that went by.

That’s actually the ARB Fragment program that gets passed to OpenGL. We’re also doing stuff in here that’s part of the reason why the code is long that will tile the image into separate tiles in order to deal with limitations of the GPU on the maximum texture size. Similarly, Core Image can leverage Open CL to do its work as well. And again here, we see the Open CL code that’s needed. And quite a bit less code in this case, the kernel is a little bit bigger because we’ve concatenated into one program, the results of these multiple filters.

So now that we’ve seen what’s involved in -- how Core Image works and how you can add it to your application, I want to do a brief demonstration to show you it live. So we have an application on our system called Core Image Funhouse, which is a great way to explore different filters and different parameters.

I’m going to open up an image here. Oops. And we have the image as it was originally displayed. We have a little dialogue here from which we can add new filters. So I’m going to go ahead and add a new filter. I can go to Color Effects and select sepia tone. And here’s an example showing the amount of sepia tone that’s being applied live. I can then apply another filter, I can do, let’s see, go to color controls. I can change saturation.

I can move that. I can add another one. Halftone effects such as the CMYK filter. Again, the great thing here is this is all being done on the GPU, so we can do all of these operations live. Let me make this halftone a little sharper so you can really see what’s going on here.

And let me do one more, just for kicks. So this is a -- one where we can adjust the number of -- do a half, no, kaleidoscope effect on the image. Create some interesting effects. So that’s a brief demonstration of the filters that we can apply in Core Image Funhouse. Let me quit this.

Another application we’re going to be showing you a lot of today is Quartz Composer, which is a great development environment for experimenting with Core Image and other graphics technologies that Apple has. So again, here we have a Quartz Composer document, and all the CIFilters that are available to your application are also available to Quartz Composer.

So again, we can take an image here. Let me -- to get started in Quartz Composer, all you have to do is drag an image from disc into Quartz Composer. And if all we wanted to do was display that image, we’d just drag it to the billboard and we can see the effect of that image here.

If we want to apply a filter, we can just go over here and type in sepia. And it will find the sepia tone filter. And again, as we did in code, we want to take the image that we get from disc and set it to be the input image.

And take the output image from the sepia tone and apply it to the billboard. And now we can see the sepia tone effect. And then I can also go and change the intensity parameters as well. And of course if we want to apply another one -- color adjustments.I’m not getting hue adjustment.

And I can set the input image and then the output image goes to the billboard. And now I can adjust the input angle on that to give different effects on my sepia tone. So that’s a brief example of the filters and how you can set them up in code and in Quartz Composer. Let me quit that. So next, for the heart of the presentation today I’m going to pass the stage over to Frank who will be talking about how you can create your own custom CIFilters. Thank you.

Thank you David.

Good morning. As David pointed out, the crowd is awake even after the party last night. That’s quite amazing. So thank you all for coming. My name is Frank Doepke and I will be talking a little bit more how to write filters. So if you want to write a CIFilter, the first question is why would you do it.

You saw that we have a base of 125 filters. But the filter that you want might not be there. So therefore it’s a good reason to write a filter. The other parts where you can actually combine filters to create a very compound problem, but it might be better to write your own filter for it. And I will give you an example of that in a few minutes as well. And last but not least, of course you can sell those as a business opportunity for you, and I know that you’re all here to make money.

So what is involved if you want to write a filter. First of all, you write a CIKernel, and -- for more of -- this is a Core of your filters. And as we saw, we saw this Open CL code and the OpenGL code. You don’t have to worry about this.

You just write it in one language and we will do the work on transposing this over into OpenGL or CL. And then you wrap it into an Objective-C code. That all together forms the new CL Floater or Image Unit, if you want to make this a plug-in. Where do you find documentation about this?

We have plenty of documentation on our WWDC web site. First and foremost, image tutorial. I’ll actually use some examples out of that one, so if you want to read along you can. We also have the CR reference guide and the Core Image Kernel Language Reference guide to give you more details.

The tools that you will be using, as Dave already pointed out, we have Quartz Composer, which is good for rapid prototyping, I will show you how to actually write a photo in Quartz Composer. And then of course Xcode is the tool of the trade for really writing the image unit of the CI Photo filter.

So nothing really special that you need to write a CF photo. What does CF photo look like? So we first have what you will call a low-level part, and that is the CIKernel, which is very much like C, and actually very much based on GL shader language. And this is where you really have your imaging algorithm.

So this is where you do your hard work. Now for those of you who have done projects image processing in the past and written this in, like, C code, this is a little bit different in the sense that here we produce exactly one output pixel. So your kernel will actually get called for each output pixel it has to produce. You don’t do any for loops, X Y and go through the image.

So this is a little bit of different thinking about this approach. But what you can do, of course, you have one out put pixel, but you can have many input pixels. You can read from more than one image and multiple places in the same image. Then we have the high level part, and that what is written in Objective-C. Here you set up your samplers. And your samplers are your access to the images. So why do we have this? It helps us to do like, a ready transformation and work for you, so you don’t have to do a lot of this stuff.

we load CIKernel object, which is basically a text -- just describes the kernel. We set up all the parameters that we want to pass to our kernel, and those can be the input parameters as we’ve seen for the filter, or parameters that you need to define, like some constant, that have to go to your kernel. Then we set up the Domain of Definition. And also the ROI. These 2 terms we’ll repeat a couple of times in the slides.

So the DOD and the ROI for the output images, the DOD and ROI on the input side are actually 2 very important terminologies that we need to deal with. So let’s look at actually how this will work. So we have a source image and a destination image. And we want to compute a pixel. So the kernel gets invoked to basically create the pixel at the position X and Y. And here we have a very simple kernel.

All it does is it samples a pixel. It does its algorithm, and returns a pixel. This is that simple. This is of course a very simple filter, we will look at more detailed ones now. But back to this DOD and ROI stuff. So the DOD is what will the photo render.

This is its output rectangle. And everything that is not busy inside this rectangle is outside, will be, or has to be, transparent black. The Region of Interest, this is not the input side, is everything that the filter needs to render. So all the source data. This is again the rectangle that encloses all the pixels that the examples will access. So let me give you an example here.

Let’s say we have a filter in which we want to sample around a given pixel. So we have -- this is our center. And we need a 2 pixel radius around it of pixels that we need to do our computation. If you look at this in an image, this looks like the following.

We have to then produce a rectangle. But since we need the pixels outside of it, we actually inset the rectangle to the outside by 2 pixels. So why is this ROI stuff so important and why am I going back to this over and over again now. Core Image, as has already been pointed out, will tile the images for you. When you deal with large images, even not enormously large ones, you will actually have to deal with uploading these images to the graphics hub, not always the most efficient way of doing it on a whole.

It’s better to work on it in little tiles. But to do this tiling we need to know which pixel does the filter or the kernel actually need to access to produce the given output. So if you have an image here, and we want a rectangle, a sub-rect is the part that we want to render.

So this could be either you say well, I just want to draw this sub rectangle in the draw method, or this could be, you know, the tile that we actually need to produce. All the pixels outside the sub rectangle would not be accessible to your kernel. So you could not use those for your computable.

But when you define an ROI function you actually expand it by the amount of pixels you need to the outside, and now the tile will be uploaded accordingly that you have access to those pixels. So after all this theory let’s start relooking at how we write a filter. And after that, if you borrow from the image in the tutorial, the pixel example, which actually starts just grading from an image little round blocks that look like this. It’s a very simple photo to follow along. First step, let’s look at the kernel.

This is where we believe the hard core part lives. We have 2 parameters that we pass to the kernel. It’s a sampler, which is HML source image, and the radius, which is the radius of our dot that we want to produce. We have a few local variables. You see, already we are using vectors types because we are a vector-based language, in which we store the position and do all of our maths in. Then we get now the destination pixel that we currently are supposed to produce.

Now we calculate the center to our block that we want to create. Need to get the distance from it. And now we actually read from our source image the pixel. Now we create an an anti-alias so it doesn’t look pixilated. And return as you see already enrolled in the alpha part is a premodified alpha pixel.

A very simple kernel, in this case. So since we saw in some of the language, let’s talk a little bit more, what is in the CIKernel language. This is not all of it. You find all the details in our CIKernel language guide. But just the most important ones so you can follow along easier. So first we saw we have a sampler.

So this is now on the kernel side, the equivalent to the CISampler that we saw on the Objective-C side. This gives you the access to the image. And it does already do the coordinate transformations for you, if the image is rotated or scaled, all this kind of work is happening there. We have a qualifier for the sampler, and that’s called table, underscore table.

This is important when you want to produce data, and we’ll see an example of that later. Other types that we have is float, and vec2, vec3, vec4. On the input side they would require NSNumbers. But these actually -- or CIVector for the vector types. That’s how you would normally post in numeric data. Then we have NS [Inaudible] color which is a special vec4 type, which is the equivalent to a, a CIColor object, so that’s probably color-managed when you actually parse it in.

I should point out that all -- what we assume in the kernels normally would be that we get our RGBS premultiplied. And we already saw as a function destCoord. This tells us actually what is the pixel in its working space that we currently want to render. SamplerCoord is on the sampler, where we are currently in the sampler, basically this is in the sampler’s working space.

Sample is how we access the pixel. And then we have some convenience functions for unpremultiply and premultiply. So if you need to for some of the maths you need to go from premultiplied data, unpremultiply it, and go back into premultiply. Now what comes probably to mind is okay, well, if I do this constantly in every filter this might not be very efficient. Well, as David pointed out already, we do have very good optimizers.

If we recognize that the filter does unpremultiply, premultiply, doesn’t do anything between with it, we actually lift those out so that you actually optimize our filter tree. So one of the functions that we’ve seen already earlier is smoothstep, which comes from the GL Shading Language, which allows us to give, like, a transition between 2 values.

We have trigonometric functions, like sign, cosign, tangent. And compare operators that we have. The one part that is important to know is that we don’t have “for” loops or “if” statements. Very much like the GL shader language in that sense. Loops have to be unrolled because we don’t have flow controls. This allows us to create this high performance vectorized code that we can easily transport.

So now after the kernel we need to look at the higher level pieces that we have to look -- right. So the first function, again the ROI part. It gets called by CI when we need to render. And the first part of the puzzle is the sampler, because there are going to be more than one. The destination rectangle.

So this is the rectangle that’s the CI tells you know, this is what I want to draw. And then you can parse in some of your own data. So what we simply do in this case is we simply sample around in a radius, we simply inset our rectangle by the negative radius, which effectively means outset it. One other method that you need to provide is the custom attributes.

Custom attributes means we tell the outside world a little bit about our filter. In this case, about the parameters. Normally the numeric range that you want to use or other details. So this allows applications as we saw, like, in Funhouse, to actually create an UI for your filter to create sliders that actually have the proper range for it. And then of course the output image method.

So this is actually what really gets called when we get the image out of the filter. So what’s in the output image method? The first step is that we set up the sampler. And based on our input image, we created something. Then we set up of the ROI function for our kernel. And now we apply our kernel.

And you see that we pass in the sampler and we pass in the radius. We need to create the [Inaudible] of definition so that we actually know how big the image will be in the end. And then we parse in as a user, simply the radius, because that’s what we need in our ROI function to calculate the ROI. So after doing this for a couple of years you run into these little things that other developers will definitely run into as well. What can go wrong when you write a filter.

The first thing -- mine might not be obvious, but if you have a numeral value that you need to pass to a kernel, you need to wrap it into an object, it has to be an NSNumber. So you can’t just say, okay, I want to pass in 0.5, I need to create an NSNumber for it. Because we need to archive this, set them up, especially when we want to run the kernel. Colors have to be CIColors. You can’t use a CGColor or an NSColor.

You have to transform them over into a CIColor first. And then if you need to pass data, data means like, for instance a color lookup table or an environment map, any kind of other, like, 2 dimensional, 3 dimensional data, you need to pass them in. There are 2 things that you need to make sure of.

First, that you don’t color manage. As we said, we don’t normally color match images. But if you say the color match -- the color space to nil will not do any color matching. Therefore not touch your data and transform it. And as I mentioned, you want to stick the table, table qualifier on the sampler so that we actually don’t do the -- go on with the transformation. And last but not least, again, keep in mind that usually what is expected between filters is that the data has to be premultiplied.

So before I actually give you a little example of how this really looks like, I thought I’d make it also a little bit more interesting. So we take the pixelate filter and actually give it a little 3 dimensional feel to it. So it looks like little LEDs. So we add some shine to it -- we do this by using a shine map. Which means now we have 2 samplers that we want to use. And that has some interesting applications on the ROI.

So let’s look at the demo now. So first of all, if we’re on Xcode and we want to create a filter we have a template actually for you, that does already the major work. To create a new project, and then if you look in the system plug-ins, we have right here our image unit plug-in. I could give it a name. So this would be My Photo.

I’ll just put this on the desktop right now. And it already sets up the project, and you see, we have, like, also some code for loading the plug-in, which you normally won’t have to touch unless you do something very special. We have the core of the filter, we have the kernel, everything is right there. So -- but as in every baking show, I already prebaked our pixelate a little bit for you, so you don’t need to see me typing here for half an hour.

So what did I do to make this photo happening? First of all, there’s one thing that we haven’t looked at in the slides so far but that is important, and that’s in the description P-list. This is also in our documentation, but just to give you a quick overview, we need to define actually what our filter is.

And you see I called it actually in this case www.pixilate. helps us with the loading part. And when I set up actually is it’s a stylized filter, in this case,, so that I know when I loaded it in which category it will show up. Then we have our kernel, which we’ve seen on our slide earlier.

Nothing surprising here. This is simply our kernel language. And now we look at the filter itself. The first part is already set up by actually the image in the template. That’s actually where we load the whole kernel. As I mentioned, we need to do that in the Objective-C code. Usually, you won’t have to touch that a whole lot unless you have multiple kernels. But it’s pretty much straight forward. Then I’ll go to our friend the RI function that we’ve seen. Our custom attributes. And last but not least, our output image message.

So let me build this. And now I need to install this. And actually, filters usually get installed in library, graphics, that’s a photo image unit, so I put it on the desktop just to make it a little bit easier for me. And all I have to do is drop my plug-in in there.

So let me now open an application, and just to show it’s not an Apple application to use it, I actually chose a third party customer who’s using it, Pixel Meter. Let me open an image that we have for this. Gives you an appetite for lunch, doesn’t it? And now I can simply go into the filter menu and -- hah, I don’t see my filter.

As I said, air compressor problems.

Let’s try this again. Ah, there it is. And now I can dial in the radius and we have our pixilate. That simple. Okay, now as I said, we want to take it up a notch. So again, so you don’t have to watch me type for half an hour, I have some snapshots.

So now we added the shine map. So what’s now new with the shine map part. Let’s go into our kernel. And we see that we have now another sampler. And we actually pass in the sampler, it’s a highlight size of them, because I need to know this. And in addition, we just have this part in the middle in which we grab from the highlight image and map in on top of the existing pixel the highlight part of it.

What changes do we have on our filter itself? Well, first of all we need a highlight image that’s in our class. And notice that I don’t call it input highlight image. The reason why I chose this is actually this highlight image, I just want to have it pre-canned -- I don’t want to use it to set it. I would call it “input highlight image,”-- it’s actually an input key that somebody could settle on the filter. Just to make the demo a little bit easier, I thought, well, we have a it pre-canned, the highlight image is part of our plug-in, and you don’t set it. So we just preload it at one point.

Which is what we are doing right then, the inner phase. So besides that, all we are setting up is we calculate the highlights, put this into a CIVector, add now both samplers to our kernel. And that should be it. So let me clean out everything, just to make sure I rebuilt the right stuff. Okay. Replace our old plug-in.

And launch Pixilator again. And now we have our shine map on it. Looks good, doesn’t it?

Something broke. This is not the air compressor, this was actually on purpose. You remember I mentioned something about the ROI? So what went wrong? If you look at our ROI function, I did not take into account actually which sampler I’m really looking at. I have that prepared as well.

Oops. What are our ROI function here should have looked like is that we actually now take into account which sampler we have to produce the output rectangle for. The reason why the effect worked first is Pixilator is actually smart. If you just want to have a preview of the photo it uses a lower res image.

But the moment I actually said okay, it now applies it to the high res image, and that’s a much bigger image, and we tile it. So the rectangle simply for my shine map for the other tiles was not correct. And this is what I simply fixed in here by looking at the correct sampler. So let’s try this. We clean our build.

Build it again. Copy. As before, okay. And voila. It works.

Thank you. So this was a very first simple example of writing a filter. Now I mentioned that you can achieve some effects by stacking some filters, but you can also achieve it with a custom kernel. So an example I chose to the chroma so whoever worked with video news, probably like a green screen.

This is also like how your weatherman appears in front of the map. So you could do this by just taking a given CIFilter, and by using a color cube and then inside that color cube actually map out those values that actually match that background. Or you write your own kernel, which gives you a little bit better way of tweaking the fuzziness around the edges.

Now to do this, normally you go into what we call the YCbr color space, which takes the luminescence out and just works on the chroma part of it. So how does this look like as you see as an example on the right-hand side. Those 2 flags where we actually mapped out the sky and put in a new one. So we first take our color, this is the chroma color that we want to extract out.

Convert it into YCbCr, and then do the same on the soft pixel. And now we simply create an alpha value based on the difference and how far we basically are off on chromaticity. And then return it by simply mapping our background image, landing it in with the alpha value on top of it.

The one thing that you might very quickly see in slide this is, okay, why do I actually do this for the mast color inside the kernel. I’m just doing this here so it’s a little bit easier to follow. Normally, this would be something, if it’s constant for every pixel, you want to normally lift that part out and actually put it into Objective-C code.

This is like a little bit of performance thing that you can do here. So and now we go back to the demo so that we can actually see this in real life. Okay, so this is now done in Quartz Composer. As I mentioned, this is a great prototyping tool. And now I have to just hook up my videos, actually, for it. So we have the source video that I set up on my -- oops. Do this correctly. Okay. We already seen our flag.

And actually have the second video that I need as well. Oops, be in the right position. Okay, so you see a few things here in the composition that are probably not - that are obvious in the first place, but don’t bother -- the critical parts. And this is actually our new filter that I wrote purely in Quartz Composer. Nothing in Xcode.

So the nice part is if you look at the patch inspector you can go into the settings and here’s our kernel. And it looks very much like what we had on the slides, except one line. We’ll come to that one later. But you can now really type in realtime C and change your filter.

So that’s why it’s so great for rapid prototyping. Okay, let’s make this work. So first we need to pick our chroma color. Let’s use our color picker. And voila. Simply maps out the color. Now, you see that I have some threshold values, so I can actually now dial in how much of the glow -- how much of the background I want to remove, and smooth this out to have a nice smooth edge.

And now this checkerboard pattern doesn’t really look particularly interesting so we’ll actually take some video here, put this in the background. And of course I can play this whole thing. And there we go. On video we have now chroma king. See this little bit of flickers on the bottom parts, and now I can change my smoothing or the threshold until this is all gone and kind of looks correct. Okay, that is the simple part of the chroma key.

But now there was something that I wanted to point out. Let me actually pass my playback here, and actually use a different background image. What you see here that may actually highlight this a little bit more is pixels. You see there’s this fringing in where we still have the original color. And if you go to your weatherman often enough, you’ll see that they have this halo around them. It’s not because they’re God-like creatures, it’s just simply an effect from the chroma work. Actually, let me keep fixing it here on top.

So how can we get rid of this. And that’s where this magic line comes into play. Now it’s mostly gone. And this is, I mean, just simple approximization. You can find SIGGRAPH papers on how to do chroma key and some -- but I wanted to keep it simple. What I am simply doing, and I can now of course play again the video. And you see there’s much less of a halo effect as there was before. This is with it, and without. The trick that we are doing is quite simple.

If we look at our code, what I’m doing here is simply when my magic value is set, I actually subtract the chroma inside the blend area, subtract the chroma off the chroma mass color out from my source color, and then composite on top. So therefore the blue halo that we saw is simply taken out. It’s that simple.

Gives you a little bit better effect so this is how you can like easily play around in your photos, and step by step improve them until you have a good result. And you don’t always have build, compile, launch any other application. So Quartz Composer is a great prototype of all of this.

Okay, let’s go back to the slides. So far we’ve now talked about very simple filters. So what is happening if your problem is a little bit more complex. And that’s what I call the salami strategy. And that way that you actually have to slice your problem. You don’t eat the whole salami, cut little slices of it, bitesize now.

Important thing to know is that a filter can use more than one kernel. So far we’ve only seen one kernel in our simple photos. But you can have more than one. And you can also use other CIFilters inside your one filter to create, for instance, like the compositing step. One of the examples that we actually do have in our image in the tutorial is the lens filter, which kind of creates a detective lens. Where we want to have a little shine map around for the ring, need to do some distortion for the filter effect.

And need to composite it, like with a background image. So we have one kernel that creates the shine and scaling part of it. We have a second kernel that we wrote that actually programatically creates this little metal ring. And now simply we use the CN source over compositing filter. You might think, well, we could probably put this all on one kernel, but there are of course limitations on how much you can really put on one kernel so that it runs on all the hardware.

So sometimes you need to slice your kernels to make them slightly smaller. And all this combines into one filter. So what your client sees is simply one filter, although underneath it’s built out of multiple pieces. So this was pretty much all to give you an introduction how to write filters, and with that I’d like to ask my colleague Alex on stage to show you some miniaturized samples. Thank you.

Thank you Frank. Okay. Good morning. My name is Alexandre Naaman. This morning I’m going to be showing you how to simulate the effect of a tilt and shift lens, which is a special kind of lens which isn’t exactly cheap. And I’m going to show you how to do that inside of Quartz Composer. And we’re going to go through all the kernel code as well that we’ll be creating. We’ll be using one standard Core Image filter and one custom filter that we’ll be creating.

And the result of this kind of lens, which is quite expensive, you can do simply with just 2 filters. So let’s take a look at what a standard image looks like here with a regular lens, which you could take with your point and shoot or 35 millimeter SLR. This is an image taken at Stanford from the Hoover tower. And let’s see what this looks like when we’ve taken this same image using a tilt and shift lens.

So here we go. You can see that we have 3 sections here. We’ve got a blurry section at the top of the image, we’ve got a sharp section in the middle, and then another blurry section at the bottom. The result of that is that we end up with what looks like a miniature. And that’s due to the reduced depth of field.

So we can simulate that within CI. And we’re going to be doing that by using a relatively well known technique that’s become quite popular as of late called fake miniatures. For our first pass at this algorithm we’ll be using 2 images. One base image, which will be our sharp image, which was taken with our regular camera with a regular lens. And then a blurred version of that image, which we’ll be using a standard CIFilter use, um, filter.

So if we look at that through assembly, we’ve got our output image which is going to be some amount of our base image and some amount of our blurred image. And we’re going to use a parameter, T, to determine how much of each we’ll be using. In terms of CIKernel language, we’ll be using the function mix, which visually looks like this. So we’ve got for each resulting pixel image, we’ll be using some amount of our sharp image, what we call base image pixel here. Some amount of the blurred image.

And then based on a value, T, which we’ll determine based on 2 points that we’ll let the user define inside the image, how much of each we should be using in the transition area. So if we look here, we see we’ve got white, which corresponds to where we’ll be using exclusively the blurred version of the image. We’ve got a black area, which is sharp. We’ll be using exclusively the initial image. And then a gray region where we’ll be using some amount of the original image and some amount of the blurred image. And we get this as a result.

So now how well does this work. Let’s take a look once again at an actual picture taken with a tilt and shift lens. And what we end up with when we use our filter. So now let’s go -- we’re going to be doing this inside of Quartz Composer. So this is a simplified version of what the composition looks like.

So it’s quite simple. We’ve got our input image, the standard CIFilter, CIDiscBlur, and we’ll put that into our custom kernel, which we’re going to go into that in detail in a few moments, to create a final output image. In actuality , we need to do a little bit more work because the output image from the blur is going to be slightly larger than what it was when it went into it.

So we’re going to have to do a little cropping. So this is the actual Quartz Composer composition that we’re going to be using. So let’s take a look at the parameter T, which is going to help us determine which part of which image we’ll be using to create our final output image.

So we’ve got 3 sections in this curve. We’ve got A, which is a nobler distance, which will be a user definable parameter, marked here as A. We’ve got B, which is our transition area, marked as T W. And that’s just simply a line equation which has been effectively clamped at 0 1, which is equal to X minus Nobler Distance over TW. And then a region C, we’ll be using exclusively the blurred image. And if we look, you can see the gray, black, and white areas marked on the right, in the gray scale image.

So let’s look in detail at the kernel code. First off, we have the convenience function, which is going to help us determine for any point that we’re currently attempting to render, what the distance is from that point to the line that we’ll be creating via the user defined points 0 and 1.

So we can have the section that’s sharp be anywhere in the image, just by defining 2 points. Now let’s look at the guts of the kernel. We’ve got -- first we’ve got our input image. This is our sampler, the sharp image. We’ve got our blurry image. 2 user defined points. The nobler distance. The transition width.

And we can infer the final dimension of where we will be using exclusively the blurry image from those 2. Now let’s look at internals. We’ve got color 0, which is going to be the current fragment we’re evaluating from the sharp image. And that will get the same thing for the blurred image, and we’ll put that into column one. We’re going to compute the distance from the point we’re currently trying to calculate to those 2 points defined -- to that line defined by those 2 points, p0 and p1.

From that, we’ll look at our line -- we’ll use our line equation we saw earlier, which is clamped, right there. Value T. And then finally, we’ll be using that value T to combine color 0 and color 1, touching only of the RGB components and leaving alpha alone so we keep -- we don’t have any additional transparency changes.

And that’s it. So that works quite well, but we can do better. In actuality , what we want to do is take into account the effect that as you get further away from the center the blur is going to be larger. So we can do that by using 3 blur radiuses.

And for that let’s pretend that our first blur is going to be a blur radius of 0. So this is our initial input image and how much we’ll be using of that. So that’s curve A. We’ll be using a medium blur, which will have -- which will give us an effect similar to what we had with the first, which was constant in our previous example. And we’ll be using a large blur. So as you get further away the contribution of curve C, which is the larger blur image, will be larger.

And so we’ll have more blur as we get further away from that line. And if we look at the contributions of B and C, we get something very similar to what we had earlier, which looks like our linear wrap so you can see that we’ll have something that looks quite close, just a little bit better.

If we take a look at the kernel code you can see that we’ve gotten rid of our 2 parameters here, and now we just have our 2 samplers, which are a medium blur and a large blur, and we still have our -- we retain our initial input image as well. Now we need to compute the weights to determine how much of each one of those images we’ll be using.

And we do that here using the smoothstep functions, and then finally compute the alpha color, once again leaving alpha alone. Let’s take a look -- we can do that, we can now -- we can add little bit more fakeness, if you will, to our image by enhancing the sharp section by increasing the saturation of our input image and the sharpness.

And this will really make it stand out and give it more of a model-like effect. So this is the resulting Quartz Composer composition. We have our medium blur and our large blur here enclosed inside of macro patches just to do the crop to the initial -- to the -- to match the initial size we saw earlier.

And then we get our new kernel and the final output image. How does this look? Let’s take a look. This is our actual tilt and shift lens image. The next thing I’m going to show you here is what this looks like when we apply the 3 images. And then finally, with the additional saturation and sharpness.

So, some notes of interest. In order to do this, you’re going to have to use CIDiscBlur not GaussianBlur, because CIDiscBlur more closely approximates what you get, the aperture from a lens. And if I were you, I’d save my money and not buy a tilt and shift lens and just get a faster GPU instead. So on that note I’m going to switch over to the demo machine and give you a quick sample of what this looks like.

On video. So there we go. And we can change -- in this case I’m just changing the Y position of what’s going to be sharp. And I can also change how much will be sharp. Makes people look small.

And on that note I’m going to pass it back over to David.

Thank you very much for your time.

Thank you, Alex, thank you Frank for those great examples. To conclude our session today I just want to give a few more references that you might want to read if you’re interested in image processing. There’s some good books to look at.

First of all, there’s a great book on image-free sampling, just a good text called Digital Image Warping. That was also the latest version of the GPU Gems book, Number 3, has a sample that shows how to use Core Image to track an object of a specific color and video at realtime. So that’s another great sample. And I encourage everyone to take a look at that.

The sample code for that is also available on our ADC web site. Final words, also, every once in a while things don’t go quite as planned, so I want to get a few slides that talk about -- or bullet points that talk about where to go if you have problems. First of all, there’s a resource, an application called QuartzDebug, which is a great application.

Among many other things, it allows you to see the Core Image filters as they’re being applied in your application, and it also gives performance data. So this is a great way to see if your filter kernel is getting applied as you expected, as your application runs. Also, if things don’t render as expected be sure to check the console logs.

There a couple of messages you want to look out for, there’s message that says ROI not tileable, which may indicate a problem in your ROI function. Also, if you have any syntax errors in your kernel they will show up in console as well. Lastly, a few other bits of advice.

Make sure you test your ROI functions with large images. As Frank showed, once a large image is passed to your filter it can expose bugs in your ROI function, so be sure to try that. It’s also a good idea to fully test your filter to pass in an image that’s been rotated, so that -- or scaled -- so that you can flush out any bugs you may have in your sampler transform.

The reason for this is that, as I’m not sure I mentioned earlier, one of the optimizations that CI will do is if you have a filter and a scale operation, depending on what the filter is, it will try to move the scale earlier into the pipeline so that the filter is being executed on fewer pixels. So it’s important to test your kernel to make sure it works in that scenario correctly.

Another bit of advice, Core Image works on all supported hardware, and some of the hardware we support doesn’t support full floats. In this situation what Core Image uses is a fix point math where 16 bit quantities are used to represent quantities in the logical range, -1 to +3.

So if you have data that you want to operate on that’s outside that range, you might want to scale it to the minus 1 to 3 range, if you want to make sure that your kernel will work as expected on the older hardware. Again, all the modern GPUs support float or half float, so this is not as much of an issue in those scenarios. There’s a lot of other great examples to look at.

If you go into the developer examples folders there’s a Quartz Core Image folder with some examples. I think -- increasingly, we’re moving some of these toward the ADC web site, so you may want to check there as well. There’s also a whole bunch of Quartz Composer examples that give great illustrations of how to use Core Image. So I like to point people in that direction as well. In the compositions, Core Image and Core Image filters you can see all sorts of clever effects that have been done in Quartz Composer.

Some other titbits of advice. If you’re working in Quartz Composer, it’s a great tool to take advantage of your eyesight camera-- you can apply your filters in realtime on the camera for results. Again, this further mentions -- another reason to mention the example I mentioned earlier which is the color tracking, which works on video. So -- and by all means, play around with our filters.

Use Funhouse or your own application to combine filters and see what new effects you can create by combining filters in unusual ways or by writing your own filters. And as I mentioned a little bit earlier, Core Image is not just for images and videos, it can also be used on UI elements, you can use it in conjunction with Core Animation, and it’s great for user interface effects. So that concludes our talk today, if you have more information you’d like, please contact Allan. 30