Okay, so thanks for coming. This is the Core Animation Techniques talk and I'm John Hopper and I work on Core Animation amongst a few other things. But today I'm going to be talking about Core Animation. And so let's get right into it and tell you what we're going to be talking about. First of all, we're going to kind of recap some things that hopefully you already know. You know, what is it? What do you use it for? Why do you care? And then Then we're going to go through some kind of high level descriptions of how the CAE model in terms of compositing and rendering and timing and all these kind of things work in general terms. And then we're going to get into some kind of nitty-gritty kind of coding sample, example type things. And finally we're going to wrap it up with just some few slides about what actually has been added in Snow Leopard and then some demos throughout to kind of tie it together.

So hopefully by now you've heard what Core Animation is. It's basically a compositing system with animation capabilities. And the reason we added this or implemented it in the first place is because this is kind of a hard problem to solve. We've done this in the past with kind of FrontRow and these types of things. And we've always found that if you take the easy coding route, you end up with something that's slow. the hard cutting route, well sorry, you take the open GL route, then it ends up taking a hell of a lot of time and you have to tailor to every single graphics card, but in the end it's pretty fast. So we try to solve that problem and basically take the open GL route for you, but then expose the API to let you use the easy method to get everything working. So to that end, our API is very declarative, it's property based, Objective C objects.

As I said, we try to hide all the GPU details, you don't have to care whether it's an ATI video card, we have to care about that. And then obviously we try to integrate as well as possible with the various view hierarchies that you already know and love, whether that be NSView on the Mac or UIView on the iPhone. And both of those can or are hosted entirely on top of Core Animation.

So the guts of the API then is really this kind of layers and animations. And these are basically the main Objective-C objects you're going to be dealing with. So that's what we're going to start talking about in a minute. But just one final thought is that another way of looking at CA is that it's kind of one of the few OS X technologies that really lets you kind of easily combine all the different kind of graphics frameworks that we have available. Whether that be Quartz 2D or Quartz Composer or Core Image, OpenGL, QuickTime. You name it, you probably have a way to let you display it through a CLR.

Okay, so what is a layer? So a layer is the thing that lets you describe your graphical scene. As we want to be able to have lots of these things, we try to make them fairly lightweight. It has a bunch of different properties, so you can describe things like, you know, where is it on screen?

What does it contain? How long does it play for? How fast? You can arrange them in a hierarchy, so you can have, you know, you typically have a root layer, and then the root layer has sublayers, and then the sublayers have sublayers, and so on and so forth. And this content, it's basically just the bitmap in the final analysis, but there's a lot of places that bitmap can come from. So you can have core graphics or Quartz 2D rendering.

You can have core graphics images. You can have OpenGL rendering. You can have Quartz Composer compositions, QuickTime movies. The list is pretty wide-ranging. And so the kind of the application programming model that we provide you is that you basically get to arrange these little layer objects, you know, to describe exactly what you want to see on screen. And then you kind of hand it off to us. And then we will take over from that point and basically make sure that it's rendered, you know, as often or as seldom as necessary to give you the best output. You know, we care about the refresh rates of the display, how the animations are running, all this kind of stuff. And the idea is we also do that in a background thread so that you really don't have to care, and it never even blocks your application.

Okay, so the second half of the API putting is the animations. And so as I was kind of describing, layers are basically these little bags of properties. So obviously kind of the obvious way to do animations is that you have things that can describe how properties change over time. So that could be something like, you know, the layer has a position property which describes where it is on the screen. And so you may want to say, you know, starting at time T for the next second, I want this thing to move from here to here. So that's the kind of thing we let you wrap up in these animation objects.

The details here are that there's lots, well, not lots, but there's a few different types of animation class. And they all do this kind of same thing, but they let you compute those kind of state changes in slightly different ways. So for example, we have this in code of basic animation, which literally does just let you have a formula to position, which is what you want in kind of a lot of, 80% of cases maybe. Then there's a parallel class called the keyframe animation which lets you describe these kind of state changes as a list of keyframe values. We will interpolate between the keyframes based on how far you are along in the timeline. And then finally there's another type of animation called a transition. Transition is not used that often but it's used when you're unable to animate the properties directly. For example, you may have a list of filters on the last, and it contains maybe a Gaussian Blur filter, and then you change that and you say, "Now I want to have a crystallized filter." And obviously there's no way we can just kind of animate property changes to kind of map from one to the other because they're fundamentally different. But what we can do is we can add a transition, and that basically takes the before and after kind of image representations of what you see on screen, and does some kind of transition effect between them. So these are kind of the things you would see in Keynote, cube transitions and cross fades and dissolves and that kind of thing. Also, animations are not limited to just changing a single property or having a single effect. Just like layers, you can group them hierarchically and that is typically done for timing control. You can create a group and have a group of sub animations and then you can define the timing curves on the group rather than each individual animation. And finally, again, just like rendering, All of these animations run entirely in the background. So once you add an animation to a layer, then you really don't need to care much about it. You can get notifications or delegate callbacks when the animation starts and stops, and you can remove them and you can query them. But other than that, you don't have to do anything every frame. Just like you heard in the session earlier this morning, the whole idea is we wanna free your application from having to care anything about what happens at frame rate.

'Cause we think we can do it faster. Okay, so just briefly, this is the kind of overview of all these animation classes, and you don't really need to absorb anything here, just to see that there are this kind of class hierarchy with different types of animation and different properties. And that's about it. So let's wrap up this section just by giving you a little example of how do you code to this stuff. So our example here is two halves. The first half is going to create a layer and basically attach it to a parent who we hope is on screen. So we're going to create a layer, set its frame, which is its geometry. In this case, we're going to say you have this rectangle, which because we're setting the frame, that is in the parent's coordinate space. And then we're going to give it an image. This is just some CG image we loaded off disk or whatever. And so the final statement there is we're going to add this layer to its parent, to the parent's list of sub layers. It will be appended at the end of this model 1. And so at this point, assuming the parent is visible, then our layer is now on screen.

So that really is what there is to it. Just to make it a little more of an example, then we're going to say, well, now we've added it, we really want it to fade in over the first second. So the way we can do that is by creating one of these animation objects, setting its key path property to be opacity, because we want to animate the opacity property of the layer, and then giving it a formula two value of zero to one. So that's gonna take the opacity from zero to one over the time period of the animation, which you can see the next line is set to one, i.e. one second. And since the default start time for animations is when you add it, then when we add that to the layer in the next statement, what'll happen is that as the layer appears, because we just added it, it'll kind of ramp, its opacity will ramp up from zero to one over a second. Once the animation is finished, it'll be deleted from the layer and everything will just go on as normal. Okay, so now we're going to talk more about the details of how CA works. And the first thing I want to talk about is this idea of having two trees.

And so, you know, we think about the layer tree a lot because it's, you know, it's the object to see data structure. It's the thing you can modify. And so in this case here, we have, you know, several layers and I arrange the hierarchy. And the content, you know, can be different things. It really doesn't matter. But always behind the scenes, we have this thing called the render tree. And the render tree is kind of our lower level representation of this same data structure. You know, every time you make a change in the letter tree at some point, we're going to have to try and do a corresponding change to kind of update the render tree to be the same thing effectively.

And this is the thing we actually render from. And the reason for that is that, you know, as I keep saying, we want to be rendering on a background thread at all times. So we don't want you to be able to lock us out of rendering ever because you know that would be bad and we drop frames So the render thread is really this thing that keeps spinning Keeps looking at this render tree and kind of pulling frames of it and putting it on the screen So then the question then becomes I guess is that you know we have Letry and a render tree and how do we get state changes from one to the other? well we had this idea of transactions. And transactions obviously come from database theory. And ours is kind of a very weak attempt at riding on their coattails or something, because our transactions are really just a way to batch up state changes in such a way that they get copied from one to the other in a single kind of atomic unit. So what this means for you is that every time you're spinning around your event loop, your run loop in Cocoa or whatever, then you're making changes to these layers for every event. And so by default, we will set things up so that every time around that event loop, we will create a single transaction, kind of batch up all the changes. And then when your event loop gets back to the run loop, we can say, oh, we saw they've done kind of some unit of stuff, and we will take that stuff and throw it over the walls of the render tree. So that's kind of what we call an implicit transaction, but that's not really necessary to know that right now. But the thing to take away is that, you know, This doesn't happen automatically. Sorry, I just said it did happen automatically. So the thing to take away is that there are two trees and every now and then you may have to care, although hopefully not very often, that you have to push things from one to the other. And like I said, it does happen automatically most of the time. So next I wanna talk about geometry.

Obviously geometry is very important for layers because geometry is how things are arranged on screen and without this we would see nothing. So every layer has a bounds rectangle And the bounds rectangle, which is the bounds property of the layer, defines kind of the layer's internal coordinate space. What that means is that any drawing you do within the layer or any positioning of sublayers is within that kind of local coordinate space defined by the bounds rectangle.

And you can see in the picture here we have a dotted line or a dotted box, which is trying to represent our superlayer. And we have a smaller graphic, which represents the layer we're talking about. And you can see we've highlighted its width and height. and those are two subparts of the bounds rectangle.

So every layer also has a position, and the position is not defined in the layer's own coordinate space. It's defined in the parent's coordinate space, the super layer's coordinate space. And what that means is that when you, because keep in mind that all the time we're trying to take this little layer and map it into its super layer. I mean, that's basically what we're talking about here. So the layer has a position, and it also has an anchor point, which is the frame of reference for the position. And in this case, we've drawn a crosshair in the middle of this little graphic, which is trying to represent that the anchor point is in the center of the layer. And that basically means that all transform and positioning is happening relative to that anchor point, the center, in this case. So you can see that when we take this thing and map it into its super layer, the first thing we do is align the anchor point in the layer eye, the center, with the position in its super layer.

Next, there's also a transform property on the layer. And what the transform does is once it's been positioned, it gives you a chance to kind of scale and rotate and all those kind of fun 3D things. So you can see that the transform gets applied once it's been positioned. And in this case, we're just going to rotate and scale a little bit. And that's basically the 90% cases of all the geometry stuff you want to care about. Obviously, we've only talked about 2D right now, but, you know, 3D works in just the same way. It's just that, you know, the matrices have an extra column or something.

So now we've talked about how this layer graphic gets mapped from one, from itself, its own coordinate space into its parent. So let's talk a little bit more about, you know, what is in that layer. So there are a number of subcomponents which come together to form the layer image, as I like to call it. And so these are things like the background color property of the layer, the contents, the sublayers, and the border color. In this graphic, they're actually flipped upside down for some reason. But basically they're sover together, sover compositing to give you one image representing all of the things in that layer. So if you have any drawn content and a border, then obviously the border will be layered on top of the drawn content. So at the end of that we have an image, but what do we do with that image? Well, there are a number of other properties which can then modify this thing we're creating. So for example, we can apply a list of core image filters to do things like blurring the contents of the layer or changing its color for some reason. We can also apply a 2D drop shadow to give the standard kind of drop shadow effects. And finally we can apply some kind of opacity fading. And this basically are all the effects that run in the coordinate space of the layer itself. So they're kind of the little things that fit within that little box of the previous diagram.

So the next part of this trip to the compositing model, I guess, is the compositing model itself. And typically what you see on screen is the 90% case again. We have the layer image, which is that little thing we were just talking about. We have the superlayer transform, which is some matrix we're constructing to represent all of that geometry mapping we were talking about. And then we have the backdrop. The backdrop is everything that has been rendered up to this point from its sibling layers. So if you have a parent layer with five sub layers, then we're going to render the first one and the next and the next. And then for each one, the things that have been rendered before constitute its backdrop.

And so you can see that typically we just do source over compositing because again, that's what you want most of the time. But there are a number of ways in which we can modify this diagram. So firstly, you can add a compositing filter, which is just a regular core image filter with two input images. And the idea of this is that you can replace that source over compositing by something more complex.

So you could think-- I think there are core image filters which represent all of the, for example, the PDF blend modes and the PortaDuff blend modes. And you could even invent your own here. So the next thing is we can add a list of background filters. And the background filters run on the backdrop of the layer. So it's kind of under siblings. And this, again, is a list of filters. And they're applied 1 through n. And then the result of that is fed into the compositing filter.

So, so far we've only talked about two of these image inputs, but there's actually a third one. You also have a mask. Every layer has a mask. And it comes from two separate components. The layer has a mask layer, potentially. And it also has this property called masks to bounds. And if masks to bounds is set, then it says there's an implicit mask, which is the shape of the layer.

And if the mask layer is set, then it says take the alpha channel of this layer and treat that as the mask of the layer itself. And so if both of those are set, we'll just multiply them together to get the cumulative mask. And so what do we do with this?

we're going to change our compositing diagram yet again. And we're actually going to add this third kind of operator with this kind of mixing or interpolating operator, which is going to take the result of the layer itself mapped into its parent's coordinate space and the backdrop and the filters potentially composited together. And then it's going to mix between that and the regular backdrop based on where that mask image is going between zero and one. And you may ask, why do we want to make things this complex? And the reason is because it lets you do a lot of interesting stuff. For example, if you want to do a blur under a layer effect, the way you would do that is set the background filters to blur, leave the compositing filter alone, and then set the master balance property on the layer. And so that's going to clip the effect of that Gaussian blur filter on the backdrop just to where the layer has non-zero alpha or some kind of interpolation of that. So as I said to start with, typically you don't really have to care too much about this. But when you do care, when you are starting to play with these properties, it's a really good thing to keep in mind this diagram and know exactly the order in which these things are applied. One final thing, which you don't have to care about that often either, is that if there are image processing transitions running in the layer tree, then they'll be slotted in after we map things into the parent's coordinate space, but before we composite them.

Okay, so moving away from geometry, now we're going to look at timing. Timing is a very important part of core animation because, you know, we're animating, so we have to have some kind of time base. So to that end, every object in our object hierarchy, be that layers and animations, they all have local what we call timing spaces, just like they have local coordinate spaces. And the way we deal with that is when we start to render a frame, we have a global time, which is wall clock time typically. And so just like we map geometry across the layer tree, we also map timing or time. So we'll start at the root and then for every layer we'll map the time from its parent into itself and then into its sub layers. And so this lets you do very interesting things because the timing model is pretty rich.

And here's a list of the properties on the bottom bullet, I guess. So things you can set are things like, you can set the delay, which is the begin time. You can set the duration, which is the length of time the thing is gonna run for. And then you can do some other things like, change the rate at which it progresses, change the number of times it repeats, and some other things. This scary looking block of code is basically the entire timing model. So it's really just here to show you that, although this stuff may seem complex, it's really kind of easy to reason about because typically you only need to look at some little parts of it. So for example, if you wanted to pause a movie or something, some open jail rendering or Quartz Composer, then by looking at this little kind of math, you can see that what you do is set speed to zero on the layer and then change the time offset to kind of push you along the timeline. Obviously, when you set speed to zero because it's multiplying the parent time, it's gonna clamp you to the start time.

The rest of that code is basically just dealing with repeats and playing backwards and forwards. So now we can look at an example of all this. So here we have some kind of object, you know, the round dot. And we have its timeline. And the timeline is going left to right. And you can see in this block there's kind of a diagonal line showing progression of local time. So that's all well and good. We have an object that's going to run for 20 seconds. And so let's add a child. And the child's timing properties are a little different in that it's set to begin two seconds after its parent began.

Its duration is set to 14 seconds, which is gonna make it stop before its parent stops. So adding a third sub layer, also with its own timing properties, you can see now that it has a begin time of 1, and what that means is that it's going to start one second after its parent, which is the blue let, and you can see that's why it's moved in a little bit. But it has the same duration as its parent, which means its end is after its parent has already ended. So what that means is that it's not going to run to completion, it's going to be clipped off when its parent ends, because timing always kind of clips. So finally let's add a third layer, or a fourth layer I should say. And this one has a little more complex set of properties.

Again it has a begin time just to shift it along a little bit more. And this time it has a really small duration, three seconds. So you can see that's just one of those blocks in the diagram. But we've set it to auto-reverse. And auto-reverse means that after you've played fours, always play backwards once. And so on top of that you can see it's also been set to repeat three times, which actually means it's going to play forwards, then backwards, then forwards, then backwards, then forwards, then backwards. And again, it's going to be clipped, but that's just how it goes. So hopefully this gives you some way of reasoning about these kind of timing models. You can see it's pretty much like any nonlinear movie editor in the way we can build up these kind of timelines.

Okay, so again, thinking about time, I wanted to talk about how animations are applied to layers. Now we have this idea of a sandwich model, which, basically comes from SMILE. SMILE is a web W3C animation standard and they kind of first, this is where I first saw it. And the idea is that you have model layers and presentation layers. Now the model layer is the thing you modify in your data structures. That's why it's called the model, it's your data model, at least for the graphics. And the presentation layer is kind of a version of that with all the current animations applied, which may modify its properties so that things that are animating from A to B have intermediate values for that position, for example. And then in between those two, kind of bottom to top of the sandwich, we have all the animations that have been attached to this object. So in this case, you can see we have four and they have various timelines. Again, time is left to right. And so then it's pretty obvious, I guess, how we're gonna evaluate this, but we're gonna take a slice for a particular time and then sample each of these animation objects for that time. And when we sample them, that basically means doing the interpolation and producing that interpolated value which they want to apply into the layer. And obviously, so we take the model layer, and after applying all the animation values, we end up with this presentation layer. That's the thing we're gonna, you know, to send off to render to the screen. One other thing worth mentioning is that there is an ordering to these animations, because often you can have multiple animations for the same property. For example, we have additive animations, which means their effect is not set into the object, it's actually added onto the current value. So ordering is basically done by sorting by begin time. This section is trying to get away from this kind of high level view we've just been talking about and actually go straight to the bottom and talk about lots of little details that will be important to you.

we want to talk about subclassing. So why would you subclass the layer? Typically, you want to subclass because you want to add your own behavior. So two common examples are you may want to add your own drawing code or you may want to add your own layout. And by layout, I mean potentially adding sublayers and positioning them somewhere within your local coordinate space.

So obviously the way we do this is subclass in the normal way, in the normal Objective-C way, and then add properties because if you want to have behavior, you typically want to customize that somehow. So in this case, we're going to add a single floating point property in that interface.

And then in the implementation, we're going to do something slightly different in that we're not going to try and implement it ourselves. We're going to allow Core Animation to do that for us. And so by declaring the property dynamic means tell CA and Objective-C that, you know, you don't have to do this at compile time, but at run time, we will synthesize all the methods or the everything that has to be done to make this property actually exist.

So the next thing you may want to do is look at this default value for key method. Default value for key is a way of providing the initial value of your property. If you don't do this, you'll just get some kind of zero like value or identity like in terms of matrices. But if you want to have it default to something other than a zero, this is the way to do it. Because it's much better than actually setting the value to be its default value, because there is no storage associated with this. So anyway, so you would overwrite the method, check the key that you're giving isn't my property, and if it is, return some value. And then any objects you instantiate from this class, if you actually looked at the property values after being created, they would have whatever default you set.

So the next thing you can look at in subclassing is the will and did change KBO methods. These are a really good way to get notifications when your properties have been changed. So continuing with this line width example, if our line width changes, and presumably we want to use this to draw a bunch of lines using core graphics, then we really need to tell the layer, you need to redraw yourself, because you're out of date now. Again, cool super.

And finally, you know, get to the meat of this, in that the reason we've done everything so far is so we can override the draw method and actually provide our custom drawing code. And so to do that, you will override this draw in context method typically. And so again, with this line width thing, we're gonna take the current value of the property, set it as the CG context line width, and then presumably go off and draw some lines. One thing to bear in mind is that if you don't use the dynamic properties and you do implement them statically using instance variables or something like that, then you should look at the documentation on the knit with layer method because you have to implement that if you wanna be able to use kind of presentation layers and animations in the best possible way. So, let's... So next let's talk a little bit more about animation. So mostly so far we've been talking about animation in terms of objects. Now objects you create and attach to your layers and then they will go off and do something. And that's great and that's an underlying fundamental model but a lot of the time you just want to set a property and make it animate. So by default we have a way of letting you do that and that's called implicit animation. And what that really means is that when you set properties layers, by default the engine will go away and create one of these animation objects for you and then attach it to the layer.

And so typically we use basic animations here which are these from two animating things because that's kind of what we want to do. And the from value is always the current screen value and the two values are obviously the new layer value. And it's worth emphasizing the screen value thing. What this really means is that we're going to take the value of the property at the previous previous transaction and animate from that. If you've already modified it, so say you were to say let position equals 100 to 100 and then the next line, let position equals 200 to 200, you won't get an animation from 100 to 100 to 200 to 200.

You'll get an animation from wherever it used to be to 200 to 200. I guess typically we feel that's really what you want. I mean, you don't want things to jump and then animate from where they are now. You want them to animate from where they were to where they're gonna be.

So finally, one more point here is that we're talking about numeric animations so far, but a lot of the properties, as I mentioned earlier, don't support numeric animation, things like filter lists and what have you. And in these cases, the default will be to create a transition here, which will just cross-fade from one state of the letter to the next.

And that's all great, but every now and then, the implicit animations become kind of annoying. You know, every now and then, you don't want every property to animate, or you want it to animate in a different way. And so when that happens, the best thing is to turn them off and then add your own animations, or don't add anything at all. And there's a number of ways in which you can add things, or sorry, there's a number of ways in which you can replace the existing ones. The simplest is to basically just set this transaction property to say disable actions, these actions are kind of animations. But the other way, if you want a more persistent thing, is you can actually subclass the layer and implement some of these action for key methods, which actually give you a chance to return any animation object for any property change.

Okay, so following on from the animation theme, as I mentioned earlier, one of the things you often wanna do is look at the current state of the layer with its animations already applied to it. If you want to hit test, then you need to know exactly where that thing is now, not where it was before the animation started. So what the presentation layer does, the presentation tree is it gives you a way to get access to these objects. So typically what you would do is you would get a layer and say I want to know its current state, so you'll then call it PresentationLayerMethod. And what that does is it gives you a copy of the layer back. And the copy has already had all the animations applied to it for the current frame, the current thing that's probably on the screen. So then you can go in and look at it and say, if I know I have an opacity animation, then I want to look at the opacity property to see what is the current value on screen.

One other interesting point about these presentation layers is they actually form a tree, which is why we call it at the presentation tree. And what that means is if you get the presentation layer and then ask it for its sub layers or its super layer, then those things you get back will also have their animations applied to them. And that makes it pretty easy to do certain types of operations on the regular layer tree and certain types on the animating layer tree, the screen layer tree. So the two examples we have here, which are the most common uses for this thing, are that you can use it to produce the form values for animations, and this is obviously what what the implicit animation does. So in this case, we say we create an animation, we set its key path, and then we'll set its from value to be the current presentation value of the property we're animating. And then obviously add it to the layer. And that's exactly what the implicit animation's doing, that's why I was stressing that they animate from the current screen value to the new value, 'cause they use the presentation tree.

So secondly, hit testing is the other big use case for this. So when hit testing, you would just, you know, use the normal hit test method, but actually hit test on the result of the presentation layer. And because, as I said, this gets you the animating subletters, then you will actually basically hit test across the entire tree in its kind of current screen state. And that will give you back something, and that will still be a presentation letter that you get back.

So that's why at the bottom we have this call to the model layer method. And what that does is it takes the presentation that we have and gives you back the original underlying model object. So obviously if you're head testing, you want to find exactly where you are or what you hit. But you typically want to know, you know, what is that in the layers I created, not these ones we've just been copied from somewhere. And again, the presentation tree is really why you want to implement that initWithLayer method I mentioned. If you don't do that and you have local state in your layers, then things may not work quite right.

Okay, so now we're gonna switch gears and move away from animations and talk about tiled layers. So we have this base class here layer, and it can display most types of, well, pretty much any CG image. But there are problems in terms of graphics cards. Graphics cards can only really display what they can texture from. And so they typically put limits on the size of the images you can display, which is normally somewhat bigger than your screen, but not as big as the largest camera picture today, typically. Other problems we have with the base class layer are that you don't have the ability to provide multiple levels of detail, or you don't have the ability to provide the image data in little chunks. It's all or nothing. So what the tiled light does is it gives you a way to define this kind of tiled image pyramid thing.

And that means that we can get the image data in lots of little chunks. And we also get them, you know, not all ahead of time, but we get those little chunks from you when we know we need them. So you can imagine you could create a million-by-a-million-pixel image through one of these tile layers.

And then, you know, scroll around the screen, and we will find out, you know, exactly which tiles we need and then ask you for them at that time. Obviously, that has a lot of advantages. The one big change, I guess, and you have to be aware of, is that if you buy into this model, then you're also buying into the fact that your drawing is now all going to be asynchronous.

That means that your drawing context method will no longer be called on the main thread. It'll be called on some background thread, potentially more than one at a time. And obviously, the reason for that, again, is that we want this thing to be kind of smooth and fluid, and we don't want the UI to block. And we want the images to come in over time and in lots of little pieces.

So here's a code example. And basically what we're going to do here is just create one of these tile layers, set its levels of detail to say we have eight layers in this image pyramid. Obviously any kind of pyramidal scheme, it's going to be top level will be 100%, then the next one will be 50% scale factors and so on. And in this case, we're also going to say, bias those scale factors up by three. And what that means is that we're actually requesting the pyramid to store three levels that are greater than 100%. So we're also going to get a 200% level, a 400%, and an 800% level of detail. And the reason we're doing that is because in this case, we could be drawing things from a PDF or from vector artwork. And in those cases, we know that we can draw those as finely as we want at any resolution. So if you zoom right in to 1,000%, then you don't want the pixels to be blown up. You want the tiles to be redrawn for the current level of detail.

Finally, we're just going to set the tile size to be something. This is a good standard value. And then we're going to give ourselves a delegate so that we actually get a chance to draw into this without doing any subclassing. And this example will follow on in the next slide, but one kind of side topic here is that we want to talk about how you use core images in layers.

One of the, the main thing about a core image, a CI image, is that it's not a raster data structure, it's not pixels, it's kind of a recipe to produce pixels. And that's great and has a lot of advantages, but it's not so great for core animation because core animation is totally built around this idea of having things we can give the graphics card to texture from. So there is no way to just take a CI image and put it into a layer. You have to draw it into that layer somehow. And the most simplest way we can find to do that is basically just create a tile layer and just draw the CI image into that tile layer.

The other method you could use is using OpenGL, and in some cases that can get you a little more performance, but again, it has all these kind of GPU limits and you have to use OpenGL, and you have to be aware of, if I create an image 10,000 by 10,000, is that gonna work on my current GPU? So really, we recommend you just use the tile layer, and then maybe think about other options if that really doesn't cut it for you, which we think it will.

So back, this is the kind of follow-up from the previous example. So here we're gonna actually implement the draw method to draw a CI image into our toddler. And you can see it really has literally one long line of code. So we're gonna take the CI image and draw it into the toddler. And because before the toddler handed us off our CG context, it actually set up the scale factors and the clipping rectangles and all the kind of CTM stuff. So all you really have to do in the toddler draw whatever you're going to draw, and then everything just works and you only really draw the tile we care about and at the right level of detail. So at this point I want to show you a quick demo of what I'm talking about.

So here we have a little application. And it's basically just the test app for the Tiled layer. So you can see what it's going to do is I've loaded a PDF. And this is kind of one of the stress test PDFs in that it has lots and lots of lines and lots and lots of little bits of text at different angles. And you can see this is all in a Tiled layer with tiles of data, so we've cached this level of resolution. But the interesting thing now is because, firstly, I guess, is that our UI remains responsive because it's cached. And also I can zoom in and then it kind of comes up. So I can kind of zoom in and out of this thing. I guess the thing to stress is that Core Animation is taking care of all of this. Really all I'm doing here is setting the scale factors of these layers. And then the Tiled layer is saying, what is he rendering this time, and what do I need to ask the application for? And also you get this kind of nice crossfade effect. Also, there's a garbage collector. so that it only lets the tile layer cache get so big. And after such and such, so many tiles have been added, you'll see holes appearing. But they page in pretty quickly, because we can draw these tiles fairly quickly. But you would definitely see some kind of UI lags and stutters if we weren't using the tile layer. So I guess the big takeaway from this is that if you want to preserve UI fluidity at all costs, maybe at the expense of things taking slightly longer to draw but look better when they show up, then the tidal error is a pretty good solution. So the next example I have to show is an image-based one. And this is using the CI raw filter, which is a way of taking a raw image and loading it, getting a filter chain and asking it for a CI image. So I'm just basically using that fragment of code I showed you a few minutes ago and rendering the CI image into the tidal error. So you can see this is a fairly big raw image. I think it's like 10 megapixels, 11 megapixels. And so you can see we're seeing it at 25% resolution. And again, I can just kind of zoom in and zoom out, and you can see things redrawing every now and then.

So same kind of stuff. I guess the other thing here is now because it's CI, and it's a CI raw filter, we can kind of modify the properties of the raw filter. And also, again, you can see that we're doing these things in little chunks. So sometimes that may be annoying, but it's the price you pay for this kind of UI smoothness. Just to show you that this really isn't a trick-- or not a trick, but scalable-- so I actually have a-- Which one is it? This is a 20 megapixel image. You can see now we're showing at 12%. This is like 80 megabytes of data. And at all times, things stay pretty smooth and fluid, and the filters kind of still work.

And you see right now the GPU is actually being pretty taxed to render these filters. You can see by the length of time it takes the tiles to show up. So-- So anyway, we think the title layer is pretty cool, and we recommend you use it. So back to the slides. Thanks.

OK, so next topic is 3D, because as everyone knows, Core Animation is a 3D engine. Well, it's not. But a lot of the demos we show you may give you that impression. But so we should really stress that CA is a 2D, 2 and 1/2D engine, which means that it's lots of planes, but they can be arranged in 3D space. So what I want to talk about is how you actually do that.

How do you create these demos or these plans? And the way you do that is you set up some kind of camera on your container layer. And the way you do that is you use the sublayer transform property. Now the sublayer transform was expressly added just so you can have some matrix which affects all of your sublayers. And this is basically ideal for doing kind of camera perspective type things because that's really what you want. I guess I should also say that this is applied relative to the center of the layer it's attached to, just like the other transforms. So in this case, this little code example is gonna show you how to set up typical camera matrix which we use on all these 3D demos.

So the first couple of lines are going to create the perspective component of our matrix. We currently don't have a function to do that for you, but given that it's only one line of code, I'm not sure it's really worth it. So anyway, you do this minus one over depth thing in the right matrix component, and this is basically going to get you this perspective effect, because what this does is the M34 basically means take the Z component of my model, divide it by something, and then add it to something else. And then you get the perspective foreshortening effect.

There's papers on the web you can find about this kind of stuff. So the next two lines are applying some kind of camera orientation and positioning effect. You do this after the projection, after the perspective. So in this case we have some precooked rotation matrix which presumably we created through the CA transform 3D rotate functions. And then we're going to just offset some XYZ amounts and then add that to the sub-layer transform. And once we've done that, any sub-layer we put, any layer we put as a sub-layer of this guy with a sub-layer transform, when we move him around and rotate him, the perspective effects and the camera effects will all be applied to him pretty much, you know, they just will. So then the next part is once you have this 3D space, then you need to position things within it. The way you do that is with the standard position property, but also with the Z position. You know, now we're in 3D space, so the Z component of the layers becomes important because that's where the foreshortening and the perspective comes from, amongst other things.

So then you can also use the anchor point again to control the reference point. And new in Snow Leopard, we've added this anchor point Z thing, which means you can actually now get full 3D control of your reference point for transforms. And that's very nice because that means you can now have 3D layers which rotate about an arbitrary point in space. Not their center, but some point elsewhere. Also, as was mentioned earlier, we are now trying to add support for true intersections between layers. If you have two layers which look like this, we will slice them up in the right way and render the slices in the right order. That's something that's expensive for us to do. Only do that if you have to, but it can make things look a lot better when they intersect.

Okay, so next on our whirlwind tour of random topics is threading. And I don't think we've talked a whole lot about this in the past. So basically, you may want to use threads for drawing these days. You have lots of cores, and lots of them are going to be used for data work, but graphics can be threaded as well. So to that end, CA layer is itself basically entirely thread-safe.

This means you can set any property, you can get any property, you can do basically any kind of restructuring, changing the sub-layers, all these kind of things. And we will pretty much guarantee that when you do that, you're never going to corrupt our data structures. And we will pretty much do that until you get the right results.

So the one thing that isn't done for you, though, is that when you fetch a property of a layer, the value you get back is not guaranteed to be valid if another thread is setting the same property at the same time. You can imagine two threads come in. They both set the background color of a particular layer. And then one of them fetches that property while the other one is about to set it. And now you're kind of in this window of uncertainty. And the object you get back may or may not still be valid. Because we don't order release things back to you for performance reasons. So it's kind of up to you to guarantee that anything you fetched remains good for the lifetime you need it.

We now have a way to let you do that, which is by adding these transaction locks. And so the idea here is that these are the same locks we use internally. And so you can take the transaction lock, do any kind of read, modify, write style code, and then unlock the transaction lock. And it has to be emphasized that this is a spin lock. So if you hold this thing for a long period of time, everything is going to grind to a halt in a pretty nasty way. But the idea is you can at least have enough synchronization to let you query a property, retain it or set something else based on its value before you then unlock and use your retained value or go along and do something else. One other thing worth mentioning about threading is that you may be able to avoid it entirely just by using the set_by_d for keypath. Obviously if you were going to set a subproperty of an object like a filter, for example, then filters are not thread safe. So you can't just go off and set those properties directly if multiple threads are going to be doing that. However, if you use the set value of keypath with the right current construction of a keypath to reference the right property, then we will make sure, because we have copies of all these filters, that we will make sure that everything works as intended.

And I guess the side effect of that is you'll also be able to get implicit animations on these filter properties because that's kind of done the same way. One more point about threading is that we try to guarantee that when you call send_needs_display on a layer, then we will call your display or drawing context method on exactly the same thread, because that's pretty important if you want to multi-thread your drawing. That also holds for the layout methods and basically any other callbacks we might be issuing. And finally, this is back to the two trees thing. As I said earlier, we need a run loop to be able to flush out your updates for you. So if you're on a background thread and you're making layer changes, then you may not have a run loop, or if you do, it may not be running, or running fast often enough to push your changes out. So the simple solution there is that every time you do something and you're about to block or sleep or whatever, then you can just call this transaction flush method. And that will basically push any cute changes that are in your current thread off to the screen. And that's the thing we're going to be doing automatically in most cases. But you need to do it sometimes.

Okay, so most of, up to this point, what I've been talking about has been very general. You know, just CA as an API, no real platform dependencies, apart from things like core image. But so obviously CA runs on both platforms we have these days, you know, iPhone and Mac. And iPhone is a lot smaller a platform than the Macintosh, the Mac. So when we took CA to the iPhone, we really decided that, you know, we could only afford to take the bits of it that we really needed, or that we really thought that were things we could do with the most possible performance. So anything else we kind of left behind, at least for now, And so there's this list of properties which are no longer, or which were removed from the Mac version as we took it to the iPhone. So you can't do things like rounded corners yet. You can't do borders, masks, shadows, filters, obviously, layout managers. And we really don't think this is a big problem. I mean, most of these things, if we did implement them, they'd be so slow you wouldn't use them anyway. And the other thing is they're just kind of convenience things. Like layout managers can be implemented just by subclassing and using the layout sublets method.

So the next point is, I'm sure you're aware of this, but the coordinate system or the coordinate space in the iPhone is upside down, at least to us Mac people. And what that means is that, you know, if you have code that you want to run in both places, then you need to deal with that somehow. And so up to this point, it was kind of hard to do that because if you just flip the transform, that doesn't do the right thing because, you know, your images flip as well, whereas images have a native kind of memory ordering, which needs to remain static.

So we added in 10.6, this new property called geometry flipped. And what that does is when you set that on a layer, the entire subtree from that point becomes flipped, but only for geometry. And so the idea here is that you can take code that runs on the iPhone for the producers, say a layer subtree, and take it back to the Mac, attach it to some layer you have, set the geometry flip property to be yes. And then hopefully everything will just work correctly.

At least that's the idea. So again, it's worth emphasizing that the images will not be flipped by this thing. You know, because images are images and they have just the standard order and you always want them to come up the right way up. So now let's talk a little bit about performance. And these things apply mostly both to the Mac and the iPhone. Obviously, performance is more of an issue on iPhone because it has a lot less capable hardware, although it does also have a smaller screen, of course. So firstly, the number one thing you want to do, if you can, is avoid all off-screen rendering. And what off-screen rendering is is that every now and then, most of the time we're going along happily using OpenGL when we're rendering things. And we're just taking the layers you give us and just drawing them straight to the screen. But every now and then there are certain types of effects which means we have to halt the screen rendering, create a new buffer to draw into, a new image if you like, draw things into this and then copy that to the screen.

And that works out fine, it's really still a lot faster than anything else we could do. But there is a bottleneck there. You can noticeably see the speed dropping, especially on the iPhone where it's much more of a cliff than a bump or something. So the list of things you should avoid if you want to avoid off-screen rendering is having group opacity effects, where you have a layer and a bunch of sub-layers, and then you change the opacity of the parent layer.

And obviously, what we have to do to render that is we can't just multiply the opacity through all the layers because we'll get the wrong result. So we have to do off-screen rendering. Similarly, any kind of masking, which isn't particularly simple, any type of core image filters or shadows, although those are already only on the Mac right now, and certain types of transition effects and certain types of 3D transforms. If you have a layer and a sublayer and they both have a 3D matrix, then because we're a 2D model, they have to be kind of projected back into 2D space. And then we have to do that twice, then we have to render one of them off screen.

So the next performance rule is minimize blending. And blending is basically where you see translucency in your UI. Obviously the way graphics cards work is if you're just drawing a pic data, you can just kind of splat it straight to the destination. But if you're drawing things which are translucent or have an alpha channel, then you really have to-- when you're drawing them, the graphics card will have to look at the pixel, take the value of the destination, read it back, and then do some kind of math between the two, and then copy that back to the destination again. And that kind of costs quite a lot, especially because it also means that we have to actually draw the things that are underneath translucent content, whereas if we know the things are opaque, then we can just most of the time avoid entirely drawing anything underneath it. So this is really key, and especially on the iPhone, where, you know, again, the hardware is more limited, so the amount of throw rate, the amount of blending bandwidth you have is a lot less. So, okay, so what do we want to do to avoid this? Basically, we just want to get rid of the alpha channel from our layers, or layer images which don't need them. So if we're drawing into the layer, the way we would do that is we will basically set the opaque property of the layer to be yes. And that basically tells CA that when it's asking you to draw into the backing store of the image of the layer, then don't bother creating an alpha channel. Similarly, if you're providing CG image data by just sending the contents of the layer to this image object, then you need to make sure that it has no alpha channel.

I should just say that sending the opaque property on these kind of layers has absolutely no effect. So how do you create an image without an alpha channel? Well obviously if you're reading an image from a disk or a flash or some kind of file, then you need to make sure that the image format does the right thing. So you may want to use a JPEG because they don't have alpha channels. Apparently there may be some ways you can create pings without alpha channels. And then alternatively if you're going to create the image from data, you have some raster memory, then just use the image alpha none bitmap format in CG. That tells us that you have no alpha data. So one thing I should also say is that if you're drawing into the layer and you -- well, if you're -- for some reason you can't move the alpha channel because you need these things to blend together and performance is still too bad, then you may want to consider collapsing some of your layer trees. Anything we can composite at runtime, if it's not animating, you can composite at draw time. You can have one layer and draw two images into it rather than having two layers with an image each. And obviously anything you can do ahead of time once rather than us doing it a million times every frame is going to make things a lot faster.

Okay, so this is kind of hard to actually even discover when this is happening. You just see things are slow. So we do give you some debugging options. And so basically you just set the CA color or paint environment variable on the Mac. And on the iPhone, the way to set the same option is via the instruments panel. So you can see I've highlighted it on screen here just so you know where it is. And instruments actually has a number of interesting core animation options to do things like flashing random parts of the layers when you're drawing them. highlighting things in other colors based on what's happening. But this diagram really just shows you this opaque opacity blending thing. And so what you can see here is I cooked up some boring Nib file in IB and then put it up on the iPhone and turned on this option. And you can see here that the green things are good, they're opaque, and the red things are bad, they're translucent. And so this is what you'll see if you turn on this option with the UI running. And the idea here is, or the goal, I guess, is to basically get rid of all the red as much as possible. And if you do that, you should see frame rates improve pretty dramatically. Okay.

Okay, so we're on to the last section of the talk now. And finally, I wanted to talk about what is coming up in the next OS release. So just a few of the key features. So first of all, we're gonna add support for trilinear filtering, and this is really just exposing a graphics card feature. Trilinear filtering is a way to get good quality downsampling of images because, that's also called MIP mapping, and the image you pass the GPU then has multiple levels of detail, again, kind of similar to the tidal app. and the graphics card will select the most optimal level. And this basically means you get better quality images on the screen because there's a lot less aliasing if you pick a small image if you need a small image. And also it's faster because you can minimize the bandwidth the graphics card is actually using. So this is very easy for you to set up. It works on all types of layers-- or all types of image-based layers, I should say. But the one drawback is that not all graphics cards support this on all images yet. So you can see the OpenGL documentation. But basically, if you have a power of two size image-- it's 256 by 256, or 256 by 128, or something like that-- then it's pretty much guaranteed to always work. If your image is not power of two size, then it will work on some graphics cards.

And anything else will just fall back to the regular method, so it's not terrible. There's also an LOD bias property, and that is a kind of a graphic C term, but it basically means you have the slider and one end is sharp, one end is blurry, and you can kind of move yourself along the slider. Oh, I should also point out that this now works with Tidal. So you can kind of get some level of trilinear, you know, three-way filtering by setting that property on the Tidal. And in this case, it's not a true per pixel graphics card based property, tri-linear filtering, but what we will do then is every time we draw the tile layer, we will pick what we think is the best level of detail and then kind of blend between the two physical surrounding levels. So if you're zooming things in and out, it can mean you get a lot less, you know, fewer pops as you transition between levels. So anyway, again, this is an example of that, and really it's just these two properties. Set the minification filter to be tri-linear and then you can play around with the bias. It really does make things look a lot better in some cases.

So secondly, we added a gradient layer. The gradient layer is just a subclass of the CA layer. And as its name suggests, it will draw a gradient into its shape. So it's just an axial gradient right now. And you get to specify, you know, an array of colors, an array of locations for those color stops. You can say, you know, this one is here to here and then here. And you also get to specify two two-dimensional points within the layer shape to give kind of the endpoints of the axis. So then that defines your gradient. And then what we do then is just draw that using the GPU. And obviously because these are just kind of numeric properties, you can animate all of them using the regular CA animation. So you can animate colors fading in and out, gradients spinning around or moving the gradient stops, kind of everything is up for grabs. Another example should be fairly self-explanatory, which is really creating less, setting up much properties again. So let's not dwell on that.

So the next new subclass is called the CA Transform layer. This is a little different. So up to this point, we've talked about 3D and we've talked about 2.5D. And layers, we've always said, are 2D, which means that you take this 2D graphic element and then you push it into its parent in some three-dimensional space and then flatten it into the parent's plane, thus preserving the 2D-ness throughout the entire pipeline. And that's great. It means you can do kind of 2D imaging effects on 3D, pseudo-3D content. But it means you can't set up very complex hierarchical models of 3D elements and transform them as groups. So for example, you wouldn't be able to create a cube out of six faces and then rotate the entire thing. You'd have to create those six layers and then rotate each one of them separately about some center point.

And as you can imagine, that's either impossible or gets really, really complex really quickly. So what a transform layer does is it basically extends the layer model a little bit and says, these are layers which only grouping constructs. They don't have any content of their own. You set their image to be something, you'll get a warning and nothing will happen. But their geometry does apply into their sub layers, but without this flattening step. So now you can see that you can do these kind of cube-like effects in this model because you can create a transform layer and give it six faces and animate the rotation properties of the transform layer, and everything will rotate without being flattened into some kind of mushy mess.

And the way we do that internally, which is worth bearing in mind for things like depth sorting, is that any sublayers of a transform layer are effectively hoisted up into its superlayer before we do any of this kind of processing. We take the matrix from the transform layer, concatenate it into the layer itself, and then push it all back up to the next level. So we'll have a demo of this in a little bit. Finally, the last feature is particle systems. And you've probably seen these things in any number of movies, and ours is not quite that cool. but it's basically a way to kind of spew out lots of little images and have some kind of control over where they go. And so, you know, you see a lot of the keynote presentations you see at this conference have these little kind of sparkly effects, and that's kind of things you can do with particle systems.

So our particle system is structured as another layer subclass called the emitter layer, and then each emitter layer has a number of cells. And a cell is basically a way to kind of a template for particles being emitted. Now the layer has what we call an emission shape, which a number of different options, things like rectangles, circles, spheres, cubes, et cetera. And then the cells then define how particles are emitted from those shapes over time and at certain rates. And obviously since the cells have these kind of template-like properties, every particle will have some set of these properties with some kind of randomization factors. And one really interesting feature is that cells are not limited to just emitting particles. Particles they can emit can also emit particles themselves because cells can have subcells and so on. So you can kind of create these nice organic-like effects with things emitting things emitting things and what have you. Finally, emitter layers are also kind of similar to transform layers in that they have this 2D/3D option. If you choose by default that 2D, and that basically means that again, all of their particles are flattened into their layer plane as they're rendered.

But if you say, I want a 3D particle emitter, then what actually happens is they act kind of like a transformer and their particles are all rendered into the parent, which means they preserve 3D geometry into whatever else is in the parent. So now I wanna show you a couple of demos about these new features.

OK, so first of all, this is the example of the transform layer. And so you can see now we have these two cubes. This is a very quick demo. So we have these two cubes. They're spinning. You can see they're spinning separately, but they're living in the same kind of 3D space. And this is really very simple now because we just have animations on the two layers. But then we thought, well, let's take it up another level and put them inside another transform layer with another cube and then just kind of rotate that as well. And you can see the things inside are still rotating in their own little space. But now they're being kind of modified by the big cube. And this is the kind of thing which-- I'm not even sure if you could do this with the old model, but it would be a lot of math and a lot of pain. OK. So now let's look at the particle system.

So this is a pretty simple particle system. It has three objects. It has an emitter layer, a cell which is emitting the first big blobby spark. And then that also has a subcell which is emitting the spark, the little shower of fireworks. And so we do something where we play with the timing of the subcell so that it only fires for a very brief period of time when the other one gets to a certain point. And then there's this big kind of shower because we set the emission rate, which is the thing that controls how many particles are emitted per second, to some big value. So if I highlight the layer here, you can see that the emitter layer is not clipping right now, but we can turn that on if we want. So now the emitter is clipping its boundaries. And as I said, it's being flattened into 2D space. So you can see it really is acting like a plane.

So it's kind of more fun, though, if I turn off the clipping and turn on the 3D. And that doesn't really do anything now, because we're face on. But you can see when I start to rotate this thing that now we have a kind of a 3D particle thing going on here. And I guess I can also animate this thing. So like I said, this is very little amount of code. And we have these nice effects going on with nice lots of smooth animation.

So one other example of these, which is basically the same demo app, so don't expect anything new, is another kind of particle system. Again, we have this kind of rotation feature. And this one has another of these kind of particles on particles effects. So you can see that it's just kind of working.

So I guess one other thing worth mentioning is I'm doing this kind of pausing thing. And that's basically done with a timing model where when I'm starting to scrub, I'm just setting the time, the speed of the emission layer to be zero and changing the time offset to clamp me to my current point as I tried to describe earlier. So it's kind of a nice effect.

So with that, I think we're about done. Tomorrow there's another core animation related session, which is Troy's layer-backed view session, which has a lot of NSView type stuff, but also a lot of just general core animation things. So you should definitely go to that. Finally, we're going to have a lab after this, which I guess is downstairs in the graphics lab.