Welcome. This is session 128, Cocoa Drawing Techniques. My name's Ken Ferry. I'm an engineer in the Cocoa Frameworks Group and I hope you enjoy the talk we have for you. So today I want to discuss drawing at the Cocoa level. First we're going to talk about just an overview of what drawing consists of in Cocoa. This will be fairly quick. It's not intended to teach anybody anything, it's just to get us all on the same page. I will try to emphasize a few areas that I think are misunderstood. The bulk of the talk is going to be about NS Image.

It's a somewhat misunderstood class. People sometimes have some trouble with it. I'd like to just talk about the way it works. I think it's probably not so bad as you may think. And then I'd like to talk about the lower level APIs, particularly at the Core Graphics level afterwards, and we'll just run through and the point of that section will really to be what all is available, under what circumstances might you want to go look into it in more detail. And we'll talk about it and just know what's it good for. Okay. So let's start out with the overview though. So in the overview there are two things I want to talk about. I want to talk about graphics contexts and I want to talk about coordinate systems.

So a graphics context, as you may have heard in other talks, is a drawing destination. It's a canvas. It's where the drawing ends up. So it's a very fundamental role in any drawing, of course. We're going to be mostly talking about drawing with NS graphic context and CG context rep, which are very, very similar objects. There are plenty of other contexts in the system.

There's a CI context, there's an NS open GL context, there's a CGL context. In all cases they are drawing destinations. That's always what it means. But it's not just a destination. The other thing that contexts do for you, the reason it's called a context instead of just a destination or a canvas, is that they hold the properties.

Where the properties are something that you set into the context, bits of state and then they affect later drawing. For example, you can set an image interpolation quality into an NS graphics context and what that does is then later, if you're drawing an image whose resolution does not match the resolution of the destination, it has an impact on how the resampling is done, how long it takes and how high quality it is. There's plenty of other bits of state that work like that.

We'll see some of them during the course of the talk, but then something you might notice with that is if that's that way it works if when you set these properties and then it has affect on later drawing, that kind of side effect can be bad in some situations. You know, if you call some drawing method, you don't want it to effect lots of other stuff later in the program. So there's some convenience methods to help manage that sort of thing.

All of those properties taken together are called the graphics state in the context, and you can save and restore individual states. So you issue, and I'm going to point this out because this is the instance level method, save graphic state, restore graphic state, you call a save, it pushes it onto a stack, you call a restore it restores whatever the last thing you saved was in those nests, that's why it forms a stack.

And those are exactly the same. There's similar functions down at the Core Graphics level. CG context save G state and CG context restore G state, they're exactly the same thing as the instance level methods up at Cocoa. They can be interchanged, now that that would be a good thing to do.

Let's talk a little bit about how they are different, however. Since I've so far emphasized that they are the same thing. Oh sorry, I'm still talking about how they are the same thing. So Cocoa drawing really is the same thing as core drawing. I've spoken to a few developers who didn't understand this really. They didn't understand NS rect field and CG context field rect were really the same kind of drawing.

You can interchange them, you inter sparse them. They are not actually bridged classes however. If you have an NS context graphics and you need to do some Core Graphics drawing that needs to take a Core Graphics context, what you do is you issue the graphics port methods to the NS graphics context, that'll get you the CG context rep.

And then if you need to go in the other direction, there's a class method, graphics context with graphics port colon. There it takes the graphics, or the CG graphics context and then flipped, which I'll discuss that parameter a little bit later. You should think of this method as being very cheap. It's just an extremely thin wrapper.

So there are some differences, however. So the reason I wanted to really talk about the save and restore in G state before, is because Cocoa has an additional stack, besides just the G state stack, which is that there's actually a stack of graphics contexts. There's always a notion of the current graphics context and you can set the current graphics context and the way you deal with the stack, again, there's a class level method, save graphic state class level restore graphic state, and they're not the same as the instance level. It's not just a cover that calls the instance level method on the current graphics context.

What it does is it takes the entire context itself and the context itself will have the stack of states within it. It pushes it on to the stack and then you would call set current context to make a new one current. And that's where your Cocoa drawing ends up going.

So that's the point, so because we have this notion of the current context in Cocoa, usually it's implicit. So at the Cocoa level, if you want to set a color into the context, you do something like this. NS color blue color set. At the CG level, you don't see that, you see explicitly passing in the context. And the difference there is because Core Graphics doesn't have a notion of the current context. And often, also, you don't work directly with the context in Cocoa as much, because often it is implicit.

If you're drawing an attributed string, one of the attributes of the string is the foreground color attribute, and the mechanics of that are that finally, when it draws, it's going to have to set that color into the context and the draw glyphs with it, but that's hidden from you, so you don't end up seeing it so much But part of the reason for saying this is that sometimes you really might want to be able to explicitly deal with the context and if you do want to deal with that, then you should look at the header CG context dot H. and one of the things that you can see with that is you'll see all of the various different bits of state that actually go into a graphics context, which are not quite so obvious at the Cocoa level because they're distributed all throughout the framework.

So that's an interesting thing to look at. But really the point of that last section was to note the difference between particularly the stack of graphics context and the stack of graphics states within each state. All right. Let's talk about coordinate systems a little bit. So when you issue a command like this, NS rect field, and then you send in a rect, zero, zero, 100, 100, what does that do? Well it's obviously going to fill a rectangle somewhere. Where is that rectangle? Well, it's with respect to the current coordinate system.

And the point of this is going to be that don't try to pretend that there's a universal coordinate system. Because really the foundation of Cocoa's resolution and dependence architecture is that there's not. There are lots of coordinate systems. There's a screen space. There's a window space. Every view has its own space. Every image has its own space.

The image space you might be a little bit less familiar with. It comes up a little less often. And then there is also, the final place that's really important when you're actually drawing is that the graphics context has a coordinate space within it, and the way this works is that in the normal view drawing, before draw a rect is called, the context in the graphics, the coordinate system in the context will have been set up to match the view context and then you can modify it from there, possibly.

Similarly, if you're dealing with images, the way you draw into an image is to issue the lock focus method, which we'll talk about later. And that has the same effect. That's going to set up the coordinate system in the current graphics context to match the native space of the image.

So it's stored in the graphics context. It's one of those properties that is something that gets saved and restored when you're using those methods. It's called the current transformation matrix, is the way it's stored, which is referred to as the CTM. This is worth pointing out because it shows up in the APIs.

So I wanted to make a little analogy here to say why I'm harping on this. Why is a rectangle like a char star array? Like a C string, that is? Well, they're similar in that we're all used to these days that a C string is sort of meaningless unless you have this string encoding that goes along with it.

Well, really a rectangle is the same way. If you just have a bare rectangle and you don't understand which coordinate space it replies to, then you don't have any information. And really what you need to do for resolution dependence, to start getting that right is to always think about that and to make sure that you're not trying to deal with rectangles that are not in the same space and comparing them as if they are. So the way of dealing with that is instead you should convert between spaces.

So there are lots and lots of different methods for this. Oh, and this is a good time to mention the difference between frame and bounds. A perennial question for Cocoa developers of all ages. Bounds is the location of a view in its own coordinate system. Frame is the location of a view in its superview's coordinate system. That's all there is to it. So if you want to move a view in it's super view, that's when you're calling set frame.

When you're drawing, the frame is entirely irrelevant, well, mostly irrelevant anyway. I mean, who knows? You can do whatever you want. But for the most part you only care about your own coordinate space, and that's the bounds. Most of the methods in NS view, if they don't say anything otherwise, that take points, sizes, rects, they're all going to be in the view's own space.

Okay. And there are just a lot of methods for converting between these spaces, some of which are new. I think it's not really worth going into these in too much detail. They're there. Please use them. All right. So another thing I wanted to talk about was the concept of flippedness before we get going too much further in. you remember in the graphics context method the way that I said you create an NS graphics context if you have a CG level context that you need to ref, you issue this method, graphics context with graphics port flipped.

Okay. We discussed the first parameter already, that's the Core Graphics context, but what is the flippedness parameter? Well a flipped coordinate system is one where we think of zero, zero, the origin of being at the top left and positive number of going down the Y axis as opposed to the unflipped coordinate system has a more mathematical model where zero, zero is in the bottom left and positive goes up. Now the interesting thing to understand about this is that it doesn't actually effect any drawing unless something looks for it. It's just meta data.

It doesn't necessarily mean that the context is set up that way, it's just sort of telling anybody who's interesting in knowing, yeah, you should think of this context as flipped. Okay? So lots of higher level Cocoa constructs like cells do look at the flippedness of the context and they will adjust for it, but lower level things like images do not.

They'll just draw however they're told to draw. Okay. So that's all I have to say about the overview, let's go ahead and talk about NS image itself for a little while. And to do that, though, we need to talk about two classes. We need to talk about NS image rep and we need to talk about NS image. To make the distinction, it's easier to talk about NS image rep first. So let's do that.

So in NS image rep is an abstract thing that can draw itself. But we don't deal with it abstractly, we always deal with the sub classes. For example, we have NS bit map image rep, we have NS PDF image rep, NS CI image rep, and NS custom image rep.

Basically what unites these sort of classes, what you see is that they all have intimate knowledge of the data that actually needs to be drawn. So when you issue the draw method on an NS PDF image rep, it's going to go to its PDF data, it's going to interpret it and it's going to put the bits in the context. That is the job of the NS image rep.

There are a few things, I guess, that I thought I should mention that are handled all the way up at the abstract level. There's a type registration system so that you can say when you make a new image rep sub class, you can say here is the data that I'm capable of dealing with.

And then when somebody goes and creates a new image from the data, if they have the type, they don't need to know particularly which image rep is appropriate. But that's what that's for. So it's sort of built to be sub classed. That's what it's for. It's really easy to sub class, not just for us, but for you. If you have new image types, this is how you want to add them.

So really what you have to do is every image rep has to have a size, so when you create it you should pass in the size in some fashion, and it has to be able to draw itself. And that's really it. I guess you need to implement NS code and NS copy so they can be archived to disk or it can be copied. But that's about it. There are other features you can do if they're appropriate.

There is, for example, pixel width and pixel height that you would use if you have a bit map type of image, but if you have something that's resolution independent natively, like a PDF, you just actually don't have to deal with them at all. We'll discuss how that works. In particular I want to suggest that as an exercise I think it would be really good to go try to write your own class that draws CG images. And the reason for this is that it's actually extremely easy.

You don't have to do it anymore because in Leopard we've added methods to make it easier to take CG images and put them into an NS image without doing any extra work. But it's actually really trivial to begin with because all you have to do is implement this draw method in a sub class as hey, draw the CG image.

So give that a shot. The methods, by the way, are NS bit map, image rep, NS CG image, and then the one that returns it is NS bitmap return CG image. You see there at the bitmap image rep level instead of the NS image level, because CG images, you'll see, actually has a, it's more like an image representation than like an NS image. NS image is a much higher level class. Okay. So that's NS image rep. So what is NS image then?

Well NS image is also something that can draw itself, but it is a much, much smarter class. So NS image rep is simple because it's supposed to be easy to sub class and so all the smarts get split off into NS image. What it really is - it's a container between NS image maps.

It's going to be able to choose between them and do lots and lots of other stuff. So here are just some of the features of NS image. Something that we'll talk about, we'll talk about this representation selection. So when you're asked to draw, how does the image choose which of the image reps to actually use to draw? Because it's always the image reps that really do the drawing. We're going to talk about compositing modes. We're going to talk about caching. That gives people some trouble. We'll talk about how to create new images from a drawing.

A few things we won't talk about later, so I'll say a few words about them now, if you want to load images from disk, there's some really nice methods for doing that. There's image name, which will search your bundle in a very convenient fashion. There's init by referencing file and init by contents of file. What is the difference between these two methods? They both take a file and create an image out of it. The first one references the file, so it's very lazy. It won't actually decode anything until it's asked to draw.

And if there are multiple representations, if you've done something at 1X and 4X for higher resolution reasons, it won't decode the 4X image unless it needs to. So that's how you avoid taking more memory if you don't need to. Init with contents of file, that one actually will suck the entire file into data, into memory right away. That's also necessary sometimes, because under some circumstances you can't guarantee that the file will still be there later.

So that's why you need to be able to do that. There's other meta data associated with an image that I think might have been covered to some extent in the controls talk that was just done, so you'll have to get that on video, I suppose, but you can also ask about it later. But I recommend that you go read the headers for things like the new alignment rect property and for things like set template, is another new property that really has to do with control drawing.

Okay, let's talk about the primary function of the image, as I said, it's to contain image reps. And when it draws, it needs to select an image rep to draw. So how does that work? Okay. Well the primary thing it's going to do has to do with resolution.

The main image drawing method here, okay, that's image draw in rect from rep, operation fraction. It has a, you have a destination rect in the context and you have a source rect within the coordinate space of the image itself. Often, by the way, if you just want to draw the whole rect, it's very convenient to just pass NS zero rect. It's a convenience that will just draw the entire image.

And then there's implicit piece of data which is always the case when you're drawing in Cocoa, which is you have the current graphics context, which has it's own coordinate transform. Between the destination rect and the current coordinate transform that's going to figure out a resolution, a number of pixels that need to be filled in in the context and that's going to be used to choose an image representation that is appropriate.

So you may have, if you have a TIFF for example that you used to create your image, there may be multiple resolutions available and this is how it's going to choose the best one, I guess I was going to say. There are other factors that used to be more important. It can also select based on things like color depth. If you have a black and white window, or a fore color window or say a printer of some sort, it can try to select an image that is more suited to that kind of context.

But these days mostly resolution independence is the really interesting bit. So how does that work? So if we look at the NS image rep level, all of the data for resolution dependence has to already be there, because that's what NS image is going to be able to work with. I already discussed that every image rep has a size. It has also these optional properties, the pixel width and the pixel height of the image.

I guess they're not exactly optional, I mean they have to do something. But the point is that for something like a PDF, what those methods will return is NS image rep matches device, which is basically how it says that whenever you're drawing, I'm always perfectly suited for the context in terms of resolution.

Okay? That's what that means. But if you have a bitmap sort of image rep, of which we have many, for example if you're implementing the CG image rep drawing sub class, you'd want to report an accurate pixel width and pixel height so that the image did a better job of choosing amongst the resolutions.

Okay, that's all I have to say about image rep choice. Let's talk about caching. So NS image is a user interface oriented class, which is to say that as opposed to sort of doing image processing, it's not for applying Photoshop filters, that's something that Core Image is very, very good for. It's not even for writing out data at the moment in very specific formats.

If you have something like, I have a JPEG and what I want to do to it is add an x-if tag that has the GPS information in it, that's not something that NS image is well suited for, today anyway. We might want to bring that up. But you need to go down to the image I/O level for that, which is part of Core Graphics.

Yeah, but the point is that NS image is more about putting things in buttons, it's more about putting things really on the screen. And so it's going to cache by default. The first time you draw it it's going to create itself a cache. And it's actually even allowed to throw out the original data, unless you tell it not to.

If you have something like a PDF, this is where this really comes into play. The first time it draws it takes a really long time to draw because PDFs are not cheap to draw. But it draws into a cache, which is a bitmap, and then the original PDF data is actually completely thrown away. So as long as you're just redrawing it in the same way, in the same button, that's going to be what you want both for speed and memory purposes.

But there are situations where that's not going to cut it for you and so you need to be able to control the cache. So for example, if you're using the image in two places in the interface, you don't want it to lose that original backing data if there was more detail that you lost when you wrote the cache. So one way to do that is just to copy the image if you're using it in two different buttons, maybe you'd just like to have a different copy.

That works fine. If you have something more like where the image is displayed repeatedly in the same place, but at different scale factors, probably what you need to do is you need to tell the image don't throw out the original information. And that's the magic method. This is set data retained, yes. So typically when you start looking at caching problems, this is really the first thing that you're going to want to look at.

We set that automatically for some kinds of images in Leopard. We've added some standard named images. I think Elise spoke about them a little in the Cocoa Today talk yesterday. Things like a network icon, or if you want the gear that shows up in the action pop up menus, we have those available for you now. And since we do kind of figure they'll probably be used in more than one place. We already set this on them. And then the big hammer we need to pull out is NS image cache never, which just tells it no caching for you.

Okay. That's all I have to say about that. Let's talk a little bit more of these other features. So there's a compositing operation, which is one of the operations that's passed into NS image when you tell it to draw. It has very good spots. You have to type it all the time. And what does this have to do with? This has to do with different ways of interpreting the alpha parameter in a color and different ways of blending an image with what it's drawing on, which is called the destination.

The screen shot down here is from a sample app, that's in developer examples, it's called composite lab. It lets you play with these modes. So doing it visually is one way to do it, but I actually like to talk about how you can figure out what they do given the mathematics of how these compositing modes are defined.

And the mathematics are actually given in the headers, especially if you look at the CG context dot H, you'll see completely spelled out exactly what all these compositing modes do. So let's start on this. So usually, like I said, so how is a color represented, colors that go to the screen?

There's a red parameter, a green parameter, a blue parameter and there's alpha. And we usually think of the alpha as meaning opacity, but it doesn't have to. And these compositing modes are basically about different ways of interpreting that. First, to understand the math, you have to understand first what is meant by pre-multiplied alpha.

It just happens as a convenience. Pretty much all the different ways you need to draw, the first thing you end up needing to do is you need to multiply the red, the green and the blue by the alpha to do all of your different kinds of drawing. And because that's always done for all the different modes, we tend to just do that once up front, and that's called pre-multiplied alpha. And that just means that the color is stored as R times alpha, G times alpha, B times alpha and then alpha. There's not data lost there, except in precision issues.

Yeah, but the mathematics already assume that you have pre-multiplied alpha. So the most standard composting mode we have is called NS composite source over. That's actually by itself an important thing to realize, which is if you want normal, if you don't want to think about compositing modes right now, but you need to pass one in, what you want is NS composite source over. Here's how it's defined.

The result is equal to the source plus destination times one minus source alpha. Okay, what do these words mean? Results mean what you get. Source is the image that you're drawing. Destination is what's already there in the context when you're drawing it. Okay. And what I want to say is the way you understand one of these formulas is to try to do it by special cases. So what would happen for a pixel where the destination was empty, where it was zero?

Well it would be result equals source plus zero times one minus source alpha, so you can see you just get the source itself. Okay? If the source is empty, what are you going to get? Well, you see, you just get the destination. And then as the opacity of the source increases, of the image increases, you see it's going to linearly show more and more of the image itself.

Okay? So that's the way that I find to be most useful to understand what these actually do. And here are a couple of cases and some of the ways that we use them within the App Kit. So like I said, NS composite source over. This is normal. Use this in almost all circumstances. Okay. NS composite copy.

The main reason for understanding this mode is not actually because it's so wonderfully useful, it's because Cocoa uses it sometimes. NS rect fill, for example, uses this. What this does is it just completely replaces the result with the new image that you're drawing, so there's no blending whatsoever. So that does not interpret the alpha parameter's opacity because it just completely replaces whatever you had. The alpha is still there, it just won't act the way you expect.

Okay. So the main reason to know about that is to understand when it's bothering you. So if you do something like, if you try to set a color, blue color with an alpha value of point three and then you use NS rect fill, it'll end up drawing that more or less opaquely.

Okay. So a couple of more interesting ones. So how about NS composite source atop? What do we end up using that for? Well, you've seen probably when you click a button in Cocoa, if there's an image on it, we need to shade the image darker so that it looks like its pressed hue.

And this is actually how we do it. What we do is we take the image and we draw a rectangle over it, which is half opaque gray, and we draw it with NS composite source atop mode. Now if you look at the definition of this mode, you'll see it's the same as the definition of source over, only the source parameter's also scaled by the destination's alpha.

So what is that going to mean? It's going to mean that where the destination is already completely transparent, you will not get any new drawing. It will stay transparent. Where the destination is completely opaque, it's going to work exactly like normal source over drawing. Where the destination is somewhat opaque, that's where you're going to see some coverage. Okay? So it's going to do exactly what you want for this kind of thing if you have anti-aliasing on the edges of your images.

Okay. And then there's NS composite plus lighter, which I'm going to go into a little bit more. This is a funny mode. This just adds just straight up the source and the destination to get the result. And what I want to say about this one is that interestingly this ends up to be a really good way to draw images flush, regions flush without pixel cracks, which is otherwise fairly hard with anti-aliasing. So here's now that ends up working.

Suppose that you have two non overlapping regions, okay? But suppose that they're meeting not on a pixel boundary. Let's see what happens if we try to draw this in the normal compositing modes. Okay so here's what we want to draw. Here's the formula that gets applied when we draw it.

And you should expect this. You get anti-aliasing. So when the middle pixel gets covered completely, you get full coverage. But on the two on the side, you get only half coverage. Now the bad thing that happens is what happens when we go to draw the other rectangle? Well exactly the same thing happens, and the way it works out is that you have not completely covered that middle pixel. Instead, you drew with half opacity twice. That doesn't work.

And you get these funny looking pixel cracks all over the image. So, here's just something that we discovered in the course of Leopard drawing. A good way to get around that turns out to be NS composite plus lighter mode. Okay? So you draw the first time, it uses this formula for applying it and you get exactly the same results. However, when you go to draw the second rectangle, okay, look at the way those two are going to add. You're just going to get complete coverage right in the center, anti-aliasing on the sides, which is exactly what you want.

What this ends up doing, yeah, exactly what you want. What this ends up doing is that when you're drawing in normal source over mode, if you have partial coverage on a pixel, partial opacity, what it's going to assume that that means, it's going to assume that it's always full coverage of the pixel with a certain level of opacity. When you're drawing in composite plus lighter mode, this is the difference, instead it's assuming that you are completely covering half of the pixel.

Okay? And the funny thing is that in our graphics system that's pixel based at its deepest levels in Core Graphics, we can't make that distinction when you're drawing in source over mode, so you end up having these resolution dependence issues. So that's a little bit subtle. But basically I recommend taking a look at the compositing modes. But that's all I have to say about them for now.

So let's move on a little bit. Let's talk about how you go about creating a new image from drawing. How you encapsulate drawing an image that you can pass off then to a button or an image rep or whatever you want, an image view, I mean. Well, you do it like this. It's very easy. You say NS image alloc init with size, you'd say lock focus, you do some drawing and then you unlock focus. Well that's easy. Now you've got an image that contains the string that you just drew.

But there's some questions. So I've claimed so far that NS image is primarily a container for its image reps. So what just happened? What are the image reps? What is the resolution of this thing? If we want to be resolution independent we need to understand these things. Okay. So the key to understanding this is to understand what is philosophically going on when you call lock focus. This is not what actually happens, but this is the right way to think of it. It'll give you the right intuition.

You're getting an off screen window, okay? There's a view in the window, the same as any view in any other window. You're drawing to that view. Lock focus sets up the context so that you are drawing to the view. Then you draw. And then when you do unlock focus it rips the pixels of the place where you just drew and assembles them into a new image rep for you.

Okay? That's what happens. So let's go through and figure out a few of those things. So first of all, it's going to be a bitmap representation. It's not resolution independent. It's going to be at the default scale factor. The way to think of this is that if you're drawing in a window, the same as you draw in any window, every window always has a default user space scale factor applied to it, so if the user has selected three as the scale factor and what you passed into init with size for the NS image was three by three, the number of pixels you have will be nine by nine. And then it's also interesting to note what it does to the image representations. It scraps them.

All of the other ones. It's replacing them. So when it draws into the view for the first time, it's going to use those representations, it's going to pick one, it's going to draw it, then you do your extra drawing and then when you pull it back out you have this new representation, and the others have to be thrown away. They really have to be thrown away.

Something I haven't mentioned yet is that all of the image representations for an image really need to represent the same drawing. They can be optimized for different destinations, but you shouldn't try to do something like put an entirely different image up there as the highest resolution image because you don't really have good control over Cocoa deciding to pick it, for some reason, and there have been, in system preferences, actually, was trying to pull tricks like this and it's causing some problems.

So, anyway, that's how that works. I think I've explained that enough. A few caveats though, that are associated with that, then. So there's a resolution issue. So this lock focus is going to be good in lots of situations. When you're drawing to image into views that you're not doing any intentional scaling with or images that you're not scaling, but sometimes it might not work.

Also for performance reasons, even though you think of a window as a good way to understand what's going on, please don't actually call the window method as a cached image rep, which is actually exposed. Because this turns out it's not a very efficient way to do it with Core Graphics now and we may actually end up having to create a window for you when you call this window method, which is a little expensive, so don't do that.

Then another interesting issue, which is not at all specific to NS cached, with lock focus is what happens with sub pixel rendering, which is called font smoothing at the Core Graphics level. You've probably seen this if you look really closely at text, you'll see that it has colored fringes on the edges. You'll see red and blue.

I always used to think that this was just some form of anti-aliasing where it's just somehow fooling your eye with the red and the blue, that's actually not the case. What's going on is that it's taking advantage of the physical layout of the LCDs on your display. They lay out from red, green, blue, in that order. And if you draw something red, that's going to look like it's weighted to the left side of the pixel.

So in a case like this W, you see it looks much more pixilated on the left than it does on the right and that's how it's coming out. These are images from Wikipedia. They're nicely in the public domain, so I can put them in my talk. But there's a really great article on sub pixel rendering, I recommend you read. But the point of this is it's not going to work if you're drawing into a transparent image.

So if you want this to work, you need to do something like fill with white first, or fill with any color. And this is just the technical detail that you end up needing to be able to track, if we wanted this to work you'd have to have separate red opacity, blue opacity and green opacity, and we don't have that. We just have a single alpha parameter, which is why that doesn't work today.

Anyway, if you need to work around the resolution issue, the right way to do it is to use NS bitmap image rep instead for your drawing, instead of using lock focus. The way that works is that there's this class method that let's you create a bitmap image rep just exactly the way you want. You have lots and lots of control. And then how do you end up drawing to it? Well, this was added in Tiger, I believe. You can create a graphics context from the bitmap image rep.

See here, graphic context width bitmap image rep. Then you would, and now you pull this stuff I was talking about earlier, you call save graphics context, necessarily the class method, which will save the current context. Then you make the new context you've made current, then you can draw to it, and then you restore back to what was already there. Okay? So not so hard.

Now one issue here that I'd like to talk about is mutability of NS image. So there are quite a few methods that we've seen which will change an image. You can call lock focus. You can call set size. You can call set flipped. We haven't really talked about set flipped too much yet, but that's the next slide.

Basically, even though you can do this, you sort of shouldn't. The right way to think of images as if they are immutable. You would create it from disk, you would get it set up properly, then you would stop messing with it. It's not something you want to continue changing. You might want to change it maybe at certain times in the future.

Maybe you say oh, now is the time for me to change my image for a while. But do it in phases. Don't just intersperse drawing with changing your images. So a good way to see what goes wrong with this sort of thing is to look at the image set flipped method, which is I think not so well understood.

So the set flip method describes the flippedness of the internal coordinate space of an image. Okay? External users don't need to know about it. So we have this dog cow, and the whole point of this is that the dog cow on the right is not actually upside down. Okay? The image may be flipped, but that just means that zero, zero is in the top left. It doesn't mean that it's an upside down dog cow.

Okay? Same as for, the right way to think as with view, some views are flipped, some views aren't. That doesn't mean the flipped draw their stuff upside down, that just means that their origin's in the top left. Okay. Yeah. So again, it has to do with coordinate systems that I want to harp on, because I would like everybody to start paying attention to them. We have them because this is really a good way to be able to compose larger, more complicated drawing out of smaller things.

When you're working within an image, you wouldn't want to need to know how the image is being drawn to the context, it may not even be being drawn to the context yet. There may be not context. So you need to be able to work locally. And again, there's no universal space. So whenever you have rectangles, just keep track of what space they're in and make sure you convert it appropriately.

So when is this internal space used? Well, it's used in lock focus. That's an obvious time. It's also used in NS custom image rep. I mentioned this very briefly before, but I didn't say what it was. This is a cool class. You should take a look at it if you're not already familiar with it.

It takes a delegate and the delegate basically is supposed to implement a draw method. So it wires up an image to work quite a bit more like a view when the image is asked to draw it's going to make this callback, say hey, do the drawing please. So it's a nice way to do problematic drawing in an image.

Okay. So the problem with flipped is not so much what it's for, as what people thing it's for. So people think it's the right way to do right side up drawing of images in flipped views, and that's not the case. So if you try to pull this sort of thing, if you say self image set flipped, self is flipped, the second one that's calling the view method is flipped, this is not going to do anything very good. It's very unreliable. If you're trying to use the image in more than one place, it's going to have weird side effects.

Okay, so let's see what happens when you do something like this. Suppose the images is originally uncached. I love this dog cow. So suppose that you brought it in from this bitmap image rep, okay? And you have this image. Now you call set flipped on the image. That's going to change the internal coordinate space.

If you think of that cache window I was talking about, it's basically if you're drawing into a flipped view instead of an unflipped view when it goes to cache. So when it goes to cache, it's going o take that bitmap image rep, it's going to call the very simple method that we saw on NS image rep, which is draw. That doesn't have anything to do with flippedness. There's nothing there. So the bitmap image rep is going to end up drawing upside down.

And what you get is you get an image which has a single cache image rep and the cache is going to have an upside down dog cow in it. Now the problem with this is that the image is now intrinsically upside down. So if you go ahead and try to draw the image upside down in a view, the two upside downs will cancel out and it will end up looking right side up. But if you try to pass it off to anything else in Cocoa, we're not going to be able to get it right because there's no way for us to know that this image is intrinsically upside down.

Okay. So again. The rule of thumb here is that you don't want to be modifying your images. And certainly something like calling set flipped, that's something you very, very rarely want to do. You would do that once if you're going to want to be drawing in it and it's more convenient for you to use its flipped coordinate space. I didn't even mention that. Pretty much the reason we have flipped coordinate space is that for certain kinds of drawing it's just much, much more convenient.

Okay. So try not to modify the images after their initial set up, except if you want to, like I said at certain times. But don't do it willy nilly. All right. But now if I'm saying that's not how you get an image to draw right side up in a flipped coordinate space, how should you do it?

Well, the easiest way to do it is to transform the coordinate system. This is actually not so hard. So I'm going to suggest, I'm going to present a category method added to NS image, which takes additional parameter, flippedness. So whether or not the image should be flipped before it's drawn.

And not I added a prefix to this method so that it doesn't possible conflict with anything AppKit might want to add in the future. So the overview of this method is we're going to transform the coordinate system, if necessary, so the positive is up. We're going to express the rectangle in the new destination coordinates. So now that that rectangle's coming in, again, it's always relative to a space.

It's relative to the ambient space at the time the method was drawn. If we transform the coordinate system, we also have to transform the rectangle so it still describes the same thing. It's the same thing that I've continued to talk about, which is what the rectangles are with respect to the coordinate space.

And then we have to draw and then we have to restore the coordinate system so we don't have side effects. So here is the complete method. I'm not going to do it in this slide. I pretty much just wanted to point out that it's pretty short. And if you're to do this, then you would never have problems with getting images upside down again probably, hopefully. Okay. So the first step, as I said, is we want to modify the coordinate system. I haven't actually talked about how you do that yet.

So up at the Cocoa level, the class that we have for this is called NS Affine transform. And the way it works is you have methods like this. You have scale X by, Y by, all right, and you pass negative one for the Y by. That means that you're going to be sending the current point, zero negative one, to the point zero one. The right way to think of that is it's modifying the axis' themselves.

Okay? The difference is that it's not modifying the things that you're drawing, it's modifying the axis'. Like if you want to shift all your drawing to the right, you need to move your axis' to the left. Okay. And the way we do that, the way we actually apply it to the graphics context is to use the concatenate method.

All right. Now this is what I was saying. So the original rectangle that was passed in as a destination with respect to the coordinate system that was active at that time. Since we have changed coordinate systems, we need to modify, we need to make ourselves a new rectangle that describes the same destination location in the new space. So how does that work?

So in the old coordinate system, it would be the way I set it up on this diagram. The space from the axis down, that was the old origin dot Y. and then the size of the rectangle was the old size, the height. Now the new rectangle, to describe that we need to pick out the origin. Well, where is the origin? It's at minus original rect dot origin dot Y and then minus original rect dot size dot height.

See, that's why you end up having to do these sort of transformations. And it's not a good idea to try to just remember like this is what I do. It's better to just try to think about the spaces, that way you won't make so many mistakes. All right.

And then we just draw, which is very straightforward. And then we need to put the context back the way it was. Now note that actually, in this case, I did not do the save and restore graphics context. Instead, I concatenated the X axis reflection one more time to put it back the way it was.

Oh, actually those two images are reversed. I concatenated that same transform to put it back the way it was. Why did I do that? Well, actually I did it for performance. So these days, the way Quartz happens to work, changing the context is relatively expensive when they switch on what's called Quartz GL, which uses open GL for more of the drawing. So you want to avoid modifying the context more than you have to. and saves and restores are relatively expensive next to things like concatenating a single transformation. So that's why, because it was easy in this case, I chose to just flip again.

Okay, now that is really the way you probably want to do this. Transforming the coordinate system is the right way to get your images drawing right side up. But just for variety I'm going to explain other methods and I'm going to show you how to do it using methods that are already on NS image, the composite functions.

So let's look at this. So you may have noticed, if you've looked at NS image dot H that there are these draw functions that we usually use, there are also these composite functions that are very, very similar looking. Composite two point from rect operation fraction versus draw at point from rect operation fraction. That's a little strange. These are almost the same declarations. So what's the difference? Well, the composite methods are kind of funny.

They're older and they actually ignore the current coordinate system, which is very odd. So what they're going to do is they're going to completely undo everything down to pixels. If we had a universal space it would be pixels, but like I said, the right way to think of it is to not think of it that way. You just always have spaces.

And then it's going to scale back up by the default scaling factor. So if you were trying to do something like tip your dog cow by rotating the coordinate system a little bit, then you're going to find that if you use the composite functions, that's not going to happen. Now this is actually pretty convenient though.

If you have a coordinate system that's flipped, where positive goes down the Y axis, well that won't effect the composite methods, because composite just always doesn't pay attention to the current transformation, to the current coordinate system, which is very convenient, as long as it's not actually wrong. If you were a framework, it would be bad to do this, because people are supposed to be able to change the way you draw.

I mean if somebody intentionally modifies the coordinate system, it should change. However, as an application developer you may know, hey, I'm not doing anything interesting at all. These views are always just sitting here just completely default an the drawing is just always going to be the same, in which case you can use composite methods if you want to.

Thee are other cases where it may be useful. An example of that is say you have a graphics drawing program, something with a canvas on it and say you have a graphics object, like in sketch, a square, and when you select it it has little grab handles, well you might want to be able to change the zoom factor of your canvas so that it gets bigger and smaller, but those grab handles, they would never change size. so a really convenient way to do that might be to use the composite methods, because they won't pay attention to the fact that the scale factor is changing.

Okay. That's all I have to say about NS image. So at this point I want to go through fairly briefly and talk about some of the Core Graphics level APIs that are available to you that are alternatives to NS images in some cases, and are compliments in some others. So first I want to talk about CG transparency layers.

These are distinct. Maybe I'll go back for a second. These are distinct from the other things we see on the slide. They're not the same as CG layer reps, if you've seen them. They're not the same as core animation layers. The word layer has been reused a few times. These are transparency layers. What are they for? Well I think the easiest way to see this is to look at an example with focus rings.

How do you draw focus rings in Cocoa? This is the question. Okay. Like this. We set something into the graphics context. We set a bit of state, which is the focus ring style. We say NS set focus ring style, focus ring style above. And then we draw, say we're trying to draw a button, okay, and say it's in three pieces of art There's a left cap, a right cap and middle part. We would draw the left piece, we'd draw the right piece, we'd draw the middle piece. All right.

the problem is that if you'd do that, the way the focus ring styles work is that each of those pieces drawn separately is going to get a separate focus ring applied to it. Also very interesting to look a this, from a users point of view you think of it as something that just floats above your views That's not the way it's implemented.

They're just pixels drawn like everything else, which means that you can't erase a focus ring, all you can do is draw over it. And if you've seen focus rings over drawn, well that's what's happening, they're drawing multiple times and the get darker. Yeah, but this is not so good. So how do we make this work? Well, actually at the Cocoa level, it's fairly hard.

But in Tiger, I believe, Core Graphics added this CG context being transparency layer with rect, and what this does is it makes sort of a temporary grouping. The way to think of this is that here you have your original context, your destination, and then you call this CG context being transparency layer, it makes sort of a new temporary buffer that you're going to draw into. You do all your drawing into that buffer and then when you end the transparency layer, everything you drew is going to be composited down, together. And that focus ring will be applied to all of them at the same time.

So the way that ends up working is that when you call the CG transparency layer begin, you get exactly the same graphics state as before, with a few changes. It's going to reset your compositing mode back to normal. It's going to reset your alpha back to 1.0. it's going to reset your style, that's things like focus rings and shadows, back to nothing at all. Okay, so that means that that allows you to draw without the focus ring to begin with, but then when you finish your drawing the entire layer gets the focus ring applied to it.

So I already mentioned that case. That's one. Another interesting case, so I kind of glossed over something before. When I was talking about using that additive compositing mode to try to draw three part flush regions together without pixel cracks, and I was saying hey, draw in additive mode.

Well, there's a little problem with that, because if you draw in additive mode, if there's anything already there that you're drawing on top of, it's not going to look right. so how do you do that? Well, you can do that with transparency layers. What you can do is the original compositing operation would be normal, you start this transparency layer. In the transparency layer you set the compositing operation to be the additive one.

You do all your drawing there, and then when you finish the layer, the entire layer, all of your drawing, gets composited down in the normal mode, which is what you were after. So it's very, very useful, it's very easy. It's very convenient. The only thing I want to mention about it that's really dangerous at all is that you see in this method that I had, it takes an additional argument, which is the rectangle, begin transparency layer with rect. This particular method that takes the additional rect is new in Leopard and what that rect is supposed to be is it's calling out the area within the transparency layer that you're actually going to draw in. so it's going to act kind of like an additional clip.

And if you don't use this new version of the method in Leopard, or if you don't set the clip to be fairly tight to begin with, this transparency layer will end up being very expensive to draw, and we've had some serious problems with that. So the new method should really alleviate those concerns. And if you have to work on Tiger, just make sure that the clipping rectangle is as tight as you can make it before doing this. Okay. That's all I have to say about transparency layers. Now let's talk about CG image a little bit.

So CG image is the primary image abstraction at the Core Graphics layer. It's not actually very similar to NS image. It's really quite a bit more like NS bitmap image rep. It has an explicit resolution. It has a number of pixels and a size. That's how it works. The main difference from NS bitmap image rep is that you cannot get at the data, all right?

Within a bitmap image rep you can always get the byte. You can always look at them. CG images are completely, well they're opaque type, which is to say that all you can do with them is draw them, you can ask for some certain limited kinds of information, but not the actual data. The reason that's done is it's done for efficiency.

That means that Core Graphics has lots and lots of latitude for how it wants to store this, when it wants to upload the data to the graphics card, when it wants to bring the data back from the graphics card, if that's something it needs to do. So it's very efficient, and that's what it's really for.

When do I suggest you use it? Well, if you're really sure what you're going for, if you have a very specific context, and you're finding NS image is kind of just too smart for you, because it's defeating what you're trying to do, you might want to look at CG image.

And also it's useful when you're dealing with APIs that already use CG images because otherwise there's an impedance mismatch and you're just constantly converting everything. But do be a little careful with it because if you do decide to use CG images, then you really have to think about a lot of extra things that you would not ordinarily, for example, resolution independence.

But these caching things that NS image does for you, they're really pretty nice and you should try to take advantage of them if you can. So don't just go down if you don't have to. all right. like I said, making them is easier than it used to be.

You can make one from a bitmap image rep and you can create a bitmap image rep from one. It's maybe worth noted that second method, init CG image, that's a very efficient method. It will take the CG image and will not do anything. It'll just hold on to it.

If you ask for the data that backs the NS bitmap image rep, at that point it will draw the CG image in order to create the data, but it will do that lazily. So you're not likely to do better than what we have done by writing it yourself.

You're welcome to write it yourself if you want, but you don't need to. and then the other thing, however, since they are completely immutable, people sometimes wonder well how do I make a new CG image? Okay. The way you do that is actually you create a CG bitmap context rep and you draw in that. And there's a method in CG image context, which can pull the CG image from it.

Okay. So let's talk about CG layer rep. okay, so this is one of those, I think, relatively unknown. It was added in Tiger in Core Graphics. It's similar to CG image, but it is mutable. So it works really a lot like NS image's lock focus and it's even the same in that the size that you pass into the CG layer rep is the same as the size when you do NS image init width size and then lock focus, and then it's going to be always scaled up by the default scale factor in terms of the number of pixels you end up getting.

It's easy to instantiate. You can see that I have the function you call right up here at the top. It takes a reference context. That context is not used for drawing, it's used to decide things like characteristics. That's how when you're drawing to the screen it's going to look at it and decide what a good pixel format is so that it can do that nice RGB LCD text rendering that we were looking at before. So how does this different from NS image rep?

well, NS image's lock focus? Well, you're actually drawing directly to a texture on the graphics card when you do this. And that's relatively exposed. So the main difference is that sometimes that original drawing can be faster if you're drawing to a CG layer rep. so if you have something that's already, if you have a CG image, and you're drawing a CG image into the CG layer, if the image has already been uploaded to the graphics card, that initial drawing can be very fast.

If you don't have to do things like that, and if it's not convenient for you for whatever reason, CG image is a little bit better, if you have the choice, if you don't need the mutability, because it gives Core Graphics more latitude to just do whatever it wants. And of course, you can stay up at the NS image level, if that's appropriate for you. CI image. This is perhaps a little bit more clear when it's appropriate. It's for image processing.

I think people probably have that idea at this point. But if you want to do things like apply Photoshop style filters, effects, complex compositing, that's what CI image is really good for. It is explicitly bitmap based, again. So you're going to have to worry about resolution dependence issues if you work with it.

How do you make them? Well, there are some new methods added to Leopard, NS bitmap image rep init width CI image. That one's new. Again, since CI images are explicitly bitmap based, it's more similar to NS bitmap image rep than it is to NS image itself. And the way you'd use this, by the way, oh, I haven't mentioned that yet.

So if you have an NS bitmap image rep and you want to make an image out of it, what do you do? You make an image and you add the representation. There's a method add representation. Okay, and then you can make a CI image from a bitmap image rep as well. And that one was already around in Tiger.

Last thing I want to talk about, Core Animation just a little bit. And it's actually not so much of a competitor with these others, but people might have questions if I didn't mention it. So Core Animation sort of acts like a merger of the image and view layers.

So in terms of performance, CA layer is pretty much like a CG image, but it has sort of an additional, all these smarts about invalidation and compositing of the images for you, they have intrinsic locations relative to each other, which is a little bit like the view hierarchy.

You can use it, I guess usually if you're using CA layers at the Cocoa layer probably all you're doing is you're sending the method set once layer yes, or checking the box in Interface Builder because that'll make it layer backed. But you can use it when it seems better to composite already created images, rather than redrawing all the time.

And that seems kind of funny because it seems like having big images with all those pixels that you have to composite, how can that possibly be faster? How can that possibly be faster than just redrawing small regions? Well the reason is because graphics cards are kind of crazy and they're really fast, and that's what it's good for.

So it turns out that if you cache these things to images than the graphics card can do a really good job of compositing them right at draw time. But of course its used for animation because when you have things moving around a lot, if you can avoid redrawing them and instead just composite the layers, that's great. So when might you want to use it explicitly instead of using the NS view stuff like I was mentioning?

Well, there are some features that are not exposed to the NS view layer. There's some automatic layout stuff that has not yet been brought up to Cocoa that you might be interested in. another thing is if you want to do explicit animation, what we have in Cocoa is we have the idea of animation in response to an action, so if you say set frame on a view, then it's going to animate from it's current location to the new location. Well what do you do if you just want to say I want this to start pulsing or I want it to start spinning right now? When should it stop?

When I tell you to stop. That's not something that's really exposed to the view layer right now, so you can go down to the Core Animation level to work with that sort of thing I you want. So that's pretty much what I want to say. So what are the big points from this talk?

It's mostly just a lot of little points, but if there are some big points, coordinate spaces. Keep track of them. Your rectangles are not just complete by themselves. You have to know with what coordinate space they're relative to. NS image, it does a lot for you, it has a lot of features, it's not so bad as I think it has a reputation for.

If you understand particularly the flippedness and if you understand the caching, I don't think you'll have very much trouble with it. And NS image rep also, it's really pretty easy if you want to extend NS image to cover more sorts of classes. And then the other APIs are also there if you, you might want to go down either for performance if you have sort of specialized needs, or just for interest.