Hi everybody. Welcome to Session 111. NSImage in Snow Leopard. My name is Ken Ferry, I'm an engineer in the Cocoa Frameworks Group. So, today we're going to talk about NSImage. Why is this? Because we've done a lot of work on it for Snow Leopard and we think you'll like what we did. So we'd like to tell you. We tried to make it simpler, more consistent in terms of behavior. We've tried to improve its performance.

And we've also improved the integration with other frameworks on the system, in particular the integration with Quartz is much tighter and with CGImage in particular, if you're aware of what that is, and if you're not, you know, we'll get to it. Okay. So the first part of the talk is going to be about the changes that we've made for 10.6.

In order to do that we're really going to have, we're going to need to talk about the Core Architecture of NSImage. So let's get going on that. One thing I want you to keep in mind, usually when you think of an image, you're going to think of something like this.

A photograph say, this is a photograph my father took. Pretty nice I think. However, for the purposes of this talk, we also want you to keep in mind images of this sort. These are, these are interface images that the user would not necessarily think of as images. Icons and such. The distinction might be the one, the images that you own, versus the images that your user owns. And it turns out that just these sorts of images have somewhat different issues associated with them.

So keep both in mind. All right. So, so if there's one thing that an image is good for its drawing. So, so let's just run down and, and see what happens when you ask an image to draw. What is the sequence of events, okay? So suppose you have something like this Pages icon, all right. The first thing to realize is that even as a user you might be aware that the Pages icon is available in multiple sizes, okay. There's a 16 x 16, a 32 x 32, 48, all the way Hup to 512 x 512. Different bitmaps associated with it.

And that's reflected in the API. So NSImage itself is actually a container for what we call NSImage Reps. And in this case, there's going to be one Rep for each of these different size bitmaps associated, okay. And we'll discuss NSImage Reps a little bit more later. Anyway, when you're asked to draw what happens? Well, you have a location that you're going to draw in. It's called NSGraphics Context. There's a rectangle in the Context.

We look at that rectangle, we select the best representation, and we draw it. Okay. We'll fill this page, we'll fill this picture out a little bit more later. But even that is enough to start talking about some differences. In 10.5, this was the name of the method that did the representation selection. In 10.6, it's a little bit different, it takes more arguments.

So in, there was just one argument in 10.5, Device, this was a dictionary. Okay. It describes something like a window, it would say like "This is 8 bit and it's RGB," something like that. In 10.6, we take these three arguments, a Rect, a Context, and Hints, which is a dictionary. Let's discuss what those are. All right.

Well, the Rect is the rectangle where the image is being asked to draw. The Context is this drawing location, it's this object that encompasses it. And it includes information such as what are, what is the relationship of the units that the Rect is in to pixels. That's information that's in the context.

Okay. And that's exactly the information that is required to pick the most appropriate size bitmap for that pages icon. Okay. Just for one example. There's other information that might be in there, such as whether it's gray. Maybe if you're printing, or one bit if you're faxing or something like that. So, and all of these things can play into which representation is most appropriate.

And the main thing to realize is just focus on this, oh, I didn't tell you what the Hints was. The Hints are sort of extra information that you can provide, it's optional. And it can also be used to override aspects that we would otherwise infer from the Context.

And, and I want to just point out, just ask you to look at this again because it's going to keep coming up. This Rect, Context, Hints triple is going to be key in all the new APIs. Okay. Anyway that's the information we need for Representation Selection. Just the dictionary before was not good enough, what would, so how did it work? Actually, it was using an internal method before in 10.5 for actual representation selection. Okay. So let's fill out that picture a little bit more.

Something else that NSImage will do when it's being asked to draw is it will cache. Okay, great. What's the point? The point is that if you're asking image to draw repeatedly to the same destination, then we can render this, this hopefully optimized object, which when you, which can be drawn down to context faster, repeatedly.

And the reason I wanted to bring this up is, you know, surprise, there are changes. In this, this caching business was, was a little bit confusing in 10.5 because there are a whole bunch of different methods that, that you can set on NSImage that had to do with caching. And unfortunately, they affected more than just performance. Some of them would affect even things such as where the image would end up drawing. How much of the screen it took up. And, and that's not really how we want caching to work.

In 10.6, hopefully, it will just be a matter of something you can't even tell if it's there unless you're using instruments to profile it. Or unless it's so slow that you usually can tell. So, and this is part of what I mean by, by making, making it simpler, and behave more the way you expect it to.

So these are all methods that are defined on NSImage that can help control caching and they're all deprecated. And their deprecated without replacement. And it's not that functionality has been removed, it's more like it's just not necessary anymore because we are doing a better job of invalidating the cache on our own and we don't need as much advice about how to do it.

Okay, so there's still more changes in caching. One change that is mostly an implementation detail, but it leaks out in a few places, is that in 10.5 and earlier we would cache to an offscreen window. Okay. It sort of makes sense in that if you're drawing to a window, maybe it makes sense for your cache to be some other window. Okay. Seems like it could make sense.

But, but there are a couple ways in which that's not optimal. It's kind of heavyweight. Windows turned out to be a scarce resource, sort of like file descriptors and as our apps get bigger and more complicated, the fact that you could run out of windows was a problem. And when you do run out of windows, it's fairly dire.

The system does not do very well. So, so that's hopefully not going to happen anymore. And also, it's just, it's just rather bigger than it needs to be. The CGImage which is a Quartz type is a very good representation for a Cache for us. So we're going to do that.

Okay. So there was a class that represented the drawing that was backed by a window NSCachedImageRep. Okay. And you see it has a method WindowinRect, that's the Rect within the window where the image is cached. So that whole thing has to be deprecated, which is fine. It still works, but we're not going to use it internally and we would suggest that you not use it in your own apps, you know, going forward.

Okay. Oh and, and the difference between the images is that they're, you'll see, but they're being used all the way throughout the, the app kit internals now in hopefully an optimal way. And, and part of that means is, is that we're going to be able to supply one of our most common feature requests for NSImage. Which is I want to take this NSImage and I want to get a CGImage out of it. For those that.

Okay. I was going to explain my, my, why you might want this. But apparently, you guys know. I'm still going to say. So why, why would you want this? Mostly because you're interacting with other APIs that deal in CGImages. That's the most common reason.

One of them for example is, but we're also taking care to try to make more APIs take NSImages, as you'll see in the talk that's appropriate. CALayer, for example, in 10.6 actually can directly take an NSImage as its contents. You don't need to get a CGImage for that.

But, that aside, we're still making it easier to extract CGImages. And we're also making it really easy to take a CGImage and represent it as an NSImage. That's the second image with CGImage Size. That wasn't superhard in Leopard. But that was certainly much easier to implement. But, nevertheless, it's extremely easy now and its colorful.

However, usually when people want to get the CGImage they want to say, they want a method called CGImage on NSImage, and this one takes three parameters. So that's not, that's not quite what people want. So what. Why did we do that? Well the problem is that a CGImage is just not capable of capturing all the information that's in an NSImage, okay. And so we need more, we need to know more to give you a CGImage. The CGImage is just a flat bitmap.

Okay. It's not going to be able to represent those, for example, in this case of the size, these, this pages icon, it's available in multiple sizes. It's not going to be able to rep, able to represent all three at the same time. Another example is that an NSImage can be backed by a PDF. Okay. Or a window, the way we were showing before. And, and CGImage is not going to be able to do that to preserve all of that information. So what it can do instead is, I've been calling it a snapshot.

Think of, it's just a flat bitmap, but, but, so I'm three-dimensional. Okay. And you can't represent me all the visuals that I encompass as a single, flat 2D image. However, if you fix the camera angle, that you're looking at me in, and fix a box that you're trying to look at, then you can produce a 2D image, which represents me from that angle, and is an accurate reproduction. And that's what the CGImage is going to be.

So when you pass a Rect and a Context, and this Hints thing, then you're going to be able to get a CGImage, which if you draw it in the same Rect as you passed, in that context, would be the same as the NSImage drawn in that Rect, in that Context.

Okay. So it's sort of like the image from an angle. Okay. So that's the CGImage. There is one sort of unfortunate little tweak to that that I, which is that the Rect that we pass in is not actually a Rect, it's a pointer to a Rect. Let's see why that is.

Okay, suppose that the place, the Rect that you're passing in is not pixel aligned in the Context, all right. So these lines are supposed to represent where pixel boundaries are. And this is the Rect you're actually drawing. And suppose the NSImage is actually just backed by a PDF. Like I said, NSImage can be vector based. So if it's just drawing, it's just going to draw.

You can render a PDF into a rectangle that has nothing to do with pixels and that's fine. However, if you want to get a CGImage for this, there's going to be a bit of a problem, because we can't give you something that's, that's some fractional number of pixels wide.

We would have to give you something that's, what we can do is we can give you something that covers this larger rectangle in the Context that, that is pixel integral, okay. And we can fill our own drawing into that and hand you back a CGImage, which is an appropriate snapshot of that larger Rect. So, so how does this end up working? It's, it sounds a little harder to explain I think than it is in practice.

But the deal is that if you pass this Rect to the method, and when it comes back, if you draw the CGImage in the Rect you get out, that's the same as if you'd drawn the NSImage in the Rect that you passed in. Okay.

Whatever. It works okay in practice I think.

All right. So, okay, so that's fine except you might say well, well that's great.

I understand why you need all these parameters, but, but sometimes I don't have a NSGraphicsContext. I need to make one of these CG, I need to get out one of these CGImages at a time when I'm not drawing. Do I expect you to make an NSGraphicsContext? No. These are acceptable parameters for all three arguments. Null, nil, nil. So that's perfectly straightforward.

And it's just that they each have a defined meaning. All right. What do they mean? Well Null is pretty much going to mean that you pass the most natural rectangle you could possibly draw the image in maybe, I don't know, if you think there is such a thing.

Which is the origin 00 and then the size of the image. And then there's this tweak business about pixel integral, which I'll explain in a second.

And nil for context means we're just going to assume it's as if you had made a window on the main display in the most natural way.

That's the kind of context it's going to be. And, and the reason we'd even say things like main display is that it impacts things like color space, because the different displays can have different color spaces. And hints nil, that's a little bit more obvious. I mean it means you didn't provide any extra information.

And, and now if I go back and what's this business about the pixel integral thing? It just means that the, that the CGImage that we're going to pass back to you will never have any of that padding on the outside like I was just saying. Instead of getting the padding, what you might get instead is that there might be some anti-aliasing artifacts because we might have had to inflate the Rect a little bit from what you've passed us. And, anyway, regardless, that's what that is. Okay. And, and I really did just want to emphasize this contract. Because it's the one really important thing about the API, okay.

Drawing the CGImage in the Rect you get out is the same as drawing the NSImage in Rect you pass in. And you know nothing else about the CGImage. So for example, suppose you have an NSImage, which was natively just backed by a single CGImage. Okay. That really is, all of its data is encompassed in just a single CGImage.

Well then, that CGImage is always going to be a valid return from this method, no matter what Rect, no matter what context you pass in. Because it does entirely encompass all of the information of the NSImage. So in particular, you know, even if you pass a 3 x 3, you might get back a 27 x 27 CGImage. You can't tell.

But this is what you can tell:, that it does capture the drawing of the NSImage. Okay. So all of this talk of snapshoting and caching and rendering PDFs in business. Is this starting to sound expensive? I don't know, maybe. But, I'd like to say that it's not. That it's only going to be doing work in the cases where you would want the work done. And a lot of that is because I haven't showed you all the API yet.

There's also the same method has been added down at the Image Rect level, CGImageForProposedRect:context:hints. All right. And what is this going to do? Well to do that let, let's go ahead and now talk about exactly what NSImageRep is. So NSImageRep, the really distinction in NSImageRep then is image, that NSImageRep is built for subclassing.

Very, very similar to NSView. Okay. In the AppKit, we have a bunch of different classes, and each of them represents the, the way it works is you just override the draw method. And you produce the drawing, whatever it is. So in the case of a BitmapImageRep, it's going to draw the bitmap. In the case of a PDFImageRep, we override draw, we draw the PDF. In the case of NSCIImageRep, that's like a CoreImage. We override Draw, and we'll just draw that.

This CacheImageRep, I told you that that represents drawing back via a window and it's deprecated. But, but it's still there. So okay. Yeah and it's, it's there to be a subclass. If you think about NSView, most people have a subclass, have NSView, or at least they're aware it's pretty easy. The one method you really have to override is DrawRect, and in that method, you're drawing in to the view's bounds. That's the location of you're feeling, filling. NSImageRep is the same way. You, except you override draw, that's the only method you really have to.

And you, the rectangular you're filling is 00 Image Size. That's it, okay. That's all you really have to do. And it's worth pointing out because NSImageRep doesn't look like it's that easy to subclass. It has all these other methods that you look at and you say I don't know how to implement these. Things like colorSpaceName, bitsPerSample, pixelsWide, pixelsHigh. Okay, well, the thing to understand about those is that they're used for representation selection.

Okay. These are best effort methods. When you say colorSpaceName, what you're saying is this is a good color space for me. All right. Or with the pixelsWide it's saying if you're drawing into something that's this many pixels wide, please choose me. Okay. Which means, and if you don't, so that's interesting both from a subclassing point of view, and also as a caller's point of view, to realize that despite the names, these aren't, you don't, shouldn't necessarily believe these things. Because they're only best effort.

Though in the case of something like bitmapImageRep, yeah, it's definitely in our contract that pixelsWide has to be the actual number of pixels in the bitmap. Okay. And, yeah and they're optional. So you don't have to do it. Later in the talk I'm going to show you an example of subclassing in NSImageRep. You'll see how easy it is. Okay, but the whole point of this is I want to talk about this other new method we added, the CGImageForProposedRect:context:hints.