Welcome to the final day of WWDC. So, kicking the graphics track off this morning with Session 211, which is Introduction to Quartz Services. And the primary focus of this session is going to be the non-drawing APIs that are part of the Quartz architecture. These APIs do things like manage the displays, even facilitate doing certain common operations that developers need to do, such as what we call screen scraping, or pulling pixels back out of the framebuffer.

And the key thing is that the important part of the session is that these APIs that we're going to be talking about in the session are essentially the APIs that lie underneath other APIs to do display management that you might be familiar with, like some QuickTime APIs, DrawSprocket APIs, and Display Manager APIs.

And one of the reasons we put this session together is because we really want developers to migrate to the Display Services APIs, because that's really where all the work actually happens. And you're getting closer to the native API and getting more functionality in doing it that way. So, it's my pleasure to invite Mike Paquette, Graphics and Imaging Engineer, to the stage to take you through the presentation.

Good morning! I'm Mike Paquette. I'm the person in the Quartz team who takes care of all the non-drawing parts of Quartz. I'm the guy working way down there in the bowels of the system, fixing the plumbing, wrapping in a little duct tape here and there as your requests come in, implementing all those nice features you need. And I'll be going over Quartz Services.

Quartz Services. What can Quartz Services do for you? We're the non-drawing part of Quartz. Yeah, there's code that doesn't draw in there. That's a lot, but eh. We handle all the display management stuff. We give you the functionality you need to take over a display to do a presentation.

We let you reconfigure the displays. We can let you position displays on your desktop. We can let you adjust the resolution of the displays, change the refresh rate, do all those fun little tweaks. The Display Preferences panel is an example of someone who uses our services. We also handle some special purpose tasks. We give you the basics you need to read the framebuffer contents off, that screen scraping thing. We also have support for doing remote control of the computer.

programs like Apple Remote Desktop and Timbuk2 can take advantage of this stuff. Now, most apps don't need Quartz Services, this low-level stuff. Most apps will run just fine and don't even need to know this stuff exists. Where does this fit in in Quartz? We're the display management code. We're your screen scraping support, and we're event injection. Event injection's a fancy way of saying, take an event and stick it into the event chain and let it dribble on down to all the applications.

We're going to give you full-screen application support information here and display configuration information, and we'll go over a few of the more obscure APIs that are in there. Some just help you out with backward compatibility and some to help you do some neat things that you might not have been able to do very easily before. We're also going to cover the Remote Operation Services. That's the screen scraping part.

I've got some sample code for you. We'll walk you through a little bit of tricks and techniques there and show you how to post some events for remote control purposes. And also give you guys who are writing user-based device drivers some tips on how to make that work right.

Now, we've got some definitions. We use some funny terms in Quartz sometimes. Display ID. Well, it's probably something that identifies a display, huh? You figured that much out. Okay, display IDs are numbers that go along with displays. They're 32-bit-ish kind of things. There's a typedef in there for them. The numbers stay the same for any specific exact hardware configuration, even across reboots.

Any changes in the hardware configuration, you unplug a display and replace it with a different model, you move the display to a different connector on the back of the machine, you're going to get a different display ID. Now, all these displays that we hook up to the machine, you'll query in Quartz Services using that display ID. That's how you identify them to Quartz, and that's how Quartz tells you about the displays.

Display States. We have a bunch of different states monitors can be in. A bunch of states the framebuffers can be in. We combine all that together and just refer to this as a display state. States include active, and active display is one you can draw on. That's pretty straightforward. It's there and it's nice and drawable. We can have displays that go to sleep. You're familiar with that. It goes blank. One of the energy star modes usually kicks in on a monitor. It gets all nice and quiet.

Don't draw on asleep displays. Now, again, if you're an ordinary application, you don't need to worry about that. It's all covered for you. The only time you really need to be concerned about these is if you're writing something that's drawing directly to the screen itself through one of our fancy APIs we're going to get into here.

Mirroring. Oh, mirroring has changed so much over the years. We've introduced things like hardware mirroring, and we've got software mirroring. We have hardware-generated display maps and all sorts of other wacky things. I'll be getting into all those different states in detail and giving you some tips on how to deal with that.

We have online displays. An online display is pretty much a monitor plugged into the computer. It may be awake or asleep, it may be mirrored or unmirrored, but it's connected. Offline? Well, that's the connector on the back of your computer with no monitor there. It's just offline, all alone, a placeholder in the system. Don't try to talk to the system about offline monitors. They aren't there. You'll see all through our programming interfaces this CG prefix. CG is, of course, as is intuitively obvious to the most casual observer, the prefix for Quartz.

Now, the Display Services part of the system is going to cover display enumeration, It's going to tell you that is, what are all the displays? Give me a list, please. It's going to cover display configuration. Place this display here, change this one's resolution, change the refresh rate, change the bit depth. We're going to cover Palette Display Support. Now there's something you haven't heard about in a while. Palettes. It's so 70s. And we've got some new goodies for Panther, our Display Fade API.

Competitive technologies. By competitive, I don't mean that, oh, it's something those people up north are building. What I mean is other technologies inside Mac OS X that do a lot of the same functions that you see in this code. There's the Sprockets stuff. Draw Sprockets has a lot of display management goodies in it.

QuickTime has a few obscure APIs in the corners for handling displays, as does the AGL, Apple Graphics Library package. And of course, there's our old buddy, Display Manager. Now, these APIs are present in Mac OS X. Why are they in Mac OS X? Oh, so you could bring your apps forward to assist you in porting. They're there so that those CFM, PEF-based apps, that you want to run on 9 and 10 can run.

You can use CG Direct Display, these new APIs, instead. Those older functions on Mac OS X are all built on top of the CG Direct Display APIs. So you don't really need to put up with those things if they don't do exactly what you want. If you cut right down the CG Direct Display APIs, you can get some finer control over the system. You might be able to make it do just what you want. I'll give you some tips on how to do that. Clients of these display services: Games! We like games. Full-screen applications, like Keynote. Configuration Tools, Display Preferences, your own apps.

New functionality that we have. Mirroring. I told you about mirroring before, right? Mirroring. Software mirroring is something that pretty much everyone is used to. We all remember writing in those good old loops. Run through the G device list. Find those displays. Figure out which ones intersect our window and draw on all of them over and over and over again. Well, you know, you don't actually have to do that in a Carbon or Cocoa app. That's covered for you. The Quartz Compositor does that work.

But if you are talking directly to the displays yourselves, you still get to do that, which might be one good reason to think about sticking with Windows systems and not bothering to go right to the display. Something to keep in mind. If we're doing software mirroring, software mirroring will happen on all these machines with different graphics cards, monitor on each graphics card, or if displays are in different depths, Software mirroring you have to go through if you are talking right to the display and draw on each display in turn, going down the line. Hardware mirroring is something that's shown up just in the past few years. Hardware mirroring happens when all those displays are running at the same depth and they're attached to the same controller.

You'll see hardware mirroring when you connect up external monitors to the newer iBooks, to the shiny PowerBooks, to those funny new graphics cards that support two monitors. Hardware mirroring. One display is the active display, the primary display. Keyword. The other displays are mirrored in hardware. You don't need to draw on those. So you just draw on the active or the primary display in a hardware mirrored set. We have more programming interfaces to tell you which ones are the primary, which ones are active, and which ones are hardware mirrors.

We also had hot plugging in the system. I'm going to get into that in some more detail in a little while. Hot plugging turns mirroring on and off automatically in some cases. In other cases, you get extended desktops. In either case, if you're going to do full-screen access using what we call display capture, you need to worry about all these things.

Display capture is something really cool. It lets you steal the display away from the Windows system. It's yours. You got it. You have lots of control that way. But, you know what? The Windows system doesn't know about it. So anything that normally goes into a window isn't going to work there. Like views, HI views, NS views, controls, buttons, all those fancy high-level widgets. They don't work on capture displays because you can't put a window on that. You've just taken over that hardware for your own purposes.

Now, when you do take over the hardware like that, though, there are advantages. OpenGL, for example, can put that display into some very unusual modes of operation that are extremely fast. Game developers have probably noticed this already, huh? So you've got more control, but less high-level functionality. In particular, because you're not going through the Windows system, you don't have that view hierarchy and all those other goodies there.

All those accessibility APIs that are going to be covered in another session today don't work. So that's something to bear in mind. Captured displays are normally used by games. That's a big use of them, particularly OpenGL games. You like to take over the display and go as fast as you can.

There's another way to do full-screen access, and this will be familiar to lots of folks coming from the Mac OS 9 world, Carbon developers, Hide Menu. If you call Carbon's Hide Menu function, the menu bar disappears, the dock goes off the screen, it looks big and empty, you then create a full-screen window and put it up there, and you draw through a window. Now, your app is still running in the Windows system. This is used by some other presentation apps, such as PowerPoint, to do its full-screen thing.

It's also used by a number of games. This is the way a lot of the games that are based on the PEF CFM technology that run on both Mac OS 9 and Mac OS X work, because this sort of functionality is available on both. You are going through the Windows system to do this. You're doing a full-screen window. That has advantages for accessibility purposes. It has disadvantages in that it might not be as fast as, for example, running OpenGL in a full-screen mode.

Display Capture. We've got functions for you. CG Capture Display. Grab one display. Take control. The display is removed from the Windows system. The app that calls this is forced to the foreground. You wouldn't believe how bad it is to leave somebody taking over an entire display in the background so the events go to some other app. It's really hard to recover from that situation. So if you capture a display, we're going to make sure you're the foreground app. You'll be pulled to the foreground. It's your responsibility then to process events and respond.

Once you've captured a display, you can draw directly to that screen using GL, Quartz, or other techniques. You can get a port to it.

If you've captured the displays, you release them to go bring up some UI elements, you can do that, and then you capture them again, it will apply the last captured mode to the displays again, so you'll be right back in the last mode of operation you were running full-screen.

After you're done using a display, you might want to release it. You can do this in the middle of your program just so that you can go ahead and use those UI elements, bring up some control panels for setting up game options, that sort of thing. CG Display Release releases one display, lets it back into the Windows system. CG Release All Displays is really handy. This releases all the displays. Okay, that was obvious.

It also restores all the display modes to the ones the user last set up in their preferences. It puts the gamma tables back the way color sync thinks they should be. And you can use it after a display capture or a capture all displays. One function, boom, puts everything back the way it was.

You don't have to keep notes yourself of what the display layout used to be. You don't have to go through, "Oh, let's see. I think this one was over here. I think this one might have been up here." None of that stuff. Just call one function, boom. You're all set.

drawing to a capture display. We have a couple of new functions, new for Panther. CG Display Get Drawing Context. This gets you a Quartz drawing context for a display. The context is owned by the framework. Notice it doesn't say create there, so don't release it. OpenGL. You can use OpenGL. Pick a pixel format. Set up a full-screen GL context, those CG GL APIs.

When you're done, destroy your context, put everything back the way it was. OpenGL is usually going to give you the best overall graphics performance. If you've been going to any of the GL sessions you've seen, you can use it to do 2D animations, you can use it to blit images to the screen, you can use it to do all sorts of interesting special effects. GL is pretty cool stuff. And even better, you can now get a CG context from your OpenGL context to get an accelerated context. That's been covered in a few sessions here, too.

Some people might need to change their display configuration. If you're playing a game and, oh, you're on one of those nice big cinema displays, well, you probably didn't do all your artwork for that 1280x900 or whatever that thing is. You probably did a lot of artwork for, oh, 640x480.

So you've got two choices. You can do a little bitty square in the middle of the display, or you can change the display mode and fill the screen. The Display Configuration API lets you do this. It's a transactional model. You tell the system, I want to begin a display configuration.

I want to accumulate a series of operations. I want to set a display mode. I want to set a position. I want to set a refresh rate. All these goodies. And then, once you've accumulated this new state that you want all the displays in, you apply it with one function call, and the system will then go through and do whatever's needed to get there. Now, the system has constraints built into it. We don't let you put gaps between your displays. You can lose your mouse cursor that way. We don't allow overlapping displays that can look kind of odd, be very confusing to a user.

We don't allow you to make up your own display modes. You go, "Oh, let's see, 42's a good number. "I'll use that one. "I wonder what that does." No, you can't do that. The system's going to reject these sort of things. If you're on, for instance, an iBook that's doing mirroring of displays, The hardware doesn't handle not mirroring the displays, so if you try to do that, the system's going to say, no, no, I don't do that. And you can't un-mirror. So.

We have these constraints that get applied. One other constraint that gets applied is it's going to look at all the displays you're setting. The one that shows up at 00 for its origin point becomes the main display. That's the one that gets the menu bar and the login window.

If you don't set any displays to 00, we get to pick one. So we'll just do it ourselves. CG-begin display configuration is how you get the whole process rolling. This is going to return to you a CG display config ref in a pointer. This little ref thing is what you're going to pass to all the other configuration functions. It's going to be the accumulator for all the state you want to build up, and it's going to be what you use to apply the state change.

You're going to call CG-complete display configuration to complete the display configuration, or if you've done all this work and decided, that's a hassle, I don't want to do this anymore, you can throw it away with cancel display configuration. To set the display origin, you use the mysteriously named function CGConfigureDisplayOrigin. This sets the origin point, the upper left corner of that display. You can move them in X, you can move them in Y. Not in Z.

and his team are working on a new application called Display Mode. CodeWise looks like a CFDictionaryRef. A display mode describes a particular display resolution, a particular refresh rate, a particular bit depth, and some options. Display modes come from a couple of different APIs that we have. We have APIs that will let you is the Chief Executive Officer of the CgConfigure Display Mode.

He's been working on the development of the CgConfigure Display Mode for a long time. He's been working on the development of the CgConfigure Display Mode for a long time. He's been working on the development of the CgConfigure Display Mode for a long time. He's been working on the development of the CgConfigure Display Mode for a long time.

You can also use our little chooser. CD Display Best Mode for Parameters and a set of related functions with really long names, longer than that even, can be used to pick a mode. These are sets of functions which take a desired width, height, pixel depth, refresh rate, and some other options that you can feed in.

They also have some options that let you specify, "I want only modes that are known safe for this monitor and graphics card." So you don't have to bother putting up that confirmation panel. That can be handy. There are options to pick stretched modes to fill in gaps on the sides of your monitor.

You can turn on mirroring. Now, some interesting things that happen here. When you mirror a display and another display, the system is going to do some interesting things. It's going to try for hardware mirroring. So if you're not explicitly setting the mode of some displays that you're putting into a mirror set, the system is going to auto-configure those to match in resolution and to match in depth so that it can use that hardware mirroring I mentioned. So if you don't explicitly set the mode of a display, we will try to match them all up to avoid those matting borders. What's a matting border? You might have seen these. We get these nifty widescreen gadgets.

So when you go to set up displays as a mirror, the system is going to try and change the modes of some of the displays in the set to match the smallest display in the set, to match the depth of the displays together, so we can use our hardware mirroring tricks. Now, unmirroring, some displays cannot be unmirrored. For example, the iBooks, that secondary display port is only for mirroring, so you really can't break them apart. The hardware doesn't handle that.

One other thing, if you look in the display mode dictionary, you might see numbers in there for things like the display row bytes. That's sort of a value taken in isolation. When you mirror displays together, the robot's values for the individual modes may change because the device driver and the hardware might require certain amounts of padding.

If we do matte generation, we may do that matte generation in hardware. So the only, the video portion that's live is the only piece of framebuffer memory that actually exists. Those black bands are just an artifact of the video system. They don't actually exist in the framebuffer. So things like row bytes may change after display configuration where you set mirroring setup.

Once you've set up your display modes, your display positions, you've got your mirroring set up the way you want it, you then call CGComplete Display Configuration. That takes that ref, pulls all the data in, grinds on it for a while, applies some of the system constraints, and applies it. It then releases that configuration ref.

There's a couple of different options. Well, there's three different options, actually, this takes. Configure for app only. This is a really handy little mechanism. If you tell the system, I want this configuration just to apply to my app, what that tells the system is, this is not a permanent configuration. This program wants it.

It might be a presentation app like this. It might be a game. It wants it in its particular display mode. If something bad happens, and that game ceases to function for some reason, say it fails, quits, escapes, dies, blows up, poof, The system says, oh, well, that was a mode only for that app. I better go back to the user's preferences. And it restores everything for you. So, if you're just setting things up for your own app's purposes, please use the Configure for App Only option. That gives us this auto recovery mechanism.

One of the other cool things is if you tell the system this configuration is for an app only, if you need to get back to the user's preferences, say, at the end of the game, you can use CG restore permanent display configuration. And that's a long one. That'll put things back the way they were. That restores the permanent display configuration. Caught you by surprise, didn't it? Now, configure for session is kind of interesting. This applies a configuration change that lasts while that user's logged in, but

and configure permanently it.

Anyone? Anyone? Bueller? It configures it permanently! writes it in Display Preferences and it comes back the next time the user logs in or the machine is rebooted. Interesting tidbit for you: if you're an administrative user, and you do a configure permanently. That also becomes the boot configuration for the machine.

The configuration is adjusted by the system after you feed it in. So, you should, after setting the configuration, I really do. Did you get what you expected? Did it do some tweaks on you that you didn't realize? We actually have functions in the system that you can use to read back the configuration. You can read back the origin point for a display. You can read back its width and height. You can read back all sorts of interesting things.

You can get the current display mode out of the system and see, "Oh, is this the mode I access for? That's pretty close." In addition, we also have some other APIs you might find handy. You can ask for Displays Vendor

You can ask for the display resolution. That's the width and height. You can get the origin point. One interesting thing is, CG display uses OpenGL acceleration. That's our little way of saying, you got Quartz Extreme.

CG Display I/O Service Port. Yeah, what is that? That's an I/O Service Port. Yeah, right. An I/O Service Port is a little representation that you can pass into I/O Kit. I/O Kit can then look at that and say, "Oh yeah, This display, and you can read back all sorts of interesting, really low-level things from I/O Kit, things like EDIT data. And everyone's eye just glazed over. Next slide.

We have display change notifications you can subscribe to. Now, quick draw. pays attention to these. It does its own thing and reconfigures. Quikter also has the implementation for Display Manager, by the way. Many of you might be calling Display Manager to get display configuration state changes. That works. Guess what it's built on? It's Friday morning, isn't it? It's built on this stuff. CG Display Register Reconfiguration Callback.

That you can pass a function into, it registers it, and it calls you back when something happens. In fact, it calls you back when something's about to happen. Is that cool or what? We know in advance, something's going to happen. The callback is invoked for each display. We'll call it back with this display, with that display, with the other display, one at a time, with a flag telling you, "Something's about to happen." And then, after something happens, displays are reconfigured, it gets called again for each display. This display, its origin changed. This display, its mode changed. This display, it's a mirror of this display. And this display was removed from the system. What? We support hot plugging of displays now.

Yeah, you got that ADC or DVI connector on the back of a PowerBook? You plug a display in that? You don't have to close the lid. You don't have to put that thing to sleep first. We noticed that, oh, look, something changed in the hardware. Quick, reorganize the system.

We reorganize the system, we bring up the new display states, and you're good. And this you have to be prepared for. Those darn users can just take that connector and pull it right out while your machine's running. Displays can vanish on you. Boy, if that doesn't mess up a game.

So you have to be aware that these state changes can happen at any time. It can be really shocking for you. We do have some cool tricks, by the way. If you've captured displays, if you captured the display, Hot plugging and unplugging isn't going to perturb your system that much.

You capture the display you're running on the built-in display in a PowerBook. Somebody plugs in another display, disconnects it. will note that, oh look, something's connected. We have to change state. After he's done, after he's let that display go, we'll change state. So we hold off for a little while. That keeps your game code from exploding on you. It's a good thing.

Now, when somebody actually hot plugs that monitor into a system, fun stuff happens. I/O Kit gets pinged by the hardware saying, "Hey, look, something changed." The device drivers go rattling along, "Oh, this one! Hey, look, there's a monitor on it now. I had one before." and the hardware and the software will work together to pull the monitor and find out, oh, what kind of monitor are you? Okay, we're one of these, huh? And you run in watch display modes? Okay, I'll just pick one. And it fires up the monitor and puts it into some mode of operation. Usually a mode of operation that's marked as a default state for that monitor.

At this point, we send notifications out to all the applications saying, "Oh, look, something changed. "You've got a new monitor." and your application can do something with that or just ignore it. Most apps will just ignore it. It's not a big deal. If you're a Carbon or Cocoa app, you don't care. It's no big deal.

If you're some kind of full-screen presentation app, you might be interested in this. You might be able to do something special with that extra display. Ah, the one that's going to get you, though, is when they unplug the display. What if you were doing something on that, too? What if you thought you had it captured, but you didn't? If you actually capture the display and it gets unplugged, Well, that's not so good. If you're capturing displays, you might want to think about keeping your UI on a display that's marked as the built-in display. And yeah, there's API that tells you, CG is display built-in. True. Okay. So you can check for that.

The old way of fading a display was grab the gamma table, stuff some values in, okay, that's pretty good, take those values up, put some slightly smaller values in, okay, keep doing that over and over and over again, hundreds and hundreds of times, burning up CPU cycles until you finally fade the screen out to black or whatever you were going to fade to.

Well, that has a couple of bad effects. First, you burn a lot of CPU cycles. And let's face it, monitors have refresh periods. They aren't going to see that change but 70 or 80 times a second. You can do it thousands of times a second. It doesn't help. It's CPU intensive. You lose your color sync data. Yeah, those color sync guys went to a lot of trouble to calibrate those displays, and now look what you've done. So, we have a new API, new for Panther! Boy, I can't wake him up at all.

The Display Fade APIs lets you tell the system, "I want to do a fade operation now. Please reserve the hardware and do a nice smooth transition." It's driven off the hardware and off of some fancy timers. It does a really nice job of fading displays in and out for you to handle all those transitions. You can use this instead of doing the Gamma Table tricks on your own. Very easy to use. Built-in. Now, we already have a fade effect built-in for display mode changes.

Yeah, we do. Yeah. Haven't you seen that fade to blue thing that happens? That's the fade transition. 0.3 seconds fade out, 0.5 fade in. You can override it when you're doing a configuration for that configuration operation only by calling CGConfigureDisplayFadeEffect. This wonderful function takes the config ref that you're using, the time to fade out, the time to fade back in, and the color to use for the fade transition. The default that's built into the system will fade to that lovely shade of French blue when you do a display change.

Now, there's other places where people like to do phase. If you're in a game, you've been running some sort of cinematic piece and you want to do a transition over to, say, your GL-based animations. Often people will do a fade out, perhaps a hardware mode change, bring up their full-screen context, and do a fade in. We can automate all this for you.

You can acquire the hardware to do the fade operation by calling acquire fade reservation. When you're done, you can release the display fade reservation. The reservation is good for up to 15 seconds. If it takes longer than 15 seconds to your fade transition, I'd think about that. That's a long time. Users get impatient.

Now, you can do the fade synchronously or asynchronously. You can tell the system, "I want to do a fade. I want to start with a normal display. I want to end up with a solid color, and I want the color to have this RGB value." This is just in the display color space. 0, 0, 0 would happen to be black.

and True. Yes, I want this to be a synchronous operation. You'll call this, and two seconds later, this call will return, and the display will then fade into black. Very straightforward. You can then do some setup operations. For instance, stop your cinematic, start your GL sequence running. You might want to fade back in while you're already rendering. In that case, you want to start the fade in and then go right to doing your animations. Let the fade proceed on its own, not worry about it.

To do that, you'd start off with, for instance, we're starting from a solid color in this case. We want to end up with the display looking normal. The starting color was black, so we're doing a fade in from black. False, meaning asynchronous. In this case, CG display fade returns immediately.

The fade occurs over the next two seconds, which is what we specified. While you get your GL animations up and running, so you do a nice fade in, your game figures are already running around, busily stabbing each other. Just having a grand old time. This can actually look pretty slick. And this is easier than trying to do a lot of the async callback stuff yourself.

All the actual callback and work to drive this is handled outside of your process in the fade machinery we've added to the system. Palette of Displays, whoa. Here we go. Who here uses Palette of Displays in their new app they're coding up? Good. So I got some news for you. This stuff's in the system for legacy support.

There's minimal to no hardware acceleration available when you're using 8-bit palettes. All that fancy hardware just loves 32-bit mode. It goes fast then. You go down to 8-bit mode and it's, "What? I've got these huge honking buses and you want me to move 8 bits at a time?" It's got a low precision.

It doesn't composite very well at all. It's barely supported by the hardware. Come on, guys! We're past the 70s. We're past the 80s even. We can get better. The configuration functions will let you select an 8-bit mode. They won't let you make it a permanent setting. There is a reason for that.

"Anyone? No? All right. We do have palette functions for you to play with. This is for legacy support, by the way. I don't know if I mentioned that." You can create a palette. You can create a palette from the display's contents. You can create a palette with an array of samples you provide. You can get the color at an index. You can set the color at an index. You can tell a display, "Here's your palette." And you can release a palette. Any questions? No good. Common developer issues.

What sort of issues could developers possibly have with this stuff? Well, there's recognizing those mirroring and display configuration changes. If you're playing with the display directly, You need to be aware of mirroring. That old device loop comes back to haunt you. You're going to need to look at all the displays in the mirroring set and make sure you draw on all of them in turn. is the Director of the Center for Data and Information Technology at the University of Most of the newer hardware will be doing hardware mirroring. That's what you're going to get in all the nice shiny PowerBooks, in the newer iBooks, the white iBooks.

That's what you're going to get with those funny cards that have two monitor connectors on a single card. Those will normally do hard remembering unless you work really hard to make it not do it by setting the displays to different depths explicitly. Don't do that. Unless you really like to work at things. If you're capturing displays, you must be aware of the mirroring state to get the correct behavior.

Capturing displays and changing the display configuration. Some people have been capturing the displays just so they could change the modes and then releasing them. That looks funny. The entire screen goes black for a couple of seconds. Your app comes to the foreground even though you thought you were doing this from some faceless UI element in the background. Looks bad. Don't do that. Please. Please. There is no need to capture the display in order to reconfigure them.

Use CG Capture All Displays to avoid funny desktop layout effects. If you capture all the displays, there's no desktop showing. And we won't bother telling things on the desktop that, "Oh, yeah, the display's changed size. Quick, cram all your icons into one corner." We can stop that behavior. We can do it, people. We can work together. Capture all the displays. Please. That hides display mode changes from the rest of the system. Only your app needs to worry about them in that case.

Display Changing. Let's take a look over on Demo Machine 1 and see what this is all about. I mentioned we had some APIs that let us actually look at the display state.

New for Panther! We have a new function that returns you the display width and height in millimeters. No, this is not actually 310 millimeters. This one here is.

This number here is the display ID for this particular display. I mentioned you can get information about particular models. This is vendor ID 610. I wonder who that could be. And product ID 9215.

Fuck yeah, there is a mode for 640, 480. How about if I go for, say, a... Oh, that's close. 800 by 600 is the nearest mode that would enclose that size.

1024 by 768 is the nearest mode that would enclose this size. Oh, look, we can get to 8-bit. Let's see what happens if we apply it.

Now, some displays have stretched modes they can support. Suppose I ask for 800 by 600 on this one. Yep, this display's nearest stretched mode is 10/24/7/68, so it would take that and try to stretch things. Safe Modes. Ah, we have a safe mode. This is all the modes that are known to function for this hardware and the graphics card that's in here. is the director of the Office of Software Engineering at the University of Michigan.

To get the information about a display, we needed to get a display ID. Let's get the main display ID in this case. We can feed in things. This returns us the physical bounds in millimeters for the display. This returns us the pixel bounds. For that display, we can ask for the current mode. That returns us that CFDictionaryRef, and we can pull information out of that.

Anything in a CFDictionaryRef is accessed by key. In this case, the values that you'll get back will all be CFNumberRef. and you can use that to fetch out the value as an integer or as a double, whatever's most convenient for you. The CFNumberRef will do the right things as far as converting those. Other APIs to let you get the vendor ID, the model number, very straightforward stuff. Searching for a display mode, this is a rough one.

It takes a display ID, a desired width, a desired height, depth, a desired refresh rate, An optional property value, you can just feed in null for the property by default, or if you're looking for only safe displays, so you don't have to put up any kind of confirmation dialogue, you can feed in, KCG Display Mode is safe for hardware. And it returns you a mode. The function will always return a mode.

If it can't figure out what on earth you're asking for, it'll hand back the current display mode and say, "I don't know. Use this. It works right now." tries for a depth-first match on those. Now, when we change the configuration, I told you this is rough stuff. We found a mode with that best mode for stuff. Begin configuration, we get back our configuration handle, configure the display mode for the main display to that mode. These modes are only valid for the display you queried them on and you don't get to make up your own.

The mode will only work if it's returned by one of the APIs that vends modes or lists of modes. And once we've got things set up, all we're doing is changing the mode. In this case, we call CG Complete Display Configuration. Configure for app only, so if something bad happens and the app crashes, we get back.

Capturing all the displays. We have a new for Panther item. You can specify an option, capture with no fill. That actually captures the display but leaves the contents as you see them. You can do your fade transition. It looks like you're fading out from the desktop, really slick, and go into your app.

I'm going to show you an example of capturing a display, doing a fade-out, we're going to draw a color bar's pattern, and then we'll do that asynchronous fade-back in. One of the interesting things, by the way, is you can start a fade-back in and release your reservation token if the fade-back in ends with a normal display.

If it ends with a normal display state, then releasing that token will have no effect on the display's appearance. The system will automatically say, "Okay, this is a deferred release. I'll release it once this fade-in is done." And once we're done, we're going to use Release All Displays to release the displays.

Let's capture the display. You'll see we're going to capture it, but not actually fill it. And we'll do a fade out. Fade back in with our color bar pattern. It's all captured. It's ours now. We can do with it as we wish. Now let's click. We restored everything. That easy. didn't have to record all the display locations, didn't have to try and put them all back ourselves. Very straightforward. This sample code will be appearing on our website soon, here in the next few weeks.

Welcome to something completely different, Remote Operations. Remote Operations is all the stuff you need to do remote control over the machine. Display Drawing Activity Monitor and Generation and Hosting of Events. What could that mean? Display Drawing Activity Monitor lets you listen for state changes on the display. Oh, look, a window just flushed over here. Oh, look, Quick Draw Movie's playing over in that corner. Stuff's happening. It tells you the areas that have changed on the display.

We'll also let you post events into the system for remote control purposes or for user space device drivers. Now, how's all this going to work? Why do you have to scrape the bits off the screen? Why can't I go get them somewhere else? Well, all the apps in the system are drawing into their own private buffers.

You don't get to play with those. They're the app's buffers, not your buffers. The buffers all go into the Quartz compositor, which takes all the bits, blends them together, seasons them lightly, and sends them to the framebuffer for display. Some bits can be transparent. Some of the bits in the framebuffer may be the result of several windows accumulated together. The only place where you can get a consistent picture of what all the pieces look like is at the framebuffer.

There's a lot of processing that can happen between here and there. If you've looked at some of the accessibility interfaces, we have things like Display Zoom that can actually occur. So what you see on the display doesn't actually correlate very closely with what you started with in the individual app buffers. The final framebuffer product, if that's what you need, you pretty much have to go to that. The Remote Operation Services is the best way of getting these finished bits.

Clients of remote services include Apple Remote Desktop, Timbuk2, good old VNC, Some accessibility applications like to play with this too. They may need to post events into the system. So they'll be using the event posting interface to post into a particular session and tell the apps in that session to show certain behaviors.

Screen scraping. All I get is bits. Well, we have this complicated pipeline. It's really hard to get to the individual bits from the window. If you wanted to get the bits on the screen but not go to the screen, eh.

So to assist you in finding what's changed on the screen, we give you a system of asynchronous notifications. Async means that we tell you something has changed, your app goes and picks it up and gets the current state of what's changed on the system. It's async so that applications that don't behave well don't get to hang the entire system. It's very straightforward.

If you have delays in your processing, delays that occur, system-wide scheduling or load that cause you to be a little slow about getting the current state back, that's not a big problem. We keep coalescing the state changes for you. They're built up, and your app will get the latest image of whatever state's on the display. There are new functions added to Panther that tell you not about just changes, but about areas that are moving on the screen. This will help speed up certain operations that you might get into if you're doing screencasting.

: Oh, good morning. Now, screen scraping performance. Now, that screen is at the end of a long bus, that long pipeline. It's out there at the distant end of an AGP or PCI bus connector. And if you go out there and read those pixels yourself, it's going to be slow.

The fastest way to read the pixels off the screen is not obvious at all. You use OpenGL. You build a full-screen GL context, select the screen with GL Read Buffer, GL Front. You read the screen with GL Read Pixels into your own buffer. That's going to get you DMA transfers from the screen right into your application's buffer, if you do it right.

If you don't do it so right, it's still going to get you a fast software path, but it won't be as fast as doing the DMA trick. Things to be aware of. GL is a little different view of the world than you might be used to if you're used to dealing with framebuffers. It puts things in quadrant one, the origin point being down in the lower left corner. It's upside down.

Now, if you're just processing the bits to pack them up and send them across for a screencasting application, that's not actually that big a deal. It just means you have to walk through the bitmap in a different order. Set your pointers up correctly and set up your stride as you traverse the data, and you'll be all set. Source code showing you how to do this is going to be online. I'm going to show you right now what this looks like over here on DEMO 1. A little bitty grab application. We can grab the screen contents and go ahead and save it to a file.

There we are. And look, it's a screen grab. Wow, that was exciting, wasn't it? He can go back to sleep now. I hooked up something interesting to this version of Grab. We can time it. We'll do 10 Grabs in a row and average it out and see how many milliseconds it takes to go ahead and scrape this 10, 24, 7, 68, 32-bit screen in software. Okay. 140 milliseconds. That's not too bad. Let's try it with OpenGL.

It got faster, didn't it? That's more time for your application to get the data compressed, shove it over the wire, and catch the next screen update. I buy a factor of 10 in this case. Now, you won't always see that big of a change, but on pretty much any of the hardware we've built in the last couple of years, this is faster than doing it in software. DMA, good. So, how did this code do it? It's rough.

We set up... For our main display, a mask bit. We say we'd like a full-screen context for that display. We choose a pixel format.

Set that context to be the current context. Tell the system, "This is full screen. Here we go." Point GL at the front buffer, that is the display. Ask it for the width and height. Get a place to put the data we're going to grab.

makes sure that all the GL commands are finished and the display is correct and current. Tell the system we'd like the data packed, please. Call GL Read Pixels with the mystical incantation GL BGRA. GL unsigned int dash 8888 rev. These are Apple extensions. They're present on all Mac OS X systems.

So I didn't bother checking for that first. Naughty me. If you ask for this pattern, this will match 32-bit deep display hardware. You'll get the DMA effect. GL unsigned int 1555 rev, the only other rev depth in there, will get you the 16-bit display format, and you'll get DMA transfers in that case also.

When you're all done, release the stuff. That's all there is to it. Less than two pages of code in really big type to capture the screen contents. And back to the slides, please. Over in the event side of systems, you need to post events into the works, right? This is interesting.

We have keyboard and mouse events coming in from the kernel, or to the kernel from our hardware. The kernel and HID system pick up those events, Chew on them a little bit, put them into a nice format for us. Pass those into the Quartz Compositor. What? The Quartz Compositor knows where all your windows live. It knows which app is foreground, so it can be used to route these events out to all your applications. There's a couple of ways to push events into the system.

IOHitPostEvent is a call from user space into the I/O Kit that takes an event structure and shoves it right in as though it were coming from other hardware. If you are writing a user space device driver or something that wants to pretend to be a device driver synthesizing events system-wide, use IOHitPostEvent. There are also some techniques that can be used to inject events at the Quartz compositor level. These are actually not appropriate for user space device drivers. They are used in some assistive technology, though.

These let you post a keyboard event in, a scroll wheel event, or a mouse event in at the Quartz compositor level. Now, why are these not appropriate? We have multiple user sessions on the machine now. There can be a second user login with a second set of Windows, a second Windows system, second set of apps, a second dock, all that. Post Keyboard Event, Post Scroll Wheel Event, and Post Mouse Event are going to post into the session they are called in. So if you're emulating hardware, you missed.

The difference there? I don't want anyone to get tripped up on this. Posting mouse events is pretty straightforward. CGPost mouse event, you give it a cursor position, you tell it a Boolean that tells it, "I want to move the cursor to this position." Normally you'd pass the true in. You pass in a count of buttons, one, two, three, and a set of Booleans for the button state.

If you push a button down, don't forget to post another event to let the button up. Thank you. That has a really bad effect on the system I'm going to get to in a minute. There's also an interface, CG Warp Cursor, you can use just to move the cursor around on the display.

Posting keyboard events. There are key codes, which are the decrypted glyph codes that you see in a system. There are also character, I'm sorry, there are key codes. Boy, is this terminology fuzzy. Key codes are virtual key codes. They're scan codes coming off the keyboard. Character codes are the encoding for a particular character or glyph. The thing you need to get right passing into this call is the key code. Most environments ignore the character code part. The virtual key code is the part you need to pass in. There's actually a dummy flag you can pass in.

For the character code, it'll cause the system to guess at the best key code to use with it. Application environments like Carbon and Cocoa use the key mapping and the key char API as part of Carbon to actually do the translation from that raw virtual key code into the character code.

Modifier flags are set by posting virtual key codes for the modifier keys. All the keys on the keyboard have these key codes. You pass them into this interface and it's just like you're pushing the keys and letting them up on the keyboard. Don't forget to release the key after you've pressed it.

Event suppression, when you post events into the system, We turn off the hardware event path for a little while by default. That's to avoid mouse wars. A student help desk system, you want to drive a system for a little by posting events into it. You don't want the eight-year-old grabbing his mouse and fighting you. So we'll turn off the local mouse for a short period of time. The force-quit key combination reconnects the local hardware if things get really bad. Say you goof when you're debugging things. You forget to let the mouse button up.

There are functions to override the default behavior. Now, I'd suggest you always set the state as you want, just in case the default behavior changes in the future. Event state combining is one of these things. That combines the flags from your keyboard modifiers from the real keyboard with the events from the remote system.

You might want to keep this off to keep local modifier keys, say the local shift key being held down from getting into your events. The most frequent question we get is, how do we turn this off? To do that, you just want to set the suppression interval to zero so it doesn't suppress things for any length of time. Tell the system, "Yeah, I want event combining. I want my local shift key and my remote shift key to all act the same." And you'll want to tell the system, "Don't bother blocking local hardware events.

Permit all events during the suppression interval and permit all local events during remote mouse drags. Told you you had to let that mouse button up. If you leave that mouse button down, you're doing a drag operation. And this will normally block all local events during that drag to keep things from going horribly astray. If you don't post that mouse up, you're sticking in a drag state. Bad. Don't do that.

Common Developer Problems, questions we run into. "I want to filter all the events in the system." Well, you don't do that at the Quartz level. That's the kind of thing that goes deep in I/O Kit. Or you might want to think about what you're doing, what is it you really need to accomplish? For example, some of the questions we've gotten on this are operations that are better handled as input managers in the system than as generic low-level, I filter every event and pick out the ones I'm interested in sort of operations. Another little comment for people.

: Documented interfaces may change. They may go away. They may stop working. Undocumented parameters to undocumented functions may do undocumented things to the system. You may let the smoke out of some parts.

The stuff in the public headers is actually better. When we do our development, we often build things first as a private SPI, and then once we've figured out all the quirks and have, well, we've gotten our act together, we turn it into a public interface. Please use the public interfaces. Who here is calling a private interface? Any volunteers? Ah, yes, yes. I'll change that for you. I'll take care of that one. And for more information, let's get Travis up here.

Thank you, Mike. I think we have some reference documentation available for you guys. We're actually going to aggregate it. It's all aggregated on sort of the reorganized ADC website where TechPubs has moved the documentation from where it was before we began WWDC. So everything should be, you know, documentation should be appearing on a lot of the stuff under the graphics and imaging area on the revised ADC website.

We also have some tech notes that are available. So there's going to be Tech Note 207. It talks about the CG Direct Display API. And then, well, probably very many people won't be going to 208, the CG Direct Palette API, considering how unpopular 8-bit palette is actually becoming and should become.

We have some additional QAs, such as many relating to Quick Draw and how to use it if I'm a Carbon Quick Draw app and I need to use CG Direct Display. You know, several Q&As up here that are available. And we have lots of sample code that are going to be appearing. I don't think all of these are up quite yet, but check the ADC website.

We plan to push a lot of the samples that you saw in today's demo up very shortly. And then obviously, if you need more information on how to use GL and GL Read Pixels, you can look at a couple different GL references at OpenGL.org, which is a good resource for OpenGL in general.