Good morning everyone. Welcome to session 511, Game Solutions: Graphics and Events. That's great, we have a full house. There are some available seats over here for you guys if you want to take them. This is the first of a series of two dedicated game sessions here at WWDC this year.

The contents of these sessions were a result of some research we've done with our game community through mailing lists and direct contact so we could tailor these sessions to the most frequently challenged space by game developing on Mac OS X. We have some really good content but we're also looking forward to your feedback and your questions afterwards so we can further improve game technologies on the Mac. We have two presenters, two game experts DTS engineers here and for the first part of the presentation we'd like to welcome to the stage Apple DTS engineer David Hill.

Thank you, Sergio. As Sergio said, welcome to our first of two sessions in this very room. And we're going to be talking to you about graphics and events today. I am David Hill and my cohort, Todd Previt is hiding somewhere. We'll be talking to you today. I'll start out the talk and then he'll come up for the second half.

So what are we going to be talking about today? We decided to take a slightly different approach to these sessions this year than we did in the past. This year, rather than focusing on a specific technology and trying to cover it and kind of repeating a lot of what the engineering groups are doing in other sessions, we're going to focus on some more unique aspects of Mac OS X game development, perhaps some undocumented or under-documented areas that you may have run into troubles with, and cover a few little tips and tricks that we've picked up, both in our work and working with developers, and just general things we've picked up off the mailing list. and also maybe cover a few things that are a little bit new to some of you that are just coming from Mac OS 9 and haven't really started exploring what you can do with Mac OS X.

So what are you going to learn? We're going to cover a wide range of topics today. First, I'm going to start out and give a brief update of what's happened with the CG Direct Display API. There have been some additional APIs added to help you out with Jaguar. Then I'm going to go into some full-screen debugging techniques.

We found some developers that were having a little bit of trouble on Mac OS X when you capture all the displays, your app takes over the displays, everything goes black. Well, if you've got a bug and your app gets hung, what do you do? We're going to cover a little bit of that.

I'm also going to give a little bit of explanation about how you can go put Carbon controls and other content on top of OpenGL. If you've seen some of the GL sessions, they've been downloading some really amazing new features and tools. They've been talking about some new things that you can do to layer Carbon and Cocoa content on top of your OpenGL surface.

I'll be talking about one example, a couple of examples of that, and going into detail about one that actually does work both on Mac OS 10.1 and on Jaguar. Once I get done, I'll bring Todd up and he's going to cover some QuickTime and OpenGL integration, how to get data out of OpenGL and into QuickTime, some OpenGL buffer operations, and then a little bit about Carbon events.

So let's talk about CG Direct Display. As some of you may or may not know, CG Direct Display is the Mac OS X API for managing displays, for doing--iterating over your displays, finding out what modes they support, switching between modes, doing gamma fades and those kinds of things. And it's been around since the beginning of Mac OS X and we've been improving it as we go.

And this year we've added a few new things in Jaguar. For some of you, you've been asking specifically, "How do I tell if a display mode is stretched?" If you don't know, on some of the monitors like the LCD panels we have and on the tie books and things like that, the aspect ratio on the display is slightly different.

So there's an 800 by 600 aspect ratio and then there's an--I don't remember what the exact numbers are, but there's another one that's a little bit wider, it has more pixels. And you have a choice whether you want to stretch that 800 by 600 content all the way across those extra pixels or whether you want to leave it 800 by 600 and have black bars on the sides. So the stretched mode is the one that stretches it out.

And depending on your content and your users, you may want to be able to tell, "Is the mode I'm going to switch to or the mode I'm currently in, is it stretched or not?" Some gameplay--some game players may want to fill the whole screen. They hate the little black bars. And some of them may want to make sure your art is at the right aspect ratio and maintain that.

So in the display mode dictionary, when you get information about a particular display mode in Direct Display, you're going to want to check the display mode. You can actually check this KCG display mode is stretched flag. And it'll tell you whether or not it's stretched. We've also added a lot of support for both software and hardware mirroring.

And in order for you to be able to tell what's going on in the system when your app comes up, you need to be able to tell whether a particular display is mirrored or not, whether it's the main display in a mirror set or whether it's one of the copies. For the most part, for your game, when you're using CG Direct Display, you're not going to care. If you capture the main display in a set, it will capture all of them. If you capture one of the copies, it will also implicitly capture all of the others.

So CG Direct Display helps you out there. You don't have to worry about that too much. But in cases where you want to, say, do a gamma fade, you may have to fade all of the monitors individually. And so you'll need to know what you're dealing with. And so we've added some extra APIs for that.

In addition, we've had some developers asking, well, if I've got multiple displays, can I change the resolutions or the bit depths or whatever on all of them at once? Well, we have a new concept of display configuration where you can say, I want to begin a configuration. I want to set monitor one to this, monitor two to this, monitor three to this. OK, end the configuration and commit it. And they'll all change at once.

So that's another useful technique for you. We've also added a convenience API to just capture all the displays with one call. So if your game is going to be drawing on one display, but you definitely want to black everything out for the most immersive experience, one call you can capture all the displays and be ready to go.

And then we've also added some APIs to allow you to control fading. A lot of times your game may want to fade to black and then back up to a movie or vice versa to do transitions between areas in your game. We've got fade APIs that are very simple to use.

You make one call to reserve a display for some amount of time and then you say fade from the current normal color set to a particular color. You can fade to black, you can fade to red, whatever, over a specific span of time. And so you can do everything from simply fading the display to black and then back up. You can do a quick flash to red if somebody gets hurt. Those kinds of things.

And one really nice thing about these APIs is there's a flag at the end, if you look in the CG Direct Display header, there's a flag at the end that tells whether it's synchronous or asynchronous. So you can actually make a call that's asynchronous and you say, okay, over the next three seconds I want you to fade to black. It immediately comes right back to you. And the Windows server handles the fade. And so you can continue to play your movie, you can continue to fade out at the end of your game play, what have you. Real nice effect.

So those are the new things for CG Direct Display. So let's talk about full-screen debugging. So a typical scenario, you've written your full-screen game. Your game captures the screen and fades to black. You know, everybody knows, the game's supposed to fade right back in and display your new content, but it doesn't.

So what do you do? How do you tell what went wrong? How do you debug that situation? We looked at a couple of possible scenarios for this. There are some standard things like printf and other logging where you just sprinkle calls throughout your code and watch the log and try to find out what's going on.

There's also, if you happen to be using DrawSprocket, if you come over from 9 and you're still using DrawSprocket, DrawSprocket has a debug mode that you may or may not know about that could be very helpful. Then there's the topic we're going to focus on a little bit today which is remote debugging.

So we'll look at each of those in turn and kind of think about what's good and bad. So for printf and other logging, it's good for post-mortem debugging because when you capture the display on Mac OS X, display goes black, the force quit command, command option escape, still works, but in this case, we don't know what state the screen's going to be in or anything, so we simply kill the frontmost app, which is very helpful when you're debugging, you get something wrong and you need to quit the app.

The problem is you don't get any state information about where the app was when you killed it. So printf and other logging, you can go look at the log after you've killed the app, take a look at it and find out what's going on. It does require that you modify your code, so you have to sprinkle these statements throughout.

And if you're not exactly sure where the problem is, you may have to get into this binary search mode where you say, okay, well, I'm going to put a log in here and a log in here, okay, it happens somewhere in between. So I put a log in in the middle, and then, okay, it happened in the first half or the second half, kind of divide and conquer approach. It's not the most efficient way to debug.

We happen to be using DrawSprocket. DrawSprocket has a nice debug mode that doesn't show the blanking window and doesn't completely fade the screen out. When you do a fade, it will actually fade partially out and then come back, so you can still see what's going on. And one good thing about this is it does work on Mac OS 9 or 10.

So if you want to turn DrawSpark at debug mode on, there's two ways. You can do it programmatically within your app. You can call it DSP set debug mode to true, call it DSP set debug mode to false when you want to turn it back off. Or you can create a folder in the same directory as your app called DSP set debug mode.

One important caveat to this is that you have to enable this before you enable your context because it takes, before you activate your context because it starts to take effect with the next context. And on 9, specifically, you must be using the debug build of DrawSpark. On Mac OS X, the DrawSpark framework takes care of this for you.

You don't have to worry about switching builds of DrawSpark in and out. So that's a good thing to think about if you're still using DrawSpark. And now we're going to talk about remote debugging. So this is a great thing, a great power of Mac OS X because unlike 9 when your game is taken over the screen and perhaps locked in a tight loop not processing events, on 10 there's all these other processes running.

So if you do have a second machine handy somewhere on the network you can actually remote login using SSH and then tell what's going on in your program. You can attach to it with GDB, find out where it's going, you can actually step through. Now of course GDB is a command line tool, doesn't provide an IDE, but contrary to some, you know, misconceptions, it does allow for source level debugging. When you compile your code with Project Builder, say, it actually embeds in there where the source files are. So when you go with GDB, it can actually pull up the source for you and you can step through line by line and watch what's going on.

So for those of you that perhaps aren't familiar with GDB, especially some people coming from 9 that have never really used something like this, Probably the first place you should start playing around is in the help. There's a basic command help, and it'll list a number of different options you can use for getting help. The two important ones are help and a command class name, in which case GDB will actually list out all the commands.

For example, there's a class of commands for handling breakpoints. There's a class of commands for dealing with the stack, for showing info about the state of the program and things. And once you figure out, okay, I need to know about that specific breakpoint command, well, then you can help in the specific command name, and that'll give you the full documentation, tell you how to use it. So there's a lot of help in there, very useful. You can just get in and start playing around with it.

So if you want to start using GDB, and in our scenario that we talked about earlier, your app is running, it's faded out, it's gone to black, hasn't come back. Well, typically you'll go to a second machine, and I will point out that because of the way this is done, it doesn't have to be another Mac.

Some of you may be dual platform developers, you have a PC and a Mac in the same office, you're running your code on the Mac and it goes black and doesn't come back. You can simply go over to the PC, and as long as you have access to secure shell, and secure shell into an IP address, you can do the same debugging, because GDB is actually running on the target machine, not on your PC. So once you've secured Sheldon and you run GDB, you need to find out what the process ID is. That's the PID that I have labeled there. And you need to attach to it.

The AppDir is a tool that allows you to run your apps in the app. If you have a problem, you can launch it from within GDB. You type in file and the path to the app you're going to debug and then run. For typing in the path, you have to watch out because once you get on 10, you're doing a lot of bundled apps. You have to give the path to the actual executable inside the bundle, not the bundle itself. In this case, if you're AppDir/content/macos/ and the actual executable, the app to debug. And then you can run that and GDB will launch it, load it into libraries and everything.

So, I don't know about you, but for me, I'm used to a debugger that has, okay, step into, step over, step out, continue, those kinds of commands. And GDB has the very same things. They're in respective order, step, next, finish, and continue. And you can typically abbreviate those. GDB is pretty good about trying to figure out what you mean.

and it will tell you if you try to give an ambiguous abbreviation. And so that gives you all of your basic source level debugging, stepping and everything. Once you've actually gotten into your source code, you need to set breakpoints. There's a couple of different ways to go about this.

You can specify by file and line number, by function name. An interesting one to point out there is that little FB there. That's a fun one for GDB that I haven't seen in some other things. It's called Future Break. You can set a future break on a file and a line number or just a function name.

And every time GDB has to load--every time your app loads in a new library, GDB loads in the symbols and tries to match that function name against the libraries that come in. And so you can actually set a breakpoint in a library that hasn't even been loaded yet. It can be pretty handy, especially if you're debugging plug-ins or other things like that.

Also, you need to know how to get rid of breakpoints. You can either clear by an address or delete by a breakpoint number. So the next logical question is, "How do you find breakpoints?" You can find out what the breakpoint number is. It typically tells you when you set it. And also you can type in info breakpoints, and it'll give you a list of the current breakpoints and whether they're enabled or disabled and those kinds of state information.

So once you've stopped at a breakpoint, you've got to figure out what's going on in your app. And there's some really simple commands. BT is a good one to remember. It gives you a stack backtrace. You can find out about the local variables. You can find out about globals and static variables. If you see a variable in the source that you're looking at and you can't remember what type it is, you can type what is variable name and it'll show you what type it is.

If you want to actually list parts of the source, you can type list. And by default it shows you the area around the program counter. You can type list and a line number and it'll show you the list and a current file. You can even type list and a file name colon and a line number and it'll show you source from a different file. And if you need to gather information about the current source file, you can type info source. That's another helpful one.

Displaying variables is interesting. There's a P command that will print variables. There's also a really powerful X command. I'm not sure exactly what it stands for. Maybe extended print. Who knows? And you type X slash a format specifier and an address or variable name. You can even include a repeat count. And you can specify a number of different formats, decimal, float, hexadecimal, how you want to display it, and the size.

And so, for example, if you had a float variable, you could type X slash F and a float variable name and it would display floating point content. Depending on what kind of debugging you're doing, you may need to tweak variables so you continue execution if you've got something broken. You can set a variable name equal to a new value. That will update it and you can continue from there.

For those of you that are starting to explore Cocoa and Objective-C, there's some built-in functionality there. There's a PO command. You give PO an Objective-C object ref, and that will actually print out the contents of that variable. That's an interesting point because what it's actually doing is calling the description method on the object. A lot of Objective-C objects have a knowledge way to describe themselves.

This call will actually call code in your app on that class and have it describe itself. So GDB doesn't have to know anything about it. So even for your custom classes, if you implement description methods, you can actually have it executed. And that brings me to perhaps one of the most powerful functions of GDB. You can actually call functions in your code from GDB. And the typical way you would do it, you would call and give it a return type, a function name, and whatever arguments. And for those of you dealing with CF, you've probably run across this one before, CFShow.

Very useful, I always forget it. Returns void, CFShow, and any CF type ref, and it will describe the object for you. So that one's pretty helpful. This is also extremely useful in your own code. A lot of you have your own debugging functions, have your own, you know, tell me the state of this object, validate this object, those kinds of functions. You can call them directly from within GDB. that's kind of hard to do in other debuggers.

You can call it directly from there, you know, dump my internal state for some object. It's important to watch out for side effects, though. This does execute code in your app. So if you make state changes to objects when you're doing these kinds of things, you may get some weird side effects as you go along because you're changing the state when you didn't expect to in your program because you're calling things sort of out of order. So watch out for that. So I thought we'd run you through a real quick demo.

This is demo two here. So we're going to pull up full-screen debug. And I've basically written a really simple app that captures the screen. and oh look, that doesn't look quite right. I'm not coming back. So if we can switch over to number three here, we've got our handy terminal. And we're going to try to do this live, so 86, 63, and 161. Yes, that's a good idea. Bigger, let's go bigger.

Can you all read that now? Is that a little better? Yeah. OK. So now we're logged into this machine. So all I had to know, I had to be able to run SSH from here, I had to know the IP address and the password. In this case, because I didn't specify to SSH who I was logging in as, it just assumed I wanted to log in as the user I'm logged in here, Apple. So I have to know the password for Apple on that machine. So here I am.

And I need to know where that process is. So we look through this output, and it looks like this is it right here. Running off the desktop. And so I can say, well, please, I'm GDB, and attach to, what was that number? 624. So GDB finds that process and it loads in the shared libraries.

Boy, this is a lot faster on a dual processor machine. I want one of these. Okay, so now we're switched. And we say, okay, well, where did we stop? Well, looking at this, I can see that from my main I was calling sleep. Told you this was a simple app. And it's stuck way down in there somewhere. So the important thing here is I don't really care about clock, sleep, trap, nano sleep, sleep, whatever. One of the things you can do is you can type finish.

This is a camera run until I exit from clock sleep trap. That puts me back at nano sleep. Well, I found this bug yesterday. If we look at the backtrace again, that doesn't look quite right. So it turns out, and this doesn't happen on all of my apps, I tried it on a couple of things, there appears to be a bug in GDB, I think it's a known bug, at least from some of the Project Builder guys, that Finish doesn't, always.

Now, it appears to just get a little confused as to what the backtrace is. Now, the important thing is the app is still fine. So I can step, and we happen to be near the end of NanoSleep. So step takes me to sleep. And if I step again, oh good, I'm back out in my code. So watch out for that. This is on the Jaguar seed that you have.

It appears to be a little flaky, at least as far as backtrace finish goes. So now we're back in our code, and I can type list. And oh look, I'm in a loop. I'm asking some other function, should I loop? And then I'm sleeping. So let's step, which will actually step in. Okay, now I'm in should I loop. Let's take a look at what I'm doing.

Okay, so I Thank you for joining us. I'm David Hill, and I'm going to talk about the most important part of the process of creating a balanced and efficient Mac OS X. I'm David Hill, and I'm going to talk about the most important part of the process of creating a balanced and efficient Mac OS X. which is, I should actually show you that.

So here are my local variables. Wait for GDB is a local variable set to one. And none of the code that I showed you am I actually changing that. So silly me, I goofed up. So let's just set that wait for GDB Back to zero. And if we continue from here, it'll go wait, sleep for a second, and then oh look, We're back. So now I'm back out of that loop and I'm going on with my program. And in this case we can just quit and go on. So let me quit out of GDB right quick.

Okay, so that's my connection. Good deal. All right, so that's your real simple introduction to GDB. It's real easy to use. You can start playing with it, get in there and take a look and see what you can do. So let's go back to number one. We've got a few more slides to talk about here.

Okay, so you know how to debug a little bit. Now let's look at something more interesting that came about with the announcement of Jaguar and Quartz Extreme. People start to get interested in, oh, well, we can now layer all this content. And, you know, I have a game editor, say, and I want to be able to have my OpenGL content displayed in all of its splendor and put some controls over the top.

How do I do that? We're going to look at a couple of possibilities. One possibility is always you can implement your own controls. You guys know how to do graphics, know how to do events and things. You can implement your own controls. You can pick a normal window and put those controls in a higher level window. You can use one of the new overlay windows in Carbon, which is completely transparent and then you can draw your content in it. Or you can use some of the new surface ordering APIs that we introduced in Jaguar.

So if we look at those one by one, you decide to implement your own controls, you have extremely precise control over the drawing because you're doing everything. And you can place it on screen with OpenGL. You would typically render your controls into a texture and put them on screen with OpenGL, say.

Your users are going to expect at least some level of consistency with the app and the OS. So you have to kind of try to mimic the Aqua user experience. And you also do have to draw everything yourself. So that's a bit of work. And unless you already have code to do that, I don't particularly recommend it.

You can place controls in a higher window. If you do an opaque window, just a normal-- you can ask Carbon to create a window with no borders and no shadow. You can layer that on top of your OpenGL content. So it's just a layer-- a window floating on top. You don't have to draw the controls. You can put normal Carbon controls in it. And you don't have to handle any of the event interaction.

The problem is it's opaque. So it blocks out a rectangular area of whatever your content is. And you may or may not want that, depending on how you have your content laid out. And you're going to have multiple windows. And I'll tell you how to get around some of the management problems with multiple windows in a minute.

So what if we wanted to show our content through the window? We just had a round button we want to put on our window. We want the rest to be transparent. We can use an overlay window. Again, the OS handles the controls. You don't have to mess with them. And your OpenGL content can show through except for where you've made that overlay window opaque. Now, the cons are you will have a speed penalty if you're in the software compositor as opposed to Quartz Extreme, and you still have those multiple windows to manage.

and then the new thing we've introduced for Jaguar is surface ordering. With surface ordering you can actually take The ordering of the surface is the OpenGL surface and the window back buffer surface. And normally, you can rearrange them. Normally, you have the OpenGL surface on top. And for those of you that have tried to actually draw, say, quick draw content on top of your OpenGL, you always say, well, it never shows up.

And then as soon as I dispose of the OpenGL context, well, there's my drawing. Why wasn't it there? Well, it's actually just in a surface behind the OpenGL surface. With this API, you can put the OpenGL surface behind the back buffer surface for the window. And if you clear out the alpha properly in the back buffer, you can actually do the same kinds of tricks where you put, say, a Carbon button that's transparent around it on top of your OpenGL content. So this you can do with Carbon and Cocoa. The OpenGL content does show through. Again, there's a speed penalty if you're using the software compositor. And it does require Jaguar, so you have to be aware of that.

So for those of you that can't quite require Jaguar yet, especially since it hasn't shipped, you might want to look at the overlay windows. Overlay windows can actually give you reasonably good speed and they do allow your content to show through and they will work on Mac OS X1 and on Jaguar when it ships. So what do you have to do? Well you have to create the overlay window obviously.

And then in order to help you manage those multiple windows, there's a concept of window groups. You can create a window group and add your windows to it and the window manager will actually manage a lot of those windows for you. And then you install your controls and your Carbon event handlers and all those kinds of things. Todd will talk a little bit more about Carbon event handlers later.

So creating an overlay window, you use the standard Carbon Create New Window call. The important thing to know is you have to pass OverlayWindowClass instead of your normal DocumentWindowClass or what have you. And you get back a window ref. So then we need to create a window group.

And a real simple API. And in this case we're not passing any group attributes in. We're going to set them at the end and I'll talk about them there. You have to tell the window group what its parent is. In this case we're just saying, "Ah, let it live with the document windows.

That's fine." And then for each of our windows, say we have an OpenGL window ref and an overlay window ref, we add each of those to that group. And then we're going to set some important attributes. And there are a number of things you can set. In my particular case, I want the windows to move together.

So I've got a main document window and I've got an overlay window over the top. And when I drag the main document window, I don't want to have to mess with the overlay window. I want the overlay window to move with it. So that's the first one. Add or move together.

Layer together means if I switch that document to the back and bring it back to the front, those windows stay together. That's very handy. And also hide on collapse means if I collapse the main window, the overlay window goes with it. And then if I bring it back, it'll automatically repaint the overlay window when it comes back. So let's look at a quick demo of that.

All right, we're up. Let's go back over here. Actually, I bet I'm in the dock. We planned ahead. So let's just pick an image. And for those of you that have been paying attention, I have liberally borrowed from Jeff's excellent OpenGL image sample. And so what we're seeing here is just OpenGL image sample and I've simply put in an overlay window over the top.

So, and I put it in a group and it's layered together. So, this big red thing here is actually some quartz 2D drawing. So that's layered on top and it's nicely, you know, transparent and you can see if you zoom in, I don't think the accessibility stuff is turned on, but if you zoom in you can see it's anti-aliased and everything. And I've also got one lonely little Carbon button.

So what happens if I click this button? So I've just leveraged the app as it exists. Jeff has a command Carbon event handler that takes commands from buttons and menus and those kinds of things. And I've just co-opted the auto-rotate command. So this button just has the same command ID as that menu item. So I can stop and start, and I can drag it around while it's rotating.

And if you notice, you look in that bottom left corner, for those that you can read it, it displays the frame rate. And we're doing this at 60 frames a second. And I'm not doing anything too terribly bright. I've got an overlay window that's the entire size of this content. And I can drag it around while it's moving.

all sorts of things, and the Windows server keeps everything nice and tidy. If you were doing this for real and, say, you had a button and this other thing, you might want to make the overlay window smaller, so especially in the software compositor case, it has less work to do, less compositing to do, because in the software compositor case, the compositing happens on the CPU, but this nice GL content happens on the card, so it has to read back across the bus, do the compositing and software, and put it back.

But you still get reasonably good frame rates, and if you optimize it some, like I said, if you make the overlay window small and another small window here, you can carry off effects like this even on 10.1 in the software render and get really good results. Okay, so that's the simple image demo.

All right, we're back to slides. So that's it for my half. I've covered CG Direct Display changes, some full-screen debugging introduction, and also how to put Carbon, and you can also do it with Cocoa, controls over OpenGL content. Now we're going to bring Todd up, and he's going to talk to you about event loop timers, quick time in OpenGL, and OpenGL buffer operations.

Thanks, David. So I'm going to start first going into event loops and timers, then into, as David said, QuickTime and OpenGL, and then following up with some buffer operations in So here's the story on Carbon events. This is how you get into it. Run application event loop from your main function is what actually kicks you into the whole Carbon event structure.

And when you're ready to quit out of it, quit application event loop is what will bring you back out next to the shell. I've also included a little addendum about wait next event down there at the bottom. Wait next event is dead, folks, and it's time to start moving on to Carbon events. Everybody has complained historically about the performance of Carbon events and with a lot of the changes that we've made in the recent builds of Carbon lib, I think you'll find that performance is definitely there and it's going to do better for you in the future.

So some basic structure on Carbon events. It's kind of broken down into event levels. These are the four main levels. I believe there might be one or two more, but these are the ones that you're really going to be concerned with. At the application level, these are events that are to be received by the application at the highest possible level. They have the highest possible priority, and they're usually reserved for things like Human Interface Toolbox Commands.

You can accept any Carbon event at this level because you can always handle it as it's the highest application level and everything will actually roll down into it. But it's kind of the catch-all and if you don't handle it here, it gets kicked back to the system and usually if you've installed a default handler, the default handler will take it from there.

Window events or the window event level constrains all your events to the window. This becomes really important for things like mouse events. If you register for mouse events at the window level, your mouse motion stops once the cursor moves outside of the window. Certain games and certain applications will want to take advantage of something like that.

Others, for full-screen apps or for windowed applications where if you have multiple monitors and you want to have the mouse still continue moving the screen in the game, even after you pass outside the window, you'd want to register for your mouse events at the application level instead of at the window level. Otherwise, you'll end up pinning your mouse cursor to the side of the window, at least as far as the game is concerned, while all the mouse moving events and stuff will stop once that cursor leaves the window region.

The menu event level is pretty much just that. It's all about your menus and that's where you would register for handling commands off your menus and for anything else that's related to them. And then the control level is basically the lowest level possible and it allows you to install individual event handlers right into the control itself. My personal preference is actually to have one handler deal with all the controls in my application if I can, as opposed to having individual handlers on the controls themselves.

A little further down are the event classes. The event levels is where you register for these events. The event classes are the actual events that you register for. I just put some samples up there. This is certainly by no means an exhaustive list, but these are some of the more important ones that you'll be using quite often. Mouse events, keyboard events, window events, and menu events.

As I discussed just a few minutes ago, the mouse events will give you things like button clicks. They'll give you your motion, your mouse deltas. That's a particularly important one for game developers. We were asked about that quite a bit. They've been there for quite a while. They work really, really well. As a matter of fact, I use them in several of my applications.

and the mouse class events deal entirely with the mouse scroll wheels, motion, all that kind of stuff. Your keyboard events will give you everything from key up and key down, all the modifiers, and everything related to receiving events from keyboards. Window events give you things like activation and deactivation, show and hide, grow, shrink, all those sort of things that control the behavior of a window. The menu events are, as I described just a few minutes ago, dealing with selections of items, which items are selected. I believe also through the Carbon event structures how you can programmatically change the text in your menu items and things like that.

So what is an event, how do you get these events and how do you register for them? Well, as part of the installation, as you'll see in a minute, you have to give an event type count as to how many events you're actually passing in to the handle that you're installing. This particular one comes from application level events.

It's the event type specifier, and I just called it App Event List, which is a list of all the events that you want to handle at this particular application level. You can pass any event list to any level to install on any handle. I just happened to choose the application one for demonstration here.

I've highlighted a couple of the more important ones. The KEVENTCLASS command, that was, as I discussed, about the human interface toolbox commands. These are things like OK, Redo, Undo, Cancel. That's where all of those will get handled. Obviously the mouse events, as I just described, are not going to be handled.

You can see how these are broken down. I'm actually registering here for, as far as the mouse event classes go, I'm registering for mouseDown, mouseUp, and for mouseDragon and mouseWheelMoved. I'm not registering for mouseDeltas, but that's another one that you could also register for. At that point, wherever you installed that particular handler is where that event would get handled.

This is what they look like. As I said, the command processes, OK, QUIT, COMMAND, OK, QUIT, UNDO, etc. This is how they're structured. How do you actually install these things? For basic use, what you want to do is, before you install any event handler, in case there's something you don't cover, you always want to install the standard handlers. This is not required, but it's usually a good idea. They're very simple to do. The function protocols are right there.

What you want to do is when you call the Install the Standard Event Handler, say you want to install it on the window, that's the top one over here. This is Get Window Event Target. You pass in a pointer to your window. I believe it's a window ref or a window pointer.

That's all you have to do. Then the system will automatically go through and it will put the default handlers and install them on that window. Now it will handle everything that you don't. If you explicitly specify other events that you want to handle in your application, those will take precedence over the standard handler.

The other two here are just the standard handlers for the application and menu events. For the application event target, you don't have to pass anything in. It will just grab the current process and install it on that. Whereas the menu event handler, you have to pass in a handle to your menu.

This is the basic setup of installing your own. The app command processor variable right there is actually an event universal process pointer. We're just creating it with new event handler UPP. The function parameter there is actually the name of the function that we're going to use as a callback when we receive events.

The following function there, Install Application Event Handler, is actually a macro for Install Event Handler, but you can see these all in carbonevents.h. It's really just kind of a shortcut for installing a handler into the application. The first parameter there is our command processor, which is the routine that will be called as our callback and will deal with the events directly. As I showed you earlier, getEventTypeCount will pass in the number of events that you specified in the app event list.

It's kind of handy because that way you don't have to count up how many events you have in your event list and then have to pass that in manually. It just does it for you. The app event list there is that list of events that we created before with the event type spec.

You can pass in any user data if you want to through the next parameter. If you want to keep a handle to this particular routine after you install it, you can pass in an event handler ref at the end. I never do. All of your event handlers will get uninstalled once you quit.

Timers are really useful for a variety of things. I use them all over the place. The three main things that I do is I use a control in my rendering loop so I can either increase or decrease my frame rate as I need simply by reinstalling the timer. It's really useful for things like that.

It works pretty well. I've never seen any major performance problems from it, although there have been a few times when I've passed in the wrong parameters and ended up slowing way down. It's also really good for handling game events, application events. If you want to have a timer that's sitting there waiting and processing anything that happens to come in, if you want to process events that you have in your game or create your own event loops, then this is another good way to do it. It's also good for time-based events.

If you wanted to, say, take a snapshot in the future, you just create a timer, you pass in the start time, you pass in a firing delay, you want to take a picture in five seconds of whatever happens to be going on, and then the timer will fire. It's a one-shot deal, and you've accomplished the task that you wanted to accomplish.

So the first thing you want to do is you want that event loop timer ref to keep track of your timer. You're going to need that later for when you want to dispose of it. The timer UPP is just that. It's the universal procedure pointer for the timer that you're going to create.

Again, you will need that later for when you want to dispose of them. The function prototype at the bottom there is for the callback routine that the timer will call every time it fires. There's really not a whole lot to them aside from passing in the timer ref and any other data that you want to send to the routine.

Here we create the snapshot timer UPP by using new event loop timer UPP and we pass in, again, the name of the routine that we want to use as our callback. The first parameter you want to pass in is where you want to install it, as in if you want to install it on your application, which is pretty much the default. I actually don't know if you can install it on somebody else's application. I haven't tried that, but it would be pretty funny if you could actually install a timer into somebody else's app.

I haven't tried that. The next parameter is the start delay, and the following parameter after that is the duration, or the time interval between each firing. You'll notice at the end of that that I have a variable attached to the multiplication of the constant there. It's because this code actually comes right out of the CarbonGL snapshot demo that you'll be seeing in just a few minutes.

What I do is I multiply the milliseconds by my time interval so I can have it take snapshots at varying intervals. The Snapshot Timer UPP is the one we created just above there. The next parameter is any user data you want to send. The last parameter is that timer ref we created on the previous screen.

So you've got this thing created and you're happily sailing along, but now you want to get rid of it. So what do you do? Remove event loop timer and dispose event loop timer UPP. Those are the only two functions that you really need to call to get rid of your timers. The first one you pass in the timer ref, the last one you pass in the timer UPP. That gets rid of them and stops them from firing regardless of where they happen to be.

I always clear the pointers on my timers. I use them for various other things like I will look at them and see if they're actually valid. If they're valid, I know I'm in the middle of taking a sequence or I'm in the middle of doing something in the timer, so I don't go through and start another sequence.

If somebody happens to be pressing one of the activation buttons very rapidly and it says, "Start another timer, start another timer, start another timer," well, I go through and I check and make sure that those are null. It's not required that you do this. I just do it as a good habit.

Now we'll move on into QuickTime and OpenGL. Most of our demonstrations that we've done with QuickTime and OpenGL in the past have been with getting movies into OpenGL, getting images into OpenGL, using all the various data that QuickTime has as an input into OpenGL. Well, I've kind of reversed that and now I'm using OpenGL as input into QuickTime. There's three ways that I'm going to show you how to do this.

The QuickTime snapshots are actually just single image files that are taken out of OpenGL. The snapshot sequences are image files that are created based on the content of an OpenGL context over time, and then a QuickTime movie from OpenGL is just that. It creates a movie based on the content that was in the OpenGL buffer at the time that the movie was being created.

So like I said, the snapshot uses data from an OpenGL context to build a simple image file. It's one function. It's a single function that creates a single file. The file format and type is arbitrary. You can specify any format that QuickTime happens to support, which is--it keeps increasing with every release that they do.

The whole goal here was to let QuickTime do all of the work for you as far as getting the image out to disk. You don't have to worry about swizzling it around much. You don't have to worry about processing it or writing your own file format parser. QuickTime does it all for you.

So the basic methodology of what we're doing is we're using GL read pixels to get the image data out of OpenGL in the first place. Then we create a new GWorld with that data so that we now have the OpenGL image data in main memory in a format that QuickTime recognizes.

Then you set up QuickTime to create the image file with that data. And those steps are very, very simple. And it really just kind of follows the same methodology as most of the other QuickTime imaging methods, which are you do a variety of things. You specify your inputs, your outputs, and you just tell it to do it, and QuickTime just does. The final step is to export the data to the file.

So how do you do that? How do you get data out of OpenGL? This is a pretty basic operation. The first thing you want to do is set your read buffer to the back. This is default. OpenGL will always read from the back buffer unless you specify otherwise, but I've found that it's usually a good idea to specify it anyways in case you have been in some other part of your app.

If you've been reading from the front or if you've been reading from an aux buffer or from the depth buffer or something like that, then you want to go ahead and set it back to the back buffer so you can read out everything that's been drawn into it.

The next command is GL read pixels. The first four parameters are the dimensions of the area that you want to read out. In this case, I've hard-coded it to 640 by 480. I start at 00, which for OpenGL is the lower left-hand corner, and end it at 640 and 480, which is the upper right-hand corner. So I'm reading out the entire contents of the back buffer as they stand when this command is issued.

The next parameter to GL read pixels is the format of the pixels that you want to read out. In this instance, it's a reversed format which is BGRA. There's a reason for this that I will go into momentarily, that you want to read things out in BGRA. It makes things much easier. It also puts you on the fast path through OpenGL.

The next parameter there is the type. So actually, what is the type of one pixel? In this case, we're talking about an unsigned integer that is broken down into four 8-bit fields that are now reversed to match the BGRA type that we have. Finally, you pass in a handle to a data buffer, and it is very important that you allocate enough memory to that data buffer before you make this function call. GL read pixels will not allocate memory for you.

The last function there is invert GL image, which is just a little function that I wrote to swizzle the image around and get it into the proper orientation. When you read data out of OpenGL, it will be upside down and inverted. So in order to get it to a normal screen orientation, so it's just as you see it on the screen from OpenGL, you have to swizzle it around.

Now, this particular function takes the data buffer, the image size, and then the row bytes because it actually goes through and does a four-byte reversal all the way across. So you actually get the proper image and the proper format all the way through. Now when you reverse BGRA, what do you come out with? ARGB. That happens to be the "native" format that the Macintosh supports, and that's what puts you on the fast path through OpenGL.

So the next thing we want to do is create a new G World. It's very important to use new G World from pointer and not just new G World. New G World will give you a buffered G World which you do not want. At that point you would end up with an image that's going to be off by a great deal because it's going to add an extra padding on the rows. So new G World from pointer gives you just a straight G World with no padding whatsoever.

The first parameter is the destination G World that you want to use, which is a variable that you would have specified. The next is the format. You'll notice that this is 32-bit ARGB, which is exactly the reverse of the one that I specified when reading the data out of OpenGL. Now that we've swizzled it around with the invert GL image function, it is in this proper format and should be no problem.

The next parameter there is a rectangle, the bounding rect of the region that you're reading in. The next three parameters are color table, G device, and flags. You can pass zero or null. They don't really have much effect in this particular situation. The parameter down near the bottom there, the data buffer, that's the actual data that we read out of OpenGL.

We'll pass that in there and this quick draw is going to build the new G world with that data. Finally, we pass in the row bytes. At the end of this, we're going to need to create a file spec somewhere, as that's usually what QuickTime wants to deal with, for QuickTime to use to write out the image when we finally get to that step.

QuicktimeComponents.h has a list of the OS file type or OS type file specifications for all the various formats that it supports. In this particular instance, I'm using JPEG files. They were the easiest for me to deal with. So these are the real, I was talking about the really simple steps that it takes to get QuickTime set up. Well these are them. You just open a default component, you tell it what the type is, in this case we're going to graphics exporter component.

We pass in the file type, in this case OS file type would be set to be a JPEG. And then finally you pass in the graphics exporter component, or the address of the graphics exporter component that you want to use, which is another variable you would have specified above.

Following that you specify the input GWIRL, which is the GWIRL we just created. You just pass that to it. Again, you're passing in the graphics exporter component first, and then you pass in the GWIRL. Following that, you pass in that file spec we just talked about. That sets the output file where QuickTime is going to actually write out its image.

Next, if you need it for the file format that you're using, you'll want to specify the compression quality. At this point, I've specified lossless quality. There are a variety of different ones that you can use. Lossless will probably provide the largest possible image and the best possible quality for you, but you can adjust that depending on what you want to do.

Finally, the command that really does the work there is "do export." Again, you just pass in which component you want to use, which is the one you've been using all along, and out it goes. That's what tells QuickTime, "Do the work. Write out the file. Give me the image.

We're done." Finally, as a clean-up step, you'll want to dispose of your component and close the component So the snapshot sequence actually extrapolates on this function itself. We started with one function which was just taking the snapshot, and the snapshot sequence is going to use that function over time to take a sequence of image shots from the same back buffer. Again, your type and format is arbitrary. It doesn't matter. Whatever you want to use, you can use here as well. And this will also be using the Carbon timers to generate the sequence.

So, the way I controlled it from within the timer was I just looked to see how many frames that I had and how many frames I wanted to take, and I decrement that every time I go through the loop. Once I hit zero, we stop the timer and stopping the timer code was exactly what you saw earlier with removing timers.

In Windows Snapshot, I'm just passing in the main window. It happens to be the function parameter that that particular snapshot function wants. It's just how it's set up. What you'd want to do here is you'd want to take your snapshot, call your function, and then decrement your frame count.

So with QuickTime Movies, it follows the same methodology. However, what you're going to be doing is you're going to be reading the images into a buffer. As a matter of fact, I re-architected the application. Initially, it was just the snapshot sequence was doing all the QuickTime stuff all in one block.

But to make things a little smoother, what I did was I re-architected it to follow this particular format, which is it reads the data from OpenGL as fast as it possibly can, throws it into a buffer, and an asynchronous task comes along to process that buffer and write them out to disk. Now with the QuickTime Movie, what you'll be doing is you'll be creating a movie using Add Movie to Frame, I believe is the function call from QuickTime, and it'll just build the movie right as you're going along.

So again, it's going to be using GeoReadPixels to grab the back buffer and to send it off into the G world, or into the G world buffer as the case may be. I created an array that is of an arbitrary size at the present time, but you can obviously hard code it or you can specify a certain amount of memory that you want to take up with it. It all depends on how you want to architect your application.

The key here is to have a separate thread asynchronously processing that particular buffer. You want to go through there and have it go through in a... If you're on a dual processor machine, use the other processor. Don't be taking up the processor that's actually processing all the GL information.

You can keep rendering, keep going, keep moving along in your game. Meanwhile, in the background, you're processing all these images out. At the end of that, once all these things have been processed, you just have quick time to dump out the movie in .mov or .avi format, whatever you want.

Some things to remember when you're doing this. The images do need to be swizzled when they're coming out of GL. The routine that I'm using is terribly unoptimized, and so you would want to do this probably with AltaVec in order to get the most performance out of it so you're not taking a hit when you're swizzling that data around. I would recommend doing the processing of the image data before you're building your images. I'd recommend doing that asynchronously. Again, if you don't, you're going to see a serious performance degradation.

Arbitrary movie sizes require much, much more complex array and buffer sizing routines. I did some dynamic allocation, but I do think I've pinned it at either 128 or 256 megabytes. So it'll suck up a lot of memory, but it is pinned in one manner or another. Your frame size and frame rate will affect how much memory you should reserve for it. Obviously, the bigger your OpenGL context, the more memory you're going to be taking up with the data you read out of it. Again, again, if you want to do this all dynamically, it's going to be more complicated. So I'm going to do a little demo.

You can see what this looks like. So I've got an OpenGL context here. It's just kind of oscillating back and forth so you can get an idea of what the motion looks like. Of course, when we click over here--it's not obscured so we're not seeing performance decrease. When I take the first snapshot, I'm going I didn't really see much happen there. So I'll take another one. And we'll take another one.

And there's the three snapshots we took right there. Zero, one, two, and three. So if we open this up in preview. There's zero, there's one, and there's the third one. All at different states in the context. You can see that the graph on the bottom there is moving back and forth. So they're just kind of like little snapshots in time.

We'll get rid of those. Let's bring the application to the front. As it's still oscillating around there, let's go ahead and do 100 milliseconds. So every 100 milliseconds we'll take a snapshot of this moving context here for a total of 10 frames. So we click our sequence off. Notice it kind of gets a little chunky there as it was going through, but there's all 10 snapshots.

And we can just go right through these. Of course, it would be nice if it put them in order, wouldn't it? So there's zero. There's one. There's two. Oh, great, now I'm on eight. So there's two, three, four.

And then we'll go back to the slides.

So, I'm going to wrap up with a little bit about OpenGL's buffers and some operations you can do with them. I'm really going to focus on the auxiliary buffers and then render to texture. We've had a lot of questions recently about some of the rendering to texture stuff, so I'm going to focus pretty heavily on that.

A lot of people have resisted using auxiliary buffers because you can't go and use them both on 9 and 10. On 9, they are available, but it will dump you into the software renderer, so a lot of people avoid them. This is going to show you how to do some relatively cool things with auxiliary buffers.

So some buffer basics. A lot of people make requests to access the front buffer directly and do this and that and the other thing. You ought to leave the front buffer alone. There's no reason to be playing around with it. You should do all your drawing and do all your reading from the back buffer. Whenever it needs to be pushed to the screen, the system will handle that for you, either the Windows Server or GL directly. There's no reason to be dinking around with the front buffer.

And finally, a little pet peeve of mine is that nobody needs any direct access to any of OpenGL's buffers. For one, it's not going to happen because OpenGL happens to move its handles around internally whenever it wants to. The memory space on the graphics cards these days is all flat.

So when OpenGL needs to make some room for a texture here or a texture there, it's going to move handles around and shuffle things around. So you'll never know whether or not they're current, even if you could read them anyways. If you could get a handle out of GL, there's no guarantee that it would be current the next time you went to use it. So that's why no one is ever going to get access to those buffers.

The Depth, Accum, and Stencil Buffers. I put them on here just to kind of give, for those of you who are not terribly familiar with OpenGL, I put them up here because they are additional buffers that are used very heavily by OpenGL. The Depth Buffer especially is for ordering and face culling on your polygons, where the Accumulation Buffer has traditionally been used for compositing, and the Stencil Buffer is for drawing containment where you want to just scissor out a portion of the screen and just draw to it or remove it or do whatever. This is just kind of a little simple overview to illustrate some of the issues that we have with the buffers of OpenGL.

So what do you do with auxiliary buffers? There's three main things you can do. You've got damage repair, you've got state saving and rendering to texture. Now this is, again, by no means an exhaustive list, but these are the common things that most people do with them. Damage repair means that if you move, say, a cursor over a particular section of your OpenGL context and you need to get the background there, you need to get that back behind it, what you would do is you would save off a little chunk over in your auxiliary buffer and then blit it back across whenever the cursor had left that area.

State saving: If, say, you wanted to disconnect your OpenGL context from your window and then put it back or play QuickTime movie through there or load something else, you could disconnect, save off your OpenGL, or before you disconnect you save off your OpenGL state, you read it over into the auxiliary buffer, disconnect, and when you reconnected it, it's going to clear that buffer, you blit it back from the auxiliary buffer, and you've got your state back. The final thing that you can do with it is rendered texture. Now there is still a There is still a pixel copy involved with using auxiliary buffers to render the texture, but it works very well and it's a whole lot less complicated than AGL surface texture.

So, how are you going to actually render to a texture with the auxiliary buffers? Well, what you do is, once you do all your drawing, you can go ahead and use a copy text image or copy text sub-image to read out the particular portion of the buffer that you want to read back in as your texture, that you want to specify as your texture.

It's very easy to do, but like I said, it still has a pixel copy. The next method is AGL Surface Texture, which is a little more complicated, but there is no pixel copy involved in it. Finally, I did want to mention Arb Render Texture, which There's an extension that has been developed by R, but we do not currently support it, As I was going through the methodology of the aux buffers, this is it. You'd want to specify your draw buffer as glAux0. I specified aux0 because it's the first aux buffer.

I believe our limit on auxiliary buffers is 4, so you could do 0 through 3. Once you have that done, you do all your drawing and you render whatever you want to in the aux buffer, and then finally you set your read buffer to the aux buffer as well.

AGL surface texture is a little more interesting. This one actually takes a separate OpenGL context and then the texture target and the format. The first parameter is the main context that will be using the texture. The second and third parameter are your texture target and your internal format. Finally, you have the surface context that you want to be using as your texture.

So the methodology of it is very simple. You create a P buffer, an off-screen context, to use as your texture. You perform your drawing operations and then you call AGL surface texture using the aforementioned parameters and then you specify the surface context as the context that you want to use as your texture.

So, AGL surface texture is Jaguar only, so it won't be terribly useful until after Jaguar is released. The Xuri buffer solution, as I said, is available on Mac OS X only, and it does still require a pixel copy. The surface texture does not have a pixel copy, but it does require another OpenGL context to be present.

This one's simple. Any of you guys were at Jeff's demos, you'd have seen this one before. So basically what he's doing is the context on the left is the surface that he's drawing into and then he's using that as a texture on all the faces of the cube on the right.

Now this uses AGL surface texture, this does not use aux buffers. Aux buffers can do the same thing except you wouldn't have the second context. So I think at this point I'm going to bring Sergio back up here and he's going to go over the roadmap and handle the Q&As.