Thanks Sergio. Today I'm going to talk about how to use the OpenGL pipeline to get real impressive 2D performance out of the platform. This includes both 2D or traditional viewing pictures, those kind of things, and 2.5D, maybe some work with sprites, work with movies and those kind of things. We'll see how you can really use the OpenGL pipeline to display those 2D kind of raster images.

So first we'll talk about why are we talking about OpenGL, a 3D API for 2D work. Then we'll move on to implementing 2D with OpenGL. Things I'll talk about first is how to display images. We'll talk about how to load images into textures and use textures to image from. You can use this, extrapolate this to using anything you generate from your program. We've seen demos before in the past with using PDFs as a source of a texture and texturing that onto an object. You can do the same techniques there.

We'll move on to talk about sprites and side scrolling. I use side scrolling kind of loosely. With OpenGL, obviously, there's no limitation on whether you're going to use it for vertical and horizontal scrolling or you're going to create the next great game, the next recreation of Defender. And then we'll talk a little about QuickTime integration and beyond using just the QuickTime API for loading textures, how we can use the QuickTime API and Movie to integrate with OpenGL to display them.

But I would be remiss if I didn't talk about why not OpenGL. What things about OpenGL may turn you away? Where may you not go down the right pipeline here, so to speak, by thinking about using OpenGL for your application? First, it works very well in 16 and 32 bit modes. But it does have some problems if you're trying to get high performance in an 8 bit mode.

The 8 bit mode is going to be, you're going to fall into the software case and in a lot of special cases, we have special, you're doing a palette animation for example. If you're doing palette animation, OpenGL doesn't have a real equivalent of doing 8 bit palette animation, so I suspect that your application could be specially coded to handle that in a case better than OpenGL. So that would be a case that you may want to not think about using OpenGL and use a special case of your own. Also, let's say you're doing fonts.

You want to have very scalable fonts. You want to do a specific font. You want to do a specific kerning with fonts or handle Unicode fonts. OpenGL doesn't have any built in properties to do this. You could combine Quartz and OpenGL to do this, but really if you're trying to do like the next InDesign or PageMaker or those kind of applications, you probably want to think about using the Quartz 2D API because that really has a robust feature set for handling images, handling fonts and those kind of things. While OpenGL really is where the high performance bit images can be handled.

So what are we going to talk about? First, we're going to hop into image display, talk about texture loading, and talk about orthographic display and scaling. Then we're going to talk about sprites with alpha blending and alpha test, and talk about using some matrix manipulations to move them around the screen. Then we'll work that right into the side scroller, showing a large scaling background with sprites flying around on that. And lastly, we'll talk about QuickTime integration.

First let's talk about image display. One of the key things here is how to load textures, how to get any texture or any image you want into your system. We're going to use a QuickTime image compression API. It's a great API for loading images. You can bring up a Nav Services dialog and select any image you want, use QuickTime to load it. You didn't have to worry about whether it was a BMP, a TIFF, a PIC, a JPEG, whatever. It'll load it for you. It'll put it in a buffer you can use.

Some of you who've looked at this may say, "Well, hold on. If I use new GWorld for that buffer, I have some padding problems." I'll show you how to get around that. We'll show you how to use any buffer that you can take. You're going to take a non-padded buffer and you can use that directly for texturing to OpenGL.

With that, you want to use PacPixels. The PacPixels extension or the PacPixels that is in OpenGL 1.2 drivers will allow you to use a texture format directly from the image to an off-screen G world to a texture without having to do any pixel swizzling whatsoever. So you don't have to touch those pixels unless you want to do some special effects. The three methods we'll talk about is first, the simple way. OpenGL must have a power of two textures currently.

So what we're going to do, the first way we'll talk about scaling with QuickTime to a power of two texture on load and then fixing it as we display it. Next we'll talk about segmenting the power of two textures. A little bit more complicated, where you take a large image that is of abstract size, any size you want, and you segment it down so you get the pixel perfect representation of that image in multiple textures.

And lastly, we'll talk about-- we're not going to really talk about it. I'll talk about it here. We'll talk about texture rectangle. Texture rectangle is an extension coming down the road that will allow you to take any shape or size-- not any shape, but any rectangular texture that is not necessarily a power of two in texture with that. All the vendors are working hard to support that. We're working hard to support that. So look for that in the future when you're talking about image loading and image handling with OpenGL. It will allow you to handle textures of arbitrary size directly into the OpenGL hardware accelerator.

I think I better take a drink of water for this one. The QuickTime Image Compression API. Let me actually back up a little bit with the code here. The code I'm putting up here will all be available in sample code. There's no code you'll see on the screen that you'll not be able to get a hold of.

So don't try and take specific notes exactly how to do it on this. Most of this stuff will be up before the end of the week. Some of it's up already, and the rest of the stuff will probably be up next week. So you'll definitely be able to get at the code, and look at it yourself, and look through it, and decipher it, and figure out what we're doing with that. So if you listen hard, I think I can go through and explain exactly what's happening here.

First thing you do is get graphics importer for a file. You have an FS spec from your Nav Services dialog or something that you create yourself. And you get graphics importer for file. It returns the GI comp variable there. What then I do is get the natural bounds of that. Basically tells me how big the image is so I know what I'm working with.

Second thing I do after that is decide on what the stride of that image is to create a pointer for it to hold that image. The reason I do this is that's a step you may not normally see. That step, creating the stride, tells you exactly, bit for bit, what's the width of that image without padding. And then that new pointer, I'm going to create a buffer that's exactly the size of that image without any padding, without any padding through a 4-byte, or anything that a GWorld may normally do. I then call qt new GWorld from pointer.

It's not a function you see very often, but it works very well because it will use that pointer I've created for that new GWorld, and you're supplying the memory, you're supplying the padding. It will say, I know the information there, I have the stride at the end, so I'm going to use that packed image buffer for the GWorld, and I'm not going to allocate any memory myself. That is key to using packed pixels and using direct texturing to OpenGL because that allows you not to have to skip rows, not to have to do-- strange things with the image before using it with OpenGL.

And you see there's two different versions here. The reason there's two different versions, and you probably could make it one with assigning that constant to a variable, but in this case, I wanted to show you that there is two K32 ARGB pixel format and K16BE555 pixel format. Those are open--sorry, those are QuickTime constants that indicate to QuickTime what kind of pixel format you're going to have in that GWorld there.

So lastly, we're going to do a graphics import set G world. Graphics import set the quality for lossless. We're going to get the G world pixmap. Of course, remember to lock that and check the value of your lock on the return because you don't want to draw into a non-lock pixmap. It's not good enough just to call that lock pixels. And then use graphics import draw, unlock pixels, and close your component. Now, in that pixmap that you've created is the image off the disk. No problems.

So what are some other things we want to talk about? Let's talk about packed pixels. How do we get that image into OpenGL? First, real critical, if you're using AGL, make sure you use AGL no recovery. AGL no recovery says, I don't want to create a software backup render to back up if the system runs out of resources for this. What that does in this case, it says, I want to respect the hardware's pixel formats only.

So if you're sending to a hardware, all you're going to do is create the pixel format. OpenGL is only going to use the pixel format for that one specific piece of hardware. It's not going to try and use a software backup, which happens to not support packed pixels, so it may convert everything to 32-bit. So if you have a 16-bit format here, you want to make sure you do that to keep it 16-bit.

You need OpenGL version 1.2 or the GL Apple Packed Pixel extension to support packed pixels. And if you notice, one thing you'll notice, the extension string is packed pixel, not packed pixels, which I think it's listed. And this is just a little caveat. So if you search for packed pixel, no matter which way we go with that, if it changes in the future, you always make sure you get it.

Because if you search for packed pixels, you may not get it in the future or the other way around. In any case, the key here is GL text image 2D with a BGRA extension and the unsigned in 8888 reversed and the BGRA unsigned short 155 reversed. The way you read that is the reverse of the pixel format applies to the BGRA. So it's ARGB. 155 or ARGB 888, which we recognize as a standard Mac pixel format. Thus, you can use these to directly texture out of your image buffer.

So scaling to power of two, what do we do with this? You notice I have the example images here. The first one's like a 256 by 256 compressed down to exactly what the texture's going to look like. What we're going to do with this is we're going to restore the aspect ratio by storing the image aspect ratio that we read in and setting the polygon's aspect ratio to the same image, texturing to fill the polygon, which will just stretch it back out. Again, we're losing information here, but the image appears correct on our screen. So scale the power of 2.

This piece of code here is actually pretty simple and maybe easier to look through it on your own. But really what I'm doing here is fairly simple. What I do is if the image size is-- I step through, I take my largest-- the largest power of two that I want, and I step through and remove that from... I'm sorry, that's incorrect. Basically what I'm going to do is I'm going to use the fact that I can shift things down a power of two at a time by using the shift registers. And I'll shift it down until there's nothing left.

When there's nothing left, I'll put a one in the first bit and shift it back up. And what that does, it tells me the largest power of two image that's nearest. And then at the end, I basically say, if it's closer to one than the other, I check the above and below. So basically I got to the point where I have a power of two that is going to be smaller or larger than the image.

So if I had a 249 pixel image, I shift it all the way down, and when I shift it back up, I got 256, because I have that one bit in one of the... I have a one in one bit of the image size. And then I compare that to that shifted to 128, and I see which is closer. So I'm getting the nearest one. So what this little piece of code does, it defines the nearest power.

So I can see the nearest power of two. There's probably different ways to slice it. I thought this was probably the easiest way to do it. And it's not real easy to explain, but I think if you step through it, you can see how it's working. And that's all. I'm just bringing up the demo machine, and I'll show you this in action here.

What we're going to do is we're going to choose a file to open here, and this is going to open an image. And it's just going to display it in OpenGL window. If you notice, it's a standard window, standard image. Remember this image does not have the complete image quality that we had originally. If you look at the text, which may be hard to read up there, but I'll tell you what it says. It's a 640x480x32 bit image.

That obviously isn't a power of 2. We've put it into a 512x512 texture. And then we've rescaled it out so it looks exactly like the right image. If you want to see the actual polygon, I can zoom it out for you. That's actually a single polygon with a single texture, 512x512 stretched back to the correct aspect ratio for the image. One of the nice things you see here is high performance 2D, right? That's what we're talking about. This is high performance 2D.

Using the OpenGL transforms, no problem rotating the image. And if you want to zoom in, we can do that too. Very easy. We'll go to full screen. Still get really nice transforms, real easy to manipulate images using OpenGL. Using OpenGL as your 2D transform. But you say, you know, really I want all that information that, uh, we can go back to the presentation now. I want all the information that I had in the original image. I want to have all that, I don't want to lose information. Oh, let me back up.

If you notice, if you look carefully at the screen saver in Mac OS X, OS X uses the same technique. It translates all the images to 512 by 1024 textures and uses that. But you said, man, that screen saver looks great, great detail. You're losing detail. If that image is 768 tall, it's squished down to 524.

If it's 900 wide, it's stretched out to 1024. So it uses that to get consistent images and to have easy texture loading. So that's an actual use of this in a 2D environment. And you can see then they can get the fades using the OpenGL hardware acceleration instead of having to do some kind of fade themselves, which is going to be real hard and real taxing for the system.

So the other option we have is segmenting to power of two. You don't want to lose that. And this is an interesting technique, a little bit more complicated, but it uses the same basis as the other ones. In this case, what we have is you can see the image there, and the green lines indicate the actual polygons that comprise that image.

And I've segmented this image to about four by four as far as using it to display. And so I've kept all the pixels directly in one texture, and then I'm sliced out of that texture for each image. The way I do that is when you call GLTextImage2D, you provide it with a base address.

That base address can be manipulated. It doesn't have to be the beginning of your buffer. So what you can do is provide it with the beginning of every single segment of that buffer very simply. The other thing you can provide is OpenGL has a constant called unpacked row length.

And normally, the width you pass into GLTextImage2D is the width that it uses. Obviously, if you were taking a piece out of this, that wouldn't make any sense because it would be all skewed. It would try and take the next pixel over and stick it on the end of your image so you get this kind of skewed image that came across your screen.

We'll set unpacked row length to actually the width of the entire image, vice the width of that little piece of texture we want. So what it'll do is for every row, it'll skip that much. That's like its image stride. It'll skip that much and come down to the next one, grab a piece of it, skip the entire rest of the row, and grab the next piece. So it'll have that nice lined up pieces of the polygons when it actually tries to draw it. So we use the image width and the unpacked row length, and then the texture width will be some smaller subset of that.

OK. Segmenting to power of two. What we're going to do is this first black box statement up here basically tells me how many textures I have in the image dimension. Same technique we kind of used before. Basically what I do is I say, OK, I know that if I look at the bitwise representation of that dimension, I can just kind of count the bits that are actually turned on. So if I look at the ones and zeros, I can go, how many ones are there? And that's the number of textures I actually need to use.

It's real simple. Works very well when you realize that that's a nice little thing to have. I then do a new pointer clear for texture names. And that's texture x by texture y. So I'm going to create an array that I can store my texture names in. I'm going to use pixel store i, as we talked about, for the image width, for the entire texture width in this case, the entire texture width.

And then I'm going to do a gen textures to generate texture names for each texture I need for that image. So in that case, you saw those are 4 by 4. I'm going to generate 16 texture names. Then when I draw the image, I'll use gl bind textures. I'll bind each texture in turn. And I'll draw the image that way. So let's look at actually getting it into memory.

Another little black box function we have here, that I'll show you in a few minutes, is get next texture size. It takes where you are currently in the image and says, what's the next size? What it does is I say I have a max texture size. This is what I was talking about earlier. I got confused. I have a max texture size I want to use, 256, 512. Maybe I don't want to go up to 2000 for an image.

Maybe I want to keep it down to 128, so I have 128 segments, whatever you prefer. It steps through 128, removing that from the length of the image. And then once it gets past that, once there's less than 128 left, it takes the next biggest power of 2 off the image, and it keeps doing that kind of down. So it keeps doing 128, 64. Okay, there's no 32s. I'll give you a 16, and then whatever else you got left. In this case, we get the next texture size in the loop.

We go for X and Y for each of our textures. Our offset is 0 for the Y offset. That means we're going in the columns. We're going to start at the top of the image for every row. Above this, we're going to have the X offset is 0 at the very beginning. And what we're going to do, is we're going to take the pointer to the buffer, is the normal image buffer we got before from Quick Draw, when we used the QGNUG from pointer.

We're going to use the offset Y by the texture width, times the image depth, divided by 8 in this case, is shifted 3. So what basically we're doing is saying that this is going to offset into the image for that corner of every texture. So we offset across for X, offset down for Y for every row, and we offset in based on what the width of each texture we need to get. to use this.

Get the next texture size for the height. We've got it for the width, so we put in the previous texture size. We call bind texture. We do that bind texture. We pick the next texture name in our list of 16. And then we basically do the same GL text image 2D. If you notice, I'm using the P buffer, and I'll go back a slide.

The P buffer is that P buffer, not the P image buffer. The P image buffer was the original pointer. And the P buffer is the pointer to whatever specific texture. And we bound, we've told it the unsigned row length, unpacked row length is the width of the whole image.

And so we use the GL text image 2D at that point to grab the right texture out of the image. So we're kind of doing the cookie cutter. Take a piece, take a piece, take a piece, move down, take some more pieces, and we slice our image up that way.

This is a little snippet of code from the Get Texture Numb from Texture Dimension. Same kind of thing. First thing I'm going to do in this is I'm going to remove what are my max size of textures. I have max text size. I'm going to divide by max text and count how many max texture images fit in there.

Then I'm going to take whatever is left over, the Texture Dim I reset there, and I'm going to, is there anything of 8,000? Is there anything of 4,000? Is there anything of 2,000? Is there anything of 1,000? I just go down each one. If you notice, when you add all that up, it will add up back to the complete size of the image. So this tells me how many textures across I have or down.

And lastly here, get next text size. This says, hey, I got the size of my current one. Is it one of the first max size ones, or am I going to step down into my list and basically do the exact same thing, checking to see if I have an 8,000, a 4,000, a 2,000, a 1,000, like that.

So, how do we display these things? First, we're going to use an orthographic display set up to the scale of the image, the scale of the screen. So, the images you saw, I scale your display to be the pixel scale of the window. So, when you're manipulating them, if I offset three pixels, I'm offsetting three pixels in the window.

So, this actually is a good way to display it, so you're thinking like a quick, like quick draw, or normal, what you're used to thinking as far as, you invert the x, you invert the y coordinate, your x is, your origin is in the upper left corner, and at that point you're moving things in a pixel centric manner. I may ignore z, I don't need z, so I don't need to establish a depth buffer, and I ignore z for this one.

I use poly, I'll use a polygon to scale the image. When you saw me scaling it out, that image before, I used a polygon to scale, I used to manipulate the polygon positions, I didn't use any texture coordinates, I didn't scale my window at all, all I did was scale my polygon size, my texture stayed the same, and OpenGL takes care of the whole thing. It takes care of the hardware accelerated transform to get it onto the polygon. And you can use the model view matrix very simply in the, in this case, to offset and rotate.

So, as you saw me drag, I just do a drag on the mouse, I use a real, real nice Carbon events, tell me when the mouse drags, I figure out what the start point and the end point for the mouse drag is, and I just drag the image by that many pixels. My offset, since I'm already in pixel space, is exactly that when I apply it into the model view matrix.

Rotation, I get the rotation in degrees for the, for the turn of the mouse, and I apply it in to the rotation matrix of the, around the z-axis for the model view matrix to manipulate the rotation of it. And now we're going to go back to the demo.

And one thing about the multi-texture demo here is that I can load the same image I did before. If you notice this one, it's tiled in multiple textures. Let's expand it out. And I'll zoom in a little bit to actually see what it looks like. What you see in there is the actual pixel grid of the actual pixels on the screen, or actual pixels of the image. And you can see how the textures actually lie right on the pixel grid. So you can see that these come together.

Right on the pixel grid of the image. And OpenGL does a great job of stretching that image. I mean, there's only a certain amount of data there, but it does a great job of scaling the image. And you've still got great performance on this 640x480 image. But you say, you know, maybe I want something bigger.

So, I had to dig around to find a bigger image, but I did find a bigger image, and I haven't tried this one for a while, so hopefully this one will be okay. This one takes a while to load, and I kind of remember how big this image is. I think it's 2800 by 2100. So it's a pretty big image. If you look at the polygons, we'll zoom out a little bit. Those are five, the max texture size, nine by six, I think the max texture size is 512 in this case. So, 2800 by 2100.

Pretty nice. Oh, you want to rotate it? We can do the rotation too. That's easy to do. You want to zoom in to see the actual inference? Nice detail there for the image. So you can see OpenGL works very well for handling 2D images. One thing that might be interesting is, see if I can find one of these. We'll help ourselves.

People who've been working with OpenGL say there's a little problem with my technique, and there actually is. If I can find the corner. It'll show up right here, probably. You kind of can see it here, not very well. Actually, it's better than I thought. There is a discontinuity along that line. And there's probably better examples of it. Let me see actually if I can find a better one, because it's kind of a good thing to understand. Probably on the edge here we'll be able to see it better. Back toward the middle you say? I can see it.

How about down here? Okay, you'll have to think-- oh, there it is. Actually, you can see it right there. See across there? You can see right across here there's a discontinuity. The discontinuity is OpenGL uses filtering. OpenGL filters across the texture to-- when it tries to magnify and minify by a texture by texture I don't know what the best way to put it.

It looks at each texture separately. Remember, you've moved this image from image space that you had to the textures, and it looks at each texture separately and wants to filter it when it tries to do the scaling for minification and magnification. What that means is it looks at the next pixel over and says, what is the color of the next pixel? Well, in this case, it says there's no next pixel. It's the edge of the texture. So there's a filtering discontinuity between two edges.

So if you really want the perfect representation of this, something I didn't do in this demo, what you can do is take a texture border. You can take a border of one row of pixels from your image. So you make your textures like one pixel smaller and take a border around them so that minification, magnification filtering is more accurate.

Not something I did here, but you can do it if you want the perfect example of this. So again, this is scaled up like two and a half times, and you can see that it does a pretty good job on this stuff. So we'll go back to the slides. Okay, let's move on to talking about sprites.

Sprites are actually pretty simple. Sprites are just going to be textures handled by OpenGL. So we'll load the sprite frames and store them in textures. Well what are we going to do for sprites that have multiple images in them? Most people who want to use sprites want to handle multiple images in one sprite. Even if you want an animated cursor, for example, how would you do that? What you can simply do is have one texture that fits all of your sprites or some of your sprites in there.

And you can set it up so that your sprite frames are offset and you use the texture coordinates to offset into those sprite frames. So you load the whole thing in one texture and instead of like we did before where you're loading a segmented, that one image into a segmented texture, you're loading one texture and then using it as a segment, in segments by using texture coordinates every time you go to a polygon.

For example, if you had a polygon, if you had ten sprites, you would use a tenth of the texture coordinates for that one sprite and you can maneuver across depending on what sprite frame you're talking about. One thing you can also, most people want to do with sprites is they want to use some kind of masking. I want to use a mask.

They don't want to have just that cursor, just my cool animated cursor spinning around or whatever. They want to use some kind of masking. So how do you do that? Well, you can easily do it by passing an alpha channel or if you read an image from QuickTime, you may not be sure of the alpha channel. You may not be sure that's the exact alpha channel you want.

So you want to go in there and blast the alpha bits as something you know. In this case, I used a background color, pure green, and I looked for it basically to the green screen kind of thing and I just removed the green screen by setting alpha of that to zero.

I did this a couple times. And the first time I did it, I was like, "Oh, I'm going to do this." And I was like, "Oh, I'm going to do this." I did this a couple times. And the first time I did it, I came out with what's on your right-hand side here with that little green line around it. It was a pretty cool little effect, but again, texture filtering.

That is because it looked at the next pixel and says, "Ah, I know there's green there. You want me to filter from that green onto your image and that's what I'm going to give you." So what you can do is the other image, what I did was I said, "I'll put kind of a neutral gray in there." And the neutral gray allowed me to filter it to a reasonable color.

So that's something to just keep in mind when you're doing it. Something that OpenGL will want to filter that. And it will use both filtering on alpha and filtering in the image domain. So you want to make sure your filtering is set up correctly if you're using an alpha mask.

So let's talk about how to do alpha mask. Basically, you want to set the alpha channel on load. You want to look at the alpha channel once you load it with QuickTime, and you want to blast the exact bits you want into it. In this case, we're talking about the top eight bits in 32-bit format, or the top one bit in a 16-bit format. If you wanted to, you can use other formats, by the way. I'm not limiting you to these. These are just something that I'm using for the talk.

But you could use a 444 format. OpenGL would understand that, which would give you the four bits of alpha in a 16-bit representation of colors. Basically, background color pixels, we're going to set the alpha to zero. And we're going to set the pixel value of those pixels to some neutral background color.

For all other pixels, we're going to make sure the pixel is set to alpha. So in this case, in the 16-bit case, we'll OR that with 8,000 hex. And in the 32-bit case, we'll set the two high bits to FF to basically make sure they're all ones in the two high bits. You want to ensure the alpha is set. Because sometimes, depending on where the image comes from, I've seen images that I thought out of Photoshop that I had set the alpha correctly.

And it comes through a format. And when I look at it coming through into the machine, the alpha is not set like I would expect it to. So just make sure you keep that in mind, that you have a known source of your alpha, or you're setting it into what you want to set them to.

So, how do we do this? It's actually pretty simple. If it's a 32-bit image what we're going to do is we're going to get the pixel from the image buffer we saw before. We're going to step through for texture width and texture height through the whole thing. We're only going to look at the bits that are our color mask. In this case I'm saying green.

FF in 32-bit format is our color mask. So, what I'm doing is I want to ignore the alpha on these bits. I don't want to compare to 00, 00, FF. I don't want to compare to all 32 bits. I only want to compare to the 24 bits of color.

So, I'm not sure about that alpha. So, I may have a difference in alpha and it won't quite match my source color key. So, I'm going to look at the color bits. So, I'm going to mask those color bits out and if they equal that pure green I'm going to set the pixel to a reasonable gray in this case with a 00 in the high bits for alpha.

So, it will be blank and the filtering will look good. If not, I'm going to OR that with FF. And the reason I'm going to OR it with FF is to make sure that those two alpha bits are set at the top. Or eight alpha bits. So, there's two in hex.

Two digits are set for alpha. And then I'm going to loop through the next pixel. For the 16 bit case, very similar, in this case I'm going to compare it to the 7FFF to mask out that one alpha bit and make sure that 3 is 0 which is the same five bits of set for the green.

I set it to a neutral gray or set the top alpha So let's talk about alpha blending. So there's another way. You say now you have the sprite mask. You have your cursor that you move around. You've cut out the background in. We're going to talk about alpha blending.

Another thing you may want to have is some kind of transparent texture, some kind of feature, some kind of, you know, you have the heads-up display in a game. You want to just use that with alpha mask. Real simple to set up an alpha mask. In this case, what I'm going to do is I'm going to use the luminance value of this texture to set an alpha mask.

That's what this is an example of here. What I'm going to do is I'm going to take the average, which is actually I'll correct myself before anyone else does. This is not actually the luminance value. This is my poor man's luminance value. If you want to really do the YUV conversion and just take the Y component, you should do a correct YUV conversion. But in this case, I'm going to assume this is grayscale. All the bits are the same. And so really the divide by three is not necessary, but it looks good for the presentation.

So, we take our poor man's luminance value, we divide by three to get the average for 2.55, we're going to shift it up into the high bit and we're going to or it into the pixel. So in this case, what we're going to say is whatever the kind of gray scale value of that pixel is, is going to be the alpha.

So we'll get transparency that varies with the luminance. This is good if you get some artist that has some kind of background or something like that, that you just want the luminance value, you want to set an alpha that equals it, you can use this kind of technique.

How do you draw with these things? What are the key open GL, the unlock the open GL kingdom here? We get GL alpha funk. What I do here, this is the masking. We talk about the alpha mask, the first thing we talked about. In this case, we're going to say greater than .5 or less than .5.

If it's greater than .5, we're going to draw it if it's less than .5 or not. In this case, when we set everything to zero, we won't draw anything. When we set it to one, we will draw. If our polygon needs alpha test or if our sprite needs alpha test, we'll turn alpha test on.

For blending for transparency, we use a blend funk. In this case, we'll use source alpha and one minus source alpha. Basically, let's say the contribution of the front pixel is source alpha. It's the source's alpha value. The contribution of whatever is there already is one minus that. If your alpha is .8, you can see the source is 80% contributor. The background is about 20% contributor. We enable blending.

If we have transparency. Lastly, depth test. You may want to do a depth test. So if you have multiple pixels--

So let's talk a little bit about how to do a side scroller. Most people, when we think of side scrollers, we think of like 1400 by 480 kind of backgrounds that you play Defender on or run around in your 2D version of Quake or whatever you're doing.

And most people think of using a 2D raster engine to move these things around to the screen because we can compact it down, we have a lot of black space, we can use the parallax scrolling. I'm saying you can use OpenGL to get a much faster version of this with a lot less limitations on you.

What we're going to do is we use tiled images to form a large textured background. Pretty unlimited. You can even load the tiles. And I mean if you think of something like Diablo 2, it actually was a similar method where it loaded terrain, loaded sprites as it needed to. It was a forced perspective kind of 2.5D thing, but in reality it was still kind of the same idea of using an accelerated engine to drive something that could have been done years ago. would attempt to be done years ago with a 2D engine.

So we use the grid of polygons for the background. And there's a lot of different ways to draw the grid of polygons. We toyed around with it. We've been talking about it. And there's some different kind of techniques you can use as far as deciding what's on the screen and off the screen, depending on your limitations.

So I'm not going to try and tell you exactly how to do that, because I found out that depending on what your limitations are of your application, you can find some ways are really easy and some ways are very difficult. The idea here, though, is that you keep a grid of polygons.

You're going to draw something slightly bigger than the screen, or make sure you draw the polygons that are actually present on the screen. And you're going to draw the ones you need to as the person moves, whether it's north, south, east, or west. When they move that way, you're going to draw some additional polygons.

Basically, when you draw the gridded polygons, you're going to find out what your world center is. In this case, we just attach it to a sprite, and we pick that sprite's world center, and we offset everything else by that world center. So everything else just moves off the screen as it would. So you don't have to worry about some kind of strange transform you're writing yourself. Just use OpenGL's model view matrix and transform it as you would with the world center attached to one of your sprites.

We're going to rotate it by world rotation. So if your sprite rotates, you can rotate the entire world, or you can rotate your sprite, whichever way you'd like to do it. So there's no limitation that everything has to be kind of orthographically displayed across your screen, which, I mean, if you're thinking about it from a 2D standpoint, I'm not thinking that it's going to be real easy to take a 1400 by 480 image and turn it slightly. OpenGL can do that for you, as we've seen.

And you're going to scale by world scale. Most of the time, you can't zoom in and zoom out. In this case, you can zoom in and zoom out. Or you could draw your little radar map up in the corner of your game or application. Or showing how you're manipulating a large image by just showing a scaled version using mipmaps of that larger texture you've already drawn.

So how do we handle textures? You load the background as one texture or a set of textures. But basically you load a large image as a single large texture. You're going to segment it with the techniques we've shown before. And you're going to let OpenGL do the texture handling for you. We use GenTextures to generate texture lists as we've seen before.

And we'll let OpenGL handle the memory management in and out of VRAM or in and out of the AGP space. Because the OpenGL implementation on 10 does a very good job of handling AGP and it's going to get better in the future. And you really should take advantage of that fact.

Instead of trying to have your own what's in memory, what's not in memory, what's load something, what's remove something, just let it handle it for you. It'll push things out you're not using. It'll bring things in that you need. And I think you'll find it'll be very effective in managing those textures for you. And this is again the same as your segmented images. This is exactly the same technique we've talked about before.

So how do you draw this? In my case, I set up an orthographic context. I thought about it later and I thought it would be really cool that you could use a perspective context actually for your side scroller. And when you zoomed in and zoomed out, it actually would have that kind of parallax, weird zooming.

But I didn't have time to do that. But that's something to think about. So, I mean, if you really don't have to set up an orthographic context, you can say, think about it as your art workbench where you're looking straight down at one of those, you know, camera mounted above a table. You actually can change or use the GL you look at.

And you can change the field of view and the aspect. If you want to zoom in, you just change your field of view to be wider. You want to zoom out, you change it to be narrower. You change your near and far planes. And that kind of thing you can do and gives you a lot of the features that you really would work hard for doing it yourself with your own 2D engine. You can give it to you with hardware acceleration in OpenGL.

In this case, we're going to pin the world to a sprite of interest. Here's another good thing you can do. Let's say you have some kind of strategy game or something that you have multiple places of interest. You have a very large document you're handling. The user can jump to an area of interest very easily by just manipulating the center of that image. Do a lookup of your sprite information or do a lookup of information into your document.

You can jump to this object on my document. I want to see how it's positioned in my preview. They can do that basically by using readout where that position is and offset to that position without having to do some kind of calculation or having to move a lot of pixels around yourself.

What I did was to draw the sprites, I basically do a--you kind of grossly call the sprites to your visible list. You don't want to send OpenGL every single sprite that's off the screen in your massive area. But you grossly call the ones you think are close to visible. Let OpenGL do the fine work.

You may find if you spend way too much time on this, you may find that you're not going to get the results you want. You may find if you spend way too much time on this, you may find that you're not going to get the results you want. if you spend way too much time trying to replicate the transformations, trying to figure out exactly what's on the screen and off the screen, you're spending a lot of time doing that.

One thing I'll tout here is the OpenGL optimization session tomorrow with John Stoffel will look very closely at what OpenGL can do for you and what it can't do as far as optimizations. So come to that session, look at his example of optimization, and apply it to your same application. Make sure you know in real time what percentage of time you're in your application and what percentage of time you're in OpenGL.

Null out those drawing routines and find out if you run your application open loop, what actually is your possible frame rate. A lot of times you'll find that you're not really spending much time in OpenGL. You're spending a lot of time in your application. It doesn't matter how fast the graphics engine is, you're going to have to improve your application speed. Gain a strong understanding of the design and construction of optimized image display, sprite, and live scrolling engines using the OpenGL API.

I'll open my sprite file. There's a couple sprites here. I think the background image is 2000 by 1600. I have another texture in there that's 1000 by 1000, both 32 bit. And then I have my sprite images which are also they're actually 256 by 256 and I just used a scaling of OpenGL to do it. So we're going to run around and see what this thing can do. So as you can see I can move the rocket back and forth. I can fly around very easily.

And we can zoom in or we can zoom out. Real nice that you get a nice feedback. And if I can do this right, I probably can find Apple on here. This happens to be a photograph of Cupertino. Let me see if I can do this. Where am I? Oh, up, up, up. I think we're up there.

No, can't find Apple. I'll find it later. In any case, you can see what I did here for the poor man's version of tiling. I flipped the image. And I just used OpenGL to do the same image. You can zoom in. You can see that. And we got the cloud layer there.

So we have a nice layer of clouds that's flying over it. One thing you can do is this is your standard side scroller. You kind of fly around and scroll side to side. One thing you can do also is you can do this very easily in OpenGL, which works the same. It's doing the same amount of work. So I can fly around, in this case, in my standard method of the top is up. have a pretty convincing game and make you all sick at the same time. So we'll go back to the presentation now.

And the last thing we're going to talk about is QuickTime integration. How do you integrate QuickTime? And I have about 10 minutes, and I think I can get through this. A lot of this we've talked about already. Same techniques. This whole thing is using the same techniques. It's all based on the same stuff. It's not rocket science. It's not separate stuff. You learn some of this stuff once and apply it to all these techniques.

Qt New GWO from Poynter. We've seen that at least three times today. That is how you're going to allocate the texture to get a packed texture format without having any extra space to help not allow you to texture. We're going to use packed pixels for direct texture handling.

We're going to synchronize to QuickTime to minimize our update. What that means is we're going to give a callback procedure to QuickTime to tell us when it updated the frame. If we haven't updated the frame, we're not going to update the texture. What this means is the only textures uploaded to the card are the ones that actually got updated.

You can see, obviously, if you upload the frame every single time, it's like playing an uncompressed game. You can see how that can compare to a compressed movie where you're only really updating the frame every time it changes. If you have an OpenGL that can do 100 frames per second and QuickTime is updating the movie at 24 frames per second, there's no reason to update that texture. Let the texture sit in VRAM and then only update it when you need to. Drawing, we're going to use subtext image to--that's wrong.

...maybe wrong. In any case, we're going to do a subtexture upload. The movies usually are not 256 by 256 or 512 by 512. They're usually a 4 by 3, which is usually not the texture size. So we're going to take the subtexture out of that and we're only going to update the part that's actually updated.

Again, saving bandwidth to the card. So, what are we going to do here? New pointer clear, we've seen that. Qt New GWorld from pointer, we've seen that. Set GWorld, set Movie GWorld, we haven't seen that. So all you're going to do here is set the Movie GWorld, same as we did with the graphics compressor, and that will tell the movie to play into that GWorld.

Texture formats. Pretty simple. GeoText Image 2D, RGB, that would be an uncompressed format, and then you have the reversed format showing the actual packed pixel format. QuickTime synchronization. Here's something new. You look up there, you see MovieDrawUPP. That's going to give you a universal procedure pointer to a movie, to the movie completion procedure. New movie completion procedure, so you're actually creating it with my MovieDraw proc. And then you're going to set the procedure very simply for the movie. Gmovie is actually the pointer to your movie, so that's something you get from QuickTime.

and then in your completion proc, oh sorry, if you look at the call it says movie drawing call when changed. So it's going to call you every time it updates that movie into the G world. So QuickTime will tell you when it has something changed. This is the only code you have to do to let QuickTime know when you've actually updated it. You can see this could actually be very useful in some other things that use QuickTime so you know when there's actually something changed in the movie and you're not trying to update something all the time.

And then the movie draw proc, really simply. All I'm going to do is, in the RefCon, I have a global variable, gfMovieTextUpdate, and I'm going to set that to true every time I get to that procedure. So when I go down to draw my polygon, I'll get all ready to draw and I'll say, do I need to update the texture? If I do, I'll re-update the texture from the movie G World.

If not, I'll just draw the polygon with the texture. You can use the idea of current textures, because you can bind it to any texture you want. You can have multiple movies playing. You can also, in this case, the example you'll see, I just have one movie, one texture, so I just leave it current.

And how do we draw? GL text subimage 2D. I was right. It was wrong. It's not subtext images. Text subimage 2D. GL text subimage 2D. You'll see here, and this basically is the exact same thing you've seen before as far as using the same kind of call. You can use the 32-bit version or the 16-bit version. Then I'm going to move the frame. What that does is that's just my call to move the frame around the screen. However, I'm manipulating that QuickTime frame. That's only the frame of the polygon. I'm using the model view matrix to do this.

I'm not doing this. This is not manipulating the movie pixels itself, just the frame. And then I'm going to do a GL begin with quads. And I'm going to set the texture coordinates to the corners. I'm going to set the color to white for the whole thing. You can colorize your movie if you wanted to by changing the polygon color. And then I'm going to do very simple vertices. Use basically square vertices. And the movie is going to be inside that square. Let's go back to this demo.

I'm going to show you a couple different things here. First, in this case, we're going to bring out a display size of 640 by 480. This demo does not use a segmented texture, but you could easily integrate that into this demo. This uses a single texture that the movie is compressed into. 256 by 256 movie.

I'll use a 16-bit off screen. Full screen, I will use PacPixels. And I will sync to the VBL, which we'll see how that works on this. Actually, let me turn off sync to VBL, and then we'll turn it back on for the next run. I'll pick a movie.

Have you seen this before? And Mr. Schaffer, would you prefer window or aisle? So I'm getting 620 frames per second playing that movie through OpenGL. It's playing the movie full screen and then it's texturing as fast as it can for a single frame. So real simple there. Now we'll stay with this demo and we will In this case, we'll just throw some effects in. We'll use fog and we'll sync the VBL. And we'll just leave it like that. And we'll play it in a window this time.

And Mr. Schaefer, would you prefer window or aisle? Middle. So again, this is using-- Middle? : I'm shooting about 70 frames per second. I'm not sure about the sync on this because it seems to be varying more than I normally see because of the video system that's going through.

But in any case, you're seeing just using a simple fog, turned on linear fog for this. Real simple to show an effect with a movie. You get the full complement of OpenGL, cool stuff you can do with it without having to do any extra work. : It's a little movie I'm working on.

So where do you get resources for this? It's pretty simple. The Red Book and the Blue Book. If you're working with OpenGL, make sure you've got the Red Book and the Blue Book. The other thing that I think that everyone should have is in our OpenGL SDK you'll see the spec.

Look at the spec. I use that more than I use the Red Book and Blue Book. If you notice, if you look at the spec, the Red Book and Blue Book look very much like the spec. But it has a lot of, it goes into more math than many of us want to know sometimes, but it actually has the real what's behind the engine, how does it actually work, which is real important if you want, if you're trying to debug something like fog. You want to know exactly how the fog's going to work.

Or you're trying to look at how the texture, the unpacked row length. You want a good explanation of unpacked row length. It goes through exactly how it handles pixels. So I think the OpenGL 1.2.1 spec is a very good thing to have and all programmers working with OpenGL should have it. It's in our SDK. It's in a PDF format. You can print it out and, or look at it online.

Good places to know if you're working with OpenGL. www.opengl.org. A lot of resources. It tells you where the current standard of OpenGL is. It gives a complete list of the extensions for OpenGL. It gives you a lot of programming examples and starter code. List.apple.com. Search for OpenGL. We have an OpenGL Mac list. I want to see you all on that list.

It's a great place to get information, ask questions, and we monitor the list and try to help out when we can. And lastly, developer.apple.com/opengl. That's another good place to go. All of developer.apple.com. All of developer.apple.com URLs. Go to the OpenGL page. It's a pointer to the SDK, pointer to the newest things we got for it.

So, what did we talk about today? We talked about using OpenGL mainly for 2D and 2.5D, and we did some movie stuff too, but how OpenGL can be used to complement your applications that aren't interested in doing the next Quake or the next Tomb Raider or the next Nanosaur or Chromag Rally. We're talking about games that want to use just the display images, want to use the power of OpenGL to manipulate those images, and it's a really great thing that you can do it easily.

You can set up your transforms to let the transforms handle the workload for you and just let OpenGL do the hard work. And we will continue to improve OpenGL, we'll continue to work with the hardware vendors, get the best hardware acceleration we can, and that will give you, that will continue to improve your app even without you doing work on it.

Next, it does allow robust hardware acceleration. You're not going to get that any other way you slice it. You're not going to be able to get to the hardware acceleration unless you're going to use OpenGL or use one of our provided APIs. You just don't have the resources and the ability to get at some of the hardware acceleration unless you're writing the drivers, unless you're writing the OpenGL implementation.

And it really shows you a new paradigm to solve these real world problems. You don't have to say, "Well, hey, I use copy bits. I'm going to rotate that and I'm going to use that and that's going to be really difficult. I'll go and swizzle now, so OpenGL." Rotate your model view matrix, throw it on the screen, scale your model view matrix, throw it on the screen.

You can shift it over, move it around. It's real easy. It's just a couple of instructions. Sample code. The samples will be up very shortly. The movie sample is up right now. The image sample should be up by the end of the week. And the rocket sample should be up probably next week.

Let's talk about the road map. Sergio mentioned this is the first of five sessions. The sessions that you want to go to are right after this is geometry modeling. After that, tomorrow morning, OpenGL optimization. John Stauffer, the head of our OpenGL team, is going to be in here and is going to take you through optimizing an OpenGL app.

It's going to be a really great session. He has some really great tools to show you how to optimize that app, and I'll show you how to get the most performance out of an OpenGL application. Advanced rendering. Troy Dawson, one of our engineers in the OpenGL team, is going to come and show some advanced rendering techniques.

You'll definitely want to see that. He talks about using the stencil buffer to do shadows. He talks about anisotropic filtering and some other topics. It'll be a really good talk. We have the feedback forum for OpenGL at the end of the day tomorrow. Please come talk to us about what you want to see in OpenGL, where we're doing well, where we're not doing well.

We want to get your feedback. We want to take your stuff in, and we want to make sure that we improve it and make it the most world-class implementation of OpenGL and 3D graphics API anywhere. Lastly, the feedback forum for QuickTime, the QuickTime API, which I think you've seen is very beneficial to your work in OpenGL for texture handling and for movie playback.

There's a feedback forum for QuickTime also Thursday at 3.30. Sergio Mello is the technology manager for 3D Technologies at Apple. It's a great contact if you have questions about implementations, questions about where we're going, questions about what we have today to solve your problems. Please contact him, [email protected], with those kind of questions.