Good morning. Thank you very much for coming. Welcome to session 505, OpenGL Integrated Graphics 1. One of the highlights of JAGWR is how OpenGL takes a fundamental role in graphics and imaging for Mac OS X. This is a one session of a part of two discussing techniques of how to utilize the OpenGL API to achieve high performance and creative possibilities for 2D, 2.5D video and sprites, etc. To present session 505, OpenGL Integrated Graphics 1, I would like to welcome to the stage Apple OpenGL engineer Geoff Stahl. Geoff Stahl Thanks, Sergio.

As Sergio mentioned, what we're going to talk about today is really the lead-in to how to use OpenGL with Mac OS X. We'll show you some new stuff, we'll show you some stuff that's been around, but really this is kind of the meat and potatoes of, you have an image, you want to display it, you have a texture you need to get into the system, you want to load a texture, you want to display a movie, you have a QuickTime VR, you want to display that. So we're going to talk all, kind of, touch base on all those areas. So in the second session, Ken's going to take you into some more detail of some high performance issues with this and how to get the most maximum performance out of the system.

So with that, let's move on. That's me. Again, first we'll talk about why would you want to use OpenGL for 2D, plain video, 2.5D, kind of the forced perspective kind of thing. Why is OpenGL a good solution for this? Then we'll go into OpenGL integration by talking about images and movies.

So why do you want to use OpenGL? First, it's the fastest pipeline in the system. As you've seen, with Course Extreme, we've utilized the power of OpenGL and the graphics processing unit for the Windows Server. That's a 2D application, basically. It is a layered application, but really, the fastest way to get that to the screen is through the horsepower of the GPU.

We've talked about it yesterday. The GeForce 4 Titanium has 63 million transistors. The PowerPC only has about 10 million transistors. The PowerPC is a general purpose computational unit where the GPU is specifically made to handle bits, handle graphics, and move things to the screen in a very fast way.

On that GPU, you have up to 10 gigabytes per second of memory bandwidth. You're only at about 800 megabytes per second in the CPU. So you look at that, and that's a factor of 10 as far as that's an order of magnitude more internal bandwidth. So if you can get your image, your texture, your memory, and your graphics to the screen, that's a factor of 10. So if you can get your image, your texture, your memory, and your graphics to the screen, that's a factor of 10.

So if you can get your image, your texture, your graphics to the screen, that's a factor of 10. So if you can get your image, your texture, your graphics to the screen, that's a factor of 10. texture, your movie, onto the GPU, you can utilize that power and really take advantage of the system, or the advancements in that GPU. We also offer with OpenGL a flexible API. Some people think of OpenGL as just 3D, but it is an open graphics language. It allows you to, if you want to, just pass values in 2D coordinate space.

You can transform your view area to window coordinates and really just work with it like a 2D API, using textures to upload your information. Also, obviously, hardware acceleration. ATI and NVIDIA have done a tremendous amount of research over the last years, along with 3D Labs and Matrox, and they're really pushing ahead. If you went to the Compositor talk yesterday, you noted that Peter Graffagnino spoke about Moore's Law. Processors were seeing about an 18-month double in power for general-purpose processors.

For GPUs, we're seeing about 6-month turnarounds. We're seeing a really, really accelerated advancement in graphics processing units. Take advantage of that. Utilize an OpenGL. Lastly, there's tons of great sample code. It's documented very well. It's an easy API to get into, take a sample, manipulate it to what you need to do, and go from there.

But why not OpenGL? Wouldn't be a fair conversation if we didn't talk about both sides. First, OpenGL accelerates 16 and 32-bit pixel formats. It doesn't handle 8-bit pixel formats very well, and it may not be the optimum solution if you have a special 8-bit case. Let's say you're writing a game that you have a custom engine that all it puts out is 8-bit index color.

OpenGL may not be a good replacement for the entire game engine. You may want to then put it into your back buffer, then use OpenGL or the accelerated Windows Server, and bring that to the screen that way. So you're utilizing the power of the graphics unit, but you have your custom code that manipulates your pixels in that 8-bit format.

I would say if you are working with 8-bit graphics, you may want to think about that you have a lot more memory, you have a lot more bandwidth, and utilize the power of the accelerated graphics, bring out the 32-bit, 16-bit, what have you, and use that for your applications.

Also, if you're looking at anti-alias fonts, complicated anti-alias lines, some of the things that we get shapes, the shape handling of a PDF. You may not want to use OpenGL actually as your drawing mechanism. You may want to draw it into a buffer. And as we'll show you later, use that as an image that you can bring to the screen. So you can composite both a PDF that you may have rendered in Quartz 2D with OpenGL, combine that with some 3D graphics and put that to the screen.

That's something you can do, but you may want to keep your Quartz 2D graphics in Quartz 2D. It is a large API. As I said, there's a lot of sample code, but with about 500 or so entry points at this point, it is a little daunting to learn.

And most people, when they start with OpenGL, they kind of pull up a sample, start writing, change some stuff, and guess what? They get a black screen. They work on some more stuff, they get another black screen. So take the sample code and work through it, make small changes, and you'll see pretty quickly, understand how the state machine of OpenGL works and how OpenGL works as an API.

So what are we going to talk about today? For images, what I really want to get across is rendering texture anywhere. So what you really have up to this point, people say, well, I have to load into a texture and then draw from that. Well, my graphic isn't a texture. It's an image, or it's a file, or it's a movie. So how does that relate to a texture? We'll go through that.

And then also, what are you rendering to? Are you rendering to, you want to get to a file? You want to get to an image? You want to get to the screen? You want to get to an off screen? We'll talk about the differences there. Now, in Mac OS X, we've integrated in such a way that it's very, very simple to use all those different targets of a texturing. We'll talk a little bit about image display and how to, if you do have 2D content, you want to get it to the screen, how you can do that.

I'll talk about a special case of planar texture handling. So instead of the chunky pixels, you have a planar texture. So you can get an image organized in a planar fashion and how you have to render that in a special way. And that's a case where you don't want to go through the image and actually swizzle all the pixels back together. Then I'll finish up with movies, talk about QuickTime VRs. QuickTime VRs and OpenGL have a really interesting integration.

The fact that a QuickTime VR is a special case of a QuickTime movie with the Cubic VR having six faces. And OpenGL also has a Cubic texture map, so you can, excuse me, map Cubic VRs very easily. So you can map Cubic VRs very easily. to the Cubics in QuickTime. And then lastly I'll talk about movie playback integrating QuickTime with OpenGL.

So for images, we're going to talk about render to anywhere, texture from anywhere, image display, and then player images. We've covered what we're going to talk about there. We'll move on. So render to anywhere. The two places you really want to render to are a surface or an image.

Everyone normally knows about rendering to a surface. That's where you, if you set up OpenGL and you render to a window, that's a surface. You're rendering to what you would consider just a normal, this is how OpenGL works. We'll then move on to how do you get that to an image.

So render to a surface, normally use AGL sector-allable, or maybe an NSOpenGL view if you're using Cocoa to actually get the image to a surface. This seems normal and ordinary. We get a lot of requests, people saying, "Hey, how about P-buffers? How about this off-screen stuff? How do I get accelerated graphics to an off-screen?" Well, in Mac OS X, the integration is such that really, you're doing exactly the same thing. You simply don't show the window.

So if I'm in Carbon, for example, and I just create a window, pixel format, attach my drawable, have the OpenGL contacts attached to it, and I don't call show window where I build a window that is off-screen, it works as you would expect. You get fully accelerated graphics to that off-screen window. And with Mac OS X, we allow you to manipulate that in new ways we'll show you in a minute.

One caveat here is, if you have a single monitor situation, it's pretty easy. You bring the window up. You know it's going to attach to the right graphics card. You know everything's going to work as expected. In multiple monitor situations, though, you have something called a virtual screen. Both AGL and Cocoa have methods to set the virtual screen to ensure that your content tracks with the window.

For example, let's say I have an off-screen window, which is just hidden, and it's on the main monitor. So that's where it was placed or what graphics card it's using. Then I have my secondary window that I'm using, I'm drawing something into that and taking something out of that window. I'm using it somewhere else. If I switch graphics cards, the user switches graphics cards, drags the window somewhere else, I need to make sure that it tracks, that the graphics accelerator tracks with where my target is. And I'll show you that in a minute.

So, let's move on to texturing from an image. So it's pretty simple to texture from an image. This is kind of your standard thing if you have OpenGL content. A lot of folks say, "Hey, I have an OpenGL screen. I want to make a movie of it. I want to take a snapshot of it.

How do I do it? It's some strange format. I can't get the back buffer." Well, it's actually almost easier. You're going to create a buffer, which is the size of your image, real simply. Height, width, and bit depth. Then you're going to read from the back buffer, setting the buffer to the back, and then using GL read pixels to simply read the full accelerator back buffer into your buffer you just created.

I'm not going to go into really depth on how to do this. In the game solutions section, Todd Prevital will actually show a demo of this. He'll talk about how to do this and even extend this into taking a series of snapshots and turning it into a movie. If you do have OpenGL content, you have single image content and you want to pull it down, it's very simple to do. It's just a few lines of code.

So let's talk about texture from anywhere. I'm going to kind of stretch the idea of texturing to file loading, texturing from an image, so moving an image into a texture, and then actually talking about surface texturing. So texture from a file. This is kind of the meat and potatoes of how to get textures in from any kind of file you have into an image, and then you can texture from that.

So I'm going to use the QuickTime image compression API in this example. It is really a little bit of setup, really powerful. Anything that QuickTime supports, you can support, and it's a really great thing because, you know, when QuickTime supports, it has TIFF support, it has JPEG support, or QuickTime has Windows and Windows Media support.

You can bring that in without an additional code. You don't have to handle, we get a lot of questions about, hey, how do I read a bitmap, or how do I read this? Just use QuickTime, read it in, you can do lossless, bring it in in a lossless way, and you get the image at full quality.

So I'm going to get a graphics importer for file. So we'll assume in this case that I have found a file that I like, and I want to get the graphics importer to bring this in. What this does is it actually does kind of a smart search. It looks at the file extension to say, hey, do I know how to do this file? Is it a .jpeg? And also, if it has to, it'll look at the file type and the file content. So if it's just a file named My File, it'll actually look in the content and try and find its decompressor for that file.

I get the natural bounds here. What that tells me is actually the size of the file, the size of the actual image, so I get the right buffer in the background. The row stride in this case is pretty simple. It's just the width and height, but if you notice, there's a target depth.

The target depth is how deep do I want my texture. I can read, in QuickTime, I can read it into any depth thing. So I can take a 32-bit image and read it into a 16-bit buffer, or vice versa. So I make sure I know where I'm reading it into.

Usually, you may want to keep textures corresponding to your screen depth, or you may want them at 32 to keep the accuracy. The base address, I'm going to use that new pointer clear. Again, basically just create a buffer and that's going to be the size of the image, row stride, and the top and the bottom of the image.

So let's build a back buffer for the image. You'll see this a lot. You'll see it in Ken's presentation. You'll see it in my presentation more than once. Qt New GWorld from Pointer. Why Qt New GWorld from Pointer? Why not some other routine? Why not some Cocoa routine? Why not New GWorld? The key here is this routine creates a buffer that is unpadded.

So there's no extra row byte padding. They didn't decide that, hey, we may want to keep each row 16 byte aligned or something like that, so we'll throw some unknown padding at the end. The key here is this is a packed pixel buffer, and you can hand this directly to OpenGL, and OpenGL will directly texture from it.

Works like a champ. So I have two cases here. One case is 32-bit, one case is 16-bit, and the only difference is actually the pixel format here. Again, this is the key. If you're bringing in a 32-bit image you want to put in 16-bit, you would use the second case there.

So now we're going to set a scaling matrix. In some cases you have an image that you really don't want a texture from that native resolution. For example, we'll talk a little bit about texture rectangle and texture 2D, geotexture 2D. Geotexture 2D had limited to two powers of two texturing, so you get 16, 32, 64. In that case you may want to bring an image in and actually shrink it down to fit one of those powers of two.

You can do that with the scale matrix in QuickTime. It's very simple to do. You set up a texture width divided by the image width, so I'm getting a ratio here. The height of the texture divided by the image height, again a ratio, and I'm not doing any kind of other transformation. So this is a simple 2D transformation. When QuickTime reads it in, it will scale it.

Interestingly to note, this also works for movies. Very similar kind of calls for movies. If you have a movie and you want to scale the movie on draw, this works. And QuickTime does a very good job of scaling. When it does a movie. It sets up the scaling matrix, does an optimized routine.

So I've used this significantly during my movie code, and it really is, you don't pay a penalty for actually doing movie scaling. It's not doing some kind of floating point scale that it calculates every single frame. It sets up a table lookup, and it uses that to scale the movie.

So this is the meat of this routine. Graphics import setGworld, so we're going to tell it to point to the right G world. We're going to set the quality to lossless. So this is saying, hey, I want to compress the image when I'm reading it in. I just want to read it in exactly the format.

So if you have a bitmap, you have a TIFF file that actually doesn't have any compression, this will make sure your image in memory does not have compression either, or it just doesn't have any loss in bringing it in. I'm going to get the Pixmap and make sure I lock it. Always make sure you lock your pixels and you check when you lock it. Graphics import draw, which will draw it into your G world now, so now your packed buffer of G world has the pixels you want.

Unlock the pixels, close the component. Now what do you have? What is in that G world? That G world has an ARGB pixel representation. Two cases: 32-bit and 16-bit. 32-bit is 8 bits per component, alpha, red, green, blue. 16-bit is 1555. One bit of alpha, 5 bits for red, green, and blue together.

So remember this when we get to other things. The key here is that you have a format now that OpenGL can understand. Using a Packed Pixels extension, which I'll bring up later, you can pass it directly into OpenGL. No pixel swizzling. You don't say, oh, I've got to get RGB, so I've got to take the AR component and move it over and pack it all in and change my bounds. No. This works very easily right out of the box.

So let's talk about texturing from an image. So now we've loaded the image. So we have this nice image in, we can texture from it. How do I do it? The easiest thing is to use texture rectangle. This was introduced in, I believe, 10.1, and it still persists in Jaguar. And the rule of thumb is, if you check for the texture rectangle extension, and it exists, use it.

It will provide the fastest method of access for the cards that support it. For example, the Rage 128 does not support non-power of two textures, which is what texture rectangle means. Most other cards do support texture rectangle. The GeForce 2 MX, GeForce 4 MX, GeForce 3 and 4, and the ATI Radeon 8500 all support texture rectangle, which means any size texture can be passed in.

It's really nice when you have an image or something, or for example, in the Windows server you have a window that's not a power of two. I mean, we're trying to play content in the Windows server that's not a power of two. A DV stream may not be a power of two. Thus, texture rectangle is key to getting this stuff out.

So here's how you check for the extension. GL_EXT_TEXTURE_RECTANGLE. Do a string comparison on the extension string. Make sure that you have that extension. Once you know you have the extension, you can enable texture rectangle just like you would enable a 2D texture. So if I'm texturing from OpenGL, normally I would call GL_TEXTURE_2D. In this case, I'll call GL_TEXTURE_RECTANGLE_EXT.

So now I'm in a texture with the correct texture target. So we have our image. It was in the buffer, and I called GL text image 2D. So this is actually pointing GL from the buffer, from the image buffer, telling it to do whether it needs to load it or just understand that that's the current texture. Texture rectangle EXT. It's an RGBA format. It's OpenGL's internal format that it's going to handle.

And the next line is pretty critical here. GL BGRA EXT GL unsigned in 888 reversed. It's kind of complicated to read and understand. What actually it means is the reverse at the end, the rev at the end, applies to the BGRA. So if you reverse that, you get ARGB 8888, and that's the format the Mac supports normally and you have in your image. Obviously, if you had done a 16-bit, it would be 1555 reversed. Very key that allows OpenGL to take that format natively, manipulate it without you having to do any pixel swizzling.

One key with using texture rectangle: when you look into some OpenGL sample code, one of the mistakes people make when they move to texture rectangle is not understanding that the texture coordinates in texture rectangle are actual image coordinates in the texture, or textual coordinates in the texture. So instead of wanting to use a full size of a texture, normally I would do 00:11 as a texturing coordinates. In this case, I'm going to use 00:320x240, or 256x256, or whatever my texture size is. So make sure you note that when you're switching over to texture rectangle.

Let's move on to talking about some other techniques for using it. So now texture rectangle is not available. What do I do with an image now? The simplest thing you can do is scaling. So we saw that QuickTime has a scaling matrix when you read the texture in, and you can scale it to any size. So, hmm, don't have texture rectangle, just do power of two textures. Well, I'll take the simple path out of this, and what I'll do is I'll scale it to 256x256, which is a lot of work.

So what I'm going to do is, as you see on the image on your left here, is squeeze down to a size that OpenGL can manage without texture rectangle. When I draw it, all I'm going to do is draw it to a polygon that's actually the correct aspect ratio for the original image.

OpenGL will take care of linearly filtering that texture and scaling it up. There is data loss, though, because you are actually squeezing it down, and you are stretching it back out. So realize that this is not the most accurate way to do things, but for a lot of cases, this may suffice. this may suffice.

So, how do we, how do we scale? Basically, this is in sample code in the OpenGL image sample. I'm going to walk you through it a little bit. Basically, what this is going to do is take a texture, a texture dimension from an image dimension. So, if you pass an image dimension in here, what's going to happen is, you're actually going to find out the largest texture size that this supports at the power of two.

So, if you can see, it says if it's larger than the max texture size supported, we're going to return that. Otherwise, it uses a, it uses a shift and shifts texture dimension down until it finds an image that is the largest size that can be supported for the, that can be scaled into.

What's another option? Now I don't want to lose pixels. I don't want to squeeze it down and lose some pixels when I stretch it back out. This is image slicing. Most engines that support-- game engines that have their own UI that support kind of a 2D presentation of things will use an image slicing algorithm.

Basically, what here you're going to do is, if you see the green lines represent actual polygons that are being drawn. So in this case, with this image, it's not a power of two image. So I'm going to slice it down to a number of polygons using an algorithm that finds the largest texture that I can fit in the space that's remaining. So if I have 320 remaining across the top, I take a 256 slice out. I subtract that out, look at what's left, and then take the next largest texture slice that I can do out of that.

Key here is when you texture from these. We saw that you actually pass a buffer into GLTextImage2D. What you need to do with that is actually offset that to the top of each texture you're texturing from. Pretty simple if you know what your offsets are, because you do have a packed buffer, you know what your row stride is, you know what the number of rows is, and you can just offset in as you walk through the textures. Use the image width for row stride so that will never be the same because OpenGL needs the entire row stride, not just that piece of the texture, to texture it from. And you need to use unpack row length to set this.

So what we're going to do here is we're going to find the number of textures and generate texture objects for that. So texture_action, texture_y are two routines that I'll show you in a minute that actually generate the number of textures, x and y, from an image size. Then we're going to use new_pointer_clear to generate a bin for the textures. This may be something like-- not like 1,000 by 1,000, but we're talking about something like 5 by 4. So maybe new_pointer_clear will generate 20 textures.

Then what I do is set the unpacked row length, as I showed you, to the texture width. So this is the texture width for all of them. And then I gen_textures the number of textures I need for the image. So what this is doing is this is basically saying I need to store five texture names, 20 texture names to texture from this image. I need to generate them, and I need to store the width of that. Moving on, this is going to texture from each slice here.

So what this does, it gets the next texture size. We talked about this already. It takes what's remaining, the 340 or whatever you have remaining, and finds the largest texture that will fit in that slice. In that case it would be 256. It then offsets the buffer pointer, as I mentioned previously, to the start point of the texture.

We then get the texture size for the height. We bind to the appropriate texture. We're going to walk through them in order. And you've seen the familiar GL text image 2D. Using texture 2D, because remember we're not using texture rectangle in this case, so we're just using the power of two textures.

You've seen the two BGRAs, unsigned int888 reversed and unsigned short888 reversed. And then we'll offset X and Y to the next texture. So this is, you just walk through the textures. When you draw them, you basically do the exact same thing. You walk through the textures and draw them.

So this is two of the routines I said I would mention. First, get text number from texture dimension. This basically is going to tell you how many textures. If you look at what it does, it starts with a very large texture. If the image is that size or larger, it just increments "i" and it walks through every texture without breaking out of that and adds through how many textures. So it says, is there an 8000 size to 8192 size texture? If the image is larger than that, it subtracts that from the image and says, is there 4000 left, is there 2000 left, is there 1000 left, etc.

through the entire image. So it's real simple. It just tells you how many textures and x and y. Again, all the sample code is in the OpenGL image sample which is currently posted on the web. This gets the next texture size. So this basically says what part of the image do I have left, what's the max texture, and I want to tell you what the next texture size is going to be.

Okay, you have a G world. We've talked about this before. We read your image into a G world, and now you want a texture from that. So what do you need to do? You want to use Packed Pixels. Because the software renderer has not supported Packed Pixels in the past, I would recommend No Recovery, which tells your implementation that you do not want to have a software fallback for your case. You want to just allocate the hardware renderer on whatever card you're on.

Then you need OpenGL version 1.2, so you can check the version string, or check the GL Apple Packed Pixels extension, which should be supported across the board on every accelerated video card. What that allows you to do is use the BGRA EXT in the reversed format. You've seen this before. Standard OpenGL texturing. You're going to texture from the G world that you have, as you saw before.

Okay, so texture from NSImage. Assuming you have an NSImage that you want to texture from, it's fairly simple. It's fairly close to the same thing. In this case, you're going to get a TIFF representation of that image, so you actually have a flat representation. We're checking to see if we have alpha, and that basically determines the OpenGL internal format. NSImages have a slightly different format than GWorlds, so in this case we're going to either do an RGBA for OpenGL internal format, or RGB, and we need to understand what pixel size we're dealing with, three or four bytes per pixel.

Then we're going to get the number of bytes per row for the image using that method. We're going to get a pointer to the bitmap data, which is basically the same as our buffer pointer. We get the image size. So, unpack row length. Again, that needs to be set to the number of bytes as the length of the row.

Set the alignment for pixel store, so we're doing a byte alignment. And then text image 2D call, very similar. Image width, image height, and image data. But notice we're using RGBA and unsigned byte as the format. Whereas GWorlds are in that kind of reversed ARGB packed format. This representation of the image is going to be an RGBA representation of the image. And so you want to use the RGBA when you pass it in from an NSImage to OpenGL. And then we're going to release the image rep that we got for the TIFF image. Pretty simple. Works well. Carbon works well in Cocoa.

Okay. What are one of the cool things that we can do? One of the cool things we can do is texture from surface. This is new for Jaguar. People have been saying, "Hey, we want that accelerated P buffer so we can render into that and use that to use somewhere else." We could render a reflection into an off-screen and then use that to texture somewhere else. In this case, we're going to use texture from surface and we'll show you how that works.

First you need to pick the appropriate texture target. Texture from Surface works on whatever texture target your card supports. It works on all of our accelerated video cards, so if you have a Rage 128, you do have this capability. But, remember, the Rage 128 does not have texture rectangle, thus you'll have to use texture 2D. And as I said before, the rule of thumb here is to use the best available. Assume texture rectangle is better than texture 2D, use that if you have it, and use the best available. That will allow the card to use its most optimum path.

In Carbon, the call is AGL Surface Texture. You pass your AGL context you're currently using. You pass your texture target. You pass RGBA8 or RGB. And then the last parameter is another context. So you have two windows. You have the context for both windows. You use one window as the source for the rendering into the second window.

In Cocoa, Create Texture, you'll pass a view into that. So you pass an NSView, and you'll have to pass an internal format, the RGB. to pass an internal format, the RGBA8, is the Cocoa method. So, let's look at a demo of this. Let me bring up the demo 2.

Okay, so what I have here is really just a cube spinning with a ship being drawn on it. I mean, what's up with that? That's not very exciting. That's easy. Anyone can do that. Well, the key here is the image on that cube is being rendered somewhere else. It's being rendered hardware accelerated. This works on my PowerBook as well as it works on here. So, to prove that to you, let's bring a wireframe up.

So now I just switched to wireframe rendering for the original source texture. So the ship is now being rendered in wireframe and I can switch back off to the normal rendering. So, let's bring that window up. Remember I said all it is is hiding the window. There's a source surface. So, I have this window here.

on my mouse. I have this window here, and it is the source for the spaces of that cube. Obviously, this is a simple implementation. You can do much more with it. You can genie it down there. It still works. It's not a problem. I bring it back up, and it's still working. I can drag it around. It doesn't matter if it's off-screen or on-screen.

The key here is that surface is the source texture, and you just created that just like you created any other window. You don't have to create a p-buffer and understand that extension. All you have to do is create a window of context. I didn't show the window, and I said, hey, HL surface texture, instead of calling text image 2D, I call the HL surface texture call. It directly substitutes in, and you can do things like this.

You can do things like stretching meshes around it. You can render a movie onto a mesh and then render that object into another scene. You can do all kinds of things that are two-stage dependent rendering here is what you get. And again, this is new for Jaguar. Bring the slides back up.

So now let's talk a little bit about image display. This is a pretty simple topic to cover, but it eludes people at some points about handling pixels and getting everything set up correctly. If you want to display images in OpenGL, what you're going to want to do is a parallel projection. Normally you think of OpenGL as a 3D API where all you're going to do with it is do a projective projection.

In this case, what we're going to do is we're going to set up a parallel projection, so an orthographic display ignoring Z. So we're going to use a painter's kind of algorithm. Whatever's in the back is going to be in the back, whatever's in the front is going to be in the front. This is very similar to how the accelerated Windows server works. Very similar algorithm.

We're going to use polygons to scale images. So your polygon is always going to have the appropriate relative ratios between the height and the width. And you're going to use the polygon size to actually scale the image. The texture size will never change. Texture coordinates will never change. So texture coordinates stay in the corners of the polygon and you actually move the polygon to scale the image.

If you wanted to, you could use texture coordinates as you could keep the polygon your full screen or your window size and you could use texture coordinates to manipulate the area of the image. I think the polygon scale is a lot simpler to think about, but may not be applicable to every single case you can think of. And we're going to use the model view matrix as you would normally in OpenGL to offset and scale.

So what that's going to do is if you want to move, if you want to rotate or move your viewpoint, you're going to actually do a translate or rotate on the model view matrix and simply to scale your position. So let's hop right into a demo with this. this.

One of the things that we talked about before, at the very beginning, was I tried to make a plea that says, use the power of the video card. We get emails, we have discussions, there's been discussions over years and years about how's the best way to get things to the screen. We get requests a lot to say, hey, can I get access to that frame buffer? I want to push pixels to the screen. Because I can write a really fast AlteVec blitter.

As we showed in the beginning, and we'll show tomorrow at the performance thing, we're getting 400 to 500 megabytes per second across the AGP bus. I don't think I couldn't write that in Blitter. I don't think many people could figure out a way, unless they're really working with the OpenGL drivers or working with the graphics drivers, to actually write a software Blitter that does that.

The second part of that is your Blitter may be great, but it's CPU intensive. It is using every ounce of horsepower of that PowerPC to get those pixels to the screen, because you're touching every pixel, whether you're using AlteVec or not. This is using the GPU. It can use texturing from AGP space. So it can actually texture in a way that the GPU is directly DMAing that from your memory. So you're not touching it. You're not utilizing CPU continuously to move those pixels around.

Also, once the texture's on the graphics card, you can see outstanding...

So let's take a simple image. I just picked a generic image. And I'm going to overlap the textures and use a tiling. So, what I'm going to do here is turn on client memory and AGP memory, which John will talk about in a performance demo. I bring this image up. And what's interesting here is that this image actually is in my system memory. I have a buffer.

The graphics card has gone, "Hmm, you have a buffer. You told me that you want to keep track of that buffer. You don't want me to keep track of the buffer." Okay, I'll remap AGP, the point to your buffer, and you can take care of it. So at this point, there's no extra copy on the graphics card. Where the graphics card does the optimal solution for all cases in this thing. We can see if I rotate the image, I get 700 frames per second rotating the image. And that's a 400 by 600 image. And I can do things like I can scale and zoom.

You can go full screen, I can move the image around, and I'm still getting 500 frames per second. So we can see that by loading this into the graphics card, You're relying on OpenGL to do these transformations, and these are things that you would have to work really hard in the CPU to do.

So let's skip to the little small image. That was 400 by 700. I mean, that's--anyone can do that. Let's bring an image in that's slightly larger. This image right here is 75 megabytes on disk. Actually, let me start it loading, because it does take a minute to do this.

And so that's rotating at 368 frames per second. And you say, nah, that's not a big image. That's not that big. There's no way I can be doing that. So if you were doing this on the GPU, that's like 36 gigabytes per second if you're walking every pixel. That's what the pixel-- what the data throughput you would have to average to manipulate the image.

So how can I prove to you that it really is that big? Oh, let me stop it from rotating. That's the image at real size. So that's one-to-one image. It's 5,900 by 4,400 at 32-bit. That is a 100-megabyte image in this card. It's all loaded onto the CPU, scaling, zooming, dragging, no problems. We're at 10% of image size there to fit it to the screen.

The key here is that you can manipulate very large images on the GPU without any CPU overhead. Actually, I'm going to take a second to actually do that, if I can. I'll have to search for the... I'm going to bring up the CPU monitor. And I have not done this with this particular image, so I have no idea what this will show me. But I'm expecting that it will show... Okay. So we have some mouse moves, but here, this again, let's zoom into an area, well, let's leave it at this size. So this is rotating at 400 frames per second. There is some CPU overhead.

Let's, uh... Really, we're using maybe half a CPU, we're using like a full CPU total, we're using half the power of the machine to get, you know, 36 gigabytes per second of effective data throughput that you would have to use completely with a CPU. So there's definitely a lot of overhead to do other things here other than just rotate the image. Let's go back to slides.

Go ahead and back to slides. Thank you. So let's talk about planar textures a little bit. So what are planar textures? Planar textures are, instead of chunky pixels, which we talked about either are ARGB or RGBA in the system, depending on whether you're in Carbon or Cocoa, these are organized in a set of red pixels, a set of green pixels, and a set of blue pixels.

I don't have an image to do that, so what I've done in the sample program, I'll show you in a minute, is actually when I load it in, I split it up, so there's no way you could pass that into anything in the system right now and actually display it. But OpenGL can handle that very easily in a simple method. We're going to do a multi-pass technique here. And I'll show you in the demo that this really doesn't affect your overhead unless you're severely fill rate limited on your card.

So first thing, we're going to use three textures. We're going to use a new type of texture format called GL luminance 8. What that says is I'm going to take that 8-bit representation of red pixels, so it's every byte, and I'm going to use it as a luminance value instead of using it as a red value.

And I'm going to do that for all three of these textures. And when I set up the textures, I'm going to offset the buffer pointer to the start of each plane, as we did with the slicing. I'm going to do this with the three sections of textures and offset the buffer pointer for each one.

I'm going to bind to the textures as appropriate and draw with a colored polygon. So I'm going to use a primary colored polygon to actually colorize what I'm drawing. So I'm going to use a bind texture 2D, the texture name, I have three textures there, 0, 1, and 2. And in this case alpha, so 0, 1, 2, and 3.

I'll use color 4f here to set, as you see this is in the red case, I would have one red, zero green, zero blue, and one alpha. So I'm setting this actually to completely be red for all the areas that are max luminance value. And then I'm going to use what's called a color mask.

The first layer drawn, I'll just draw normally, I'll draw normally red so it knocks out all the background. But then the color mask says, "Hey, only draw the pixels that have a value that is in that color." So in this case, if I set it to red, I'm only going to draw the red pixels in the image.

So even if I have green and blue pixels that end up in a luminance, for example, and have a gray scale value, they're not going to be drawn. So I'm only going to draw the red component of that. So it masks the components, all the others are passed through. So, a simple demo of how to do planar, doing planar textures.

Okay, if you notice, that's one layer. That's the red layer of drawing the chameleon, still shot of the chameleon. So we can easily switch on in this case. I added the green layer in, and added the blue layer, and you see the original image. Let me zoom out to the window.

So that's the original image of the chameleon, and by simply telling OpenGL not to draw certain layers, I can get to the full image. So that's a simple technique for using OpenGL for planar images. A lot of people get data in planar images and want to know how to do it. So you just use a color mask, you draw in three separate passes, works very well. And again, performance. How's performance on this? 330 frames per second, rotating the image. Whole image is up on the graphics card.

And it's not the fact that we're doing texture loading here. That's not what I want to show. What I want to show is that doing the three passes on this image doesn't really fill rate limit this card. There are some cases that you could have an image big enough that three passes on this would be a problem.

But the Windows server, again, does multiple passes when it needs to and uses the same technique. There are techniques you can do to compress this to one pass. But with unextended OpenGL in a simple case, this would be the simplest. solution. Go back to the slides now, please.

So let's talk about QuickTime VR. I'm not going to go into the infinite detail on this. I'm going to have some sample code up that shows how to do this. But basically, a cube map is six faces of a texture. All six faces are connected together: positive x, positive y, negative x, negative y, positive and negative z.

And it's very similar to a QuickTime VR cubic texture because it has the same faces. They're not in quite the same order, but they're there. So what we're going to do to render a QuickTime VR movie, we're going to initialize and open the VR movie, set up the movie and count the frames, read the VR frames, and in the front, right, back, left, top, and bottom, and then we're going to render the cube map.

There's a lot of code here, and again, this will be a sample posted. But some of the key points, as you look down here, is standard kind of opening movie. You're going to get the track count on the movie. You're going to get a media handler for it as you step through each track. And what you're looking for is the video media track. If you notice the QuickTime VR, QTVR, QTVR type, and the Panorama type, we're going to disable those. We're looking for the video media type.

And then what we're going to do is look for, usually there's two video media tracks. One is a preview, a very small image. One is the normal image. So I'm going to look for the largest one. Video media track, largest image, that's the track we're going to look at.

Now we're going to open the movie. So we're going to use whatever NavKit file or whatever way you want to open the movie. We're going to open the movie file, new movie from file, we're going to close the movie file so we have our movie now. This is kind of standard QuickTime and it's in a lot of QuickTime examples that you can look this up.

So now we need to set up the VR movie here. We set the play hints, we want to get the highest quality. We obviously want to get the natural bounds right, because cubic textures need to be square textures, and we need to know the bounds to appropriately set up our textures. So we're going to do that, set the movie box up to that, get the width and height for storage later.

We're going to make sure we know what the texture size is and then do the power of two. So if we need to do a power of two change on the movie, we'll do that. So we'll actually scale it or whatever into power of two. If you add a texture rectangle, you could also do a non-power of two in this case. I'm sorry, you can't do a non-power of two. It needs to be power of two.

So again, you've seen this code before. Set the matrix up. So we're going to set up a scaling matrix on this movie. And we're going to scale it to the appropriate size as we read each frame in. Use a set movie matrix instead of set the image decompressor matrix.

And we're going to set up the off-screen just as we did before. I'm not going to include the code here, but this is the same off-screen setup, Qt New GWorld from Pointer, as you've seen before. So you're using that same kind of routine, same technique. We're going to set the movie GWorld to point to that off-screen we just created. And the last thing I'm going to do is check to make sure the frame count is correct.

So we're going to set the frame count to 6. There are cylindrical VRs. Those are the VRs you can't really look all the way up or look all the way down. There's an earlier format. Most VRs these days are in cubic format, but you want to make sure you have six frames there.

This is count the frames. This actually is a little routine that the QuickTime folks helped me out with. The key here is the middle routine there: get movie next interesting time. A very interesting thing, but basically that's going to look at every time that a frame is being redrawn. In this case, it really doesn't matter what the time scale of a cubic VR is, but there are six frames.

They are sequenced, and what you can do with that is you can step through interesting times. You don't care how far apart they are, because it's kind of arbitrary for QuickTime VR, but they are going to be separated by time. And this is what this will do: move to the next time and the next media sample.

So this is reading the VR frames, and this is kind of the meat of how you do a cubic here. As you notice that there's texture types for negative Z, positive X, positive Z, negative X. All those are enumerated in the cubic texture map extension or type. So it's not just one type, it's actually a type for each face of that cube.

What I've done here is modify the order of these types to be the same order as the QuickTime VR stores it. And then I take my target and I'm going to texture into it from that buffer that I just got. Same code we showed before. So all this code kind of builds on itself. And you can use the same code you used for one thing for this.

So this is kind of the get the next frame. So we move frame by frame. And again, this is the key. The meat of this routine is the movie get next interesting time. And we're going to set the movie time value. Calling movie task at the bottom for all movies is what draws the movie.

So actually, again, think of a QuickTime VR as a sequence of six frames that actually show you, or time separated by an arbitrary time that you don't care about. In this case, we're going to draw the cube map. We're going to draw six faces. We're going to draw four coordinates per face, both the texture coordinate and the vertex, and we're going to enable cubic texture maps before. So, this is a little demo of this.

So this is based on a demo that was done by NVIDIA a long time ago, back in the G4 TNT2 days, I believe, actually, when this demo was done. But the thing we've added here, I added the QuickTime VR reading to it. So you can take any QuickTime VR and display it. So you can see that the performance of the QuickTime VR is very nice. You get very good performance. And also, the quality, you can actually use this in. You can see that the quality of the QuickTime VR is outstanding.

is that you can do any QuickTime VR. So your content does not have to be limited to, I have to create two special images or I have to package it up in a special way. You can take any QuickTime VR image, add it in, and you can see that it's a... gets reflections on both sides of the bubble. And it's pretty impressive for a very, very simple technique. I mean, this is really simple, just a few lines of code. Back to the slides here.

So the last thing we're going to do is build upon these techniques and show movie playback. This will lead into Ken's discussion later in the next session, which will actually take this to the next level and tell you about optimized movie playback. So, you've seen a lot of this before, so I'm going to kind of skip over the parts and just mention them, the things we've talked about and built on already.

Qt New GWorld from Poynter. We've seen that. That's going to allocate your pack to buffer. Texture format: the movie can be read directly into a GWorld. Thus, you have a packed pixel format that you can directly read in. We saw that with BGRA reverse and the 8888 or 1555.

QuickTime synchronization. One thing you need to do with QuickTime is realize that QuickTime needs to draw that frame and needs to know the interesting time, so to speak, of when that frame was updated. In this case, time does matter because you're playing a movie. So, if a movie is at 20 frames per second, you don't want to just start willy-nilly texturing from whatever buffer you have. You would like to limit your texture uploads to actually when QuickTime updates the frame and has new information. So, this is a way to do callback to synchronize that. And drawing lastly, text subimage 2D.

It's your friend in this case. GL text subimage 2D basically says, "Hey, I updated an area of a texture and I want to just upload that texture." So, in this case, the example you're going to see doesn't use texture rectangle. It's a 256 or whatever power of two texture you want to use. And it's only updating the movie area every movie update frame. So, in this case, it may almost be half of the data that you would normally update.

So, pretty simple. I think we've seen almost exactly the same code. Amazing. It's the same code again. Continue G World from Pointer. Make sure you have a buffer the right size. Set the G World. In this case, you also do set movie G World. Make sure the movie's pointing to the right place.

Hey, I've seen this before too. This is exactly the same code. Texturing from a buffer in a G world, you're going to use a BGRA format with the two, depending on 16 or 32 bit, depending on what you have. So let's talk about QuickTime synchronization. This is the key again for this area. You're going to do a movie draw complete UPP and then you're going to take a movie completion proc. And what that's going to do is that's going to basically tell you when to update the texture. The completion proc in this case is a movie text update.

So my movie completion process is really simple. All I'm going to do is take, set the long RefCon, which in this case is my, my, should I update the texture, and set that to true. So basically every time I complete a frame, that actually I've drawn something into it, not just, not just, this is actually a QuickTime frame that's actually been drawn, I'm going to say, hey, your textures need to be updated, or your texture's dirty.

So how do we draw? Looks pretty similar. We've seen this before. So if I need to update the texture, I'm going to... I'm sorry. If I need to do the texture update, I'm going to use the text subimage 2D. So I'll reset the texture, and that updates the area that the movie is actually updated, the actual bounds of the movie in this case.

I'm going to use texture width and texture height. And then I'm going to move the frame wherever I want to move the frame to. So I do any of my manipulation, any of my translation rotation. And then I'm going to simply draw a colored quad that is the movie and is textured correctly. So let's do the demo.

So in this case, I'm going to scale everything to 512x512. I'm going to use 800x600 display initially, and I'm going to use a 32-bit back buffer. I am going to use PacPixels as we've shown, because if you... all of Mac OS X uses PacPixels. With our previous OS, in a lot of cases, you had to do an extra texture copy. This case, at QuickTime and OpenGL, can tie in directly to each other and can directly texture from that same piece of memory.

So what this is, is just a simple movie. And you've probably seen this commercial before. Some of the keys here though are that your texturing, if you can't read it, the upper number in the corner tells you frames per second. It's about 350 frames per second. So OpenGL is drawing full screen using QuickTime, having QuickTime update the movie into about 500 by 500 texture and drawing that at 300 frames per second. That's pretty good. That's not a bad frame rate. I think we can play almost anything really smoothly.

One of the keys here, again, is only update the movie when you have to. You don't need to update the movie every time you want to draw a-- update the texture every time you want to draw a frame. What I'm going to do here is I'm going to put kind of a lock on it that says, "Hey, this--I'm going to draw every single time I need to draw a frame with OpenGL, I'm going to upload that texture." In this case, you see it dropped to about 80 frames per second. So that's the difference between leaving the texture on the graphics card and only manipulating it when you need to, which--it may not apply to just movies.

If you have--if you're drawing, for example, a--a sprite engine that you have a set of textures that don't change frame to frame, put them on the graphics card and leave them there. You don't need to upload them every single time. If you do that, you're going to cut your--cut your frame rate because you're dealing with a bus bandwidth every time. In this case, I went from 300 to 85 frames per second just by saying I need to upload that texture every time. And what you can--what I'm--as another thing, I'm going to go to synchronous. I think I'm going to go synchronous.

Okay, oh yeah, there, okay, that's right. In this case, I'm synchronous now, and it's 24 frames per second. So that's the native frame rate of the movie. It's playing at 24 frames per second. I'm now updating it at 24 frames per second. It's not bad, it's kind of a little chunky.

But again, if we take out that synchronicity and allow OpenGL to manipulate and draw every single frame in there, and then allow the movie to update when it needs to, you get much smoother animation, you have a much smoother convincing playback. And the things that I haven't shown here as a simple demo, things you could do with this, you can stretch this texture around an object.

You can twist the texture. You can put a grid behind this and twist it in any way you want. You can paste it on a wall. You can have a control panel in a game that has a movie play on it. And you can just put the texture in the right place. You can do effects with it. We talked about programmability yesterday.

You can do a sepa-tone movie with a pixel shader. And, I mean, all these things are possible using the power of OpenGL, integrating it with the graphics. You can do effects with OpenGL, integrating it with QuickTime, integrating it with QuickDraw, with GWorld, integrating it with Cocoa. Really simple to do. And if you notice, a little bit of work on this code, a little bit of investigation will yield dividends because you'll be able to integrate with almost all the graphics in Mac OS X.

Let me bring the slides back up, please. So what do you learn about this stuff? You learn about it from two books, from the web, from the tons of examples that are out there. Two books I'm saying, if you're working with OpenGL, definitely get these two books. The red book and the blue book, they're affectionately called. The red book is the programming guide, and that's kind of a step-by-step, let's get introduced to OpenGL. Take you through a whole bunch of different options, take you through the OpenGL end of things, where I showed you the kind of the system integration end of things.

And the blue book is just a reference manual. The blue book is a reference manual of every single OpenGL call. The blue book covers calls that are actually in the spec where there is, we've shown some extensions today, the texture rectangle extension and some other extensions. Those are available on the web. The best place to get for extension is the extension registry, which can be accessed via the OpenGL.org website. And that shows you all the extensions.

Tomorrow's advanced 3D talk will actually go through all the new extensions for Jaguar. And you always can get the extension list just by using some of the GL info stuff that we have built in. Also, if you're building OpenGL, we invite you to subscribe to the OpenGL mailing list, list.apple.com. You can search for OpenGL and find that. And lastly, developer.apple.com/opengl, where we'll put updates and information that we have as it becomes available.

Okay, so what did we cover? OpenGL is not just for 3D. OpenGL can be used for 2D, and as we've shown with the accelerated Windows server, OpenGL makes a great 2D platform to build your application on. Whether you're displaying data, showing movies, doing the latest game, OpenGL can be the engine behind that. You can do all of your data manipulation. You can do your hard math.

You can do your gravitational effects. And then when you want to show that and put it on the screen, let OpenGL do a lot of the work for you. Because it has a hardware accelerated GPU behind it, which has tremendous power, more power than you can even get out of the CPU, and it frees up that CPU to let you do other things, other great things with it. Sample code will be up on the samples. And that's all I got.

Let me talk about the roadmap a little bit, and then we'll take Q&A. Right after this session, in this room, Ken Dykes is going to take this a step further and show you really how to get to the high horsepower, how to really do these things in a very interesting manner as far as getting to max performance and acceleration.

There's color sync and digital media later today. Tomorrow we have game solutions, two game solutions. And as I mentioned, in the first game solutions, we'll talk more about using that read pixels, saving it to a frame or saving it to a movie, and show you how to get your OpenGL content out that you can put into a movie and send it off to your best friend.

That afternoon we have Advanced 3D, which will talk about all the new extensions, take you through kind of deconstructing the Wolfman. How did they pull off the magic in that demo? OpenGL Performance and Optimization. If you're doing OpenGL, go to that session. We'll show you some new tools that will really make your life much easier.

Graphics Imaging and Performance Tuning Friday and following last session of the conference, come and tell us what you want, what you need, and we'll try and get it into the OS. Sergio will come back up and moderate the questions here, and that's his contact information if you have specific questions about our OpenGL implementation.