Hello, welcome everyone. Thank you very much for being here today. Hope you're enjoying WWDC this year with all the really exciting things that Jaguar brings and possibilities that Jaguar brings to you. We're particularly happy today because right here, right now, is the first time, first showing of Apple's implementation of hardware programmability.

The OpenGL team has done a great job in providing support for the mainstream programmable graphics cards currently in the market, but they have done a lot more than that. What you see today is a set of tools to allow you to preview, debug and optimize your shader code in real time.

These tools are available currently right now on your Jaguar seating CD and we hope that this will become a staple for you for hardware programming, certainly on the Apple platform, even perhaps on any other platform you wish to deploy your application. We are very excited about this first coming out and to welcome to the stage Mr. OpenGL Engineer Geoff Staahl.

We're going to talk about graphics programmability today across the spectrum from both vertex programming and pixel programming. First, just an interesting show of hands, who's actually written a vertex program or tried to even look at one? and more than I thought, maybe 20, maybe 20 out of the audience.

So what I'm going to try and do is I'm going to try and give you a good introduction to what vertex programming is, what pixel programming is, so you can go out and use the tools and experiment and put these new features into your applications and actually take advantage of the power that we have.

So I'm Geoff Stahl. Myself and James McComb will be doing the session today. I'll talk you through a little bit about graphics programmability. Then James will take you through a tutorial and I'll finish up at the end. What we're going to cover is, first we're going to cover a little bit of evolution of programmability in the graphics space. About 20 years ago, Cook wrote a paper called Shade Trees.

And that's kind of when everyone started thinking about programmability. It followed on, and I'll go into details in a few minutes, but we ended up where we are today with commercial graphics processing units on your desktop that can handle the programmability and do amazing things like we saw at the beginning. So, then we'll cover what is graphics programmability. Then vertex programming. Finally, we'll go to James and the Shader Builder tutorial and finish up with the fragment shading.

So first, let's talk about a little history here. 1984, and this is the same time frame when people were writing papers on how to blend two objects on the screen. If you came from the accelerated Windows Server, the same date, same year as that paper about blending. Cook wrote the paper on shade trees, which described a hierarchical shading language or a shading setup that allowed you to put multiple programmable shaders onto different objects and really set the groundwork in place for programs like RenderMan. Two years later, Pixar brings out RenderMan, which still today is the kind of benchmark by which all of their shaders or shading languages are measured.

Moving on, 20 years later almost, or 17 years later, Nvidia demonstrates, brings out the GeForce 3 and demonstrates the Chameleon demo you saw. And more recently this year, they demonstrated the Werewolf demo, which shows really desktop shading, desktop programmability really that is fully programmable and you can fully utilize. Up to this point we had some configurability, but really didn't have that programmability. So now it's on your desktop, now you can buy a GeForce 4 Titanium, get an ATI Radeon 8500, and you get that programmability onto your desktop.

So where are we going from here? What's the future look like? Well, this project is like the Stanford shader language. What that is, is trying to take a render man type of language and bring it down to the desktop being fully accelerated. Up to this point, when you, companies that would implement things like render man, what they would do is have a server farm out in the back that would just churn away at frames.

And so when, you know, Pixar or film company, what they would do is they, when you're not using your machine, they grab it and they use it as a rendering engine. Well now, your desktop can be your own rendering engine. You can do what you did 20 years ago that took a cray, days to do in real time on the desktop.

I mentioned Doom 3 here also. Why Doom 3? I think you'll be surprised in a few weeks when id shows it at E3 and then later in, whenever they release that, that you'll see that it really requires that programmable GPU on your desktop to get that good immersive performance. Up till now, games had, you know, you can add on this feature, add on that feature. You need a little more fill rate. But this would be the first thing that really requires mass market graphics programmability in the desktop to run it.

So we saw the chameleon originally. Why is the chameleon important? Well that was one of the first demos for the GeForce 3 and what it shows is a couple of different areas of programmability. When you look at it at first it looks cool and you think wow that's pretty neat, look at those effects. What are they really doing there? One thing they're doing is a skinned mesh.

All the vertices for the chameleon are passive static vertices. It has no movement per se. They're not manipulating a vertices sending it to the graphics card. They're actually sending one about 4 meg chunk of static data. Then they use a set of bones and vertex programs to actually manipulate that static data and they use a weighting structure to kind of stretch the vertices across the bones. And that's only available through using a vertex program.

It's also bump mapped as you saw. That bump mapping, that per pixel bump mapping requires a combination of vertex programs and pixel shaders. It has gloss mapping, so you have multi pass. It has that shininess on it. Also it has a cubic environment map. You may have seen the reflective one. It uses a cubic environment map and the chameleon itself to reflect the environment around it.

Further, it has a glass version that has glass in it. And if you also notice carefully, and you look and see the leaf in the back of this picture, you can see that the fogging, it's a very nice fogging. Things are fading out to the horizon. They're not just turning a different color. That's done again through a vertex program by manipulating the fog color. James will show you that later, how you can actually manipulate the alpha value as a way of fogging objects.

So for all those of you who have never looked at a vertex program, what is programmability? What is the difference between a traditional graphics pipeline and the programmable graphics pipeline? Transcription by Captioning & Subtitling @Captioning.com.au So if you're writing a vertex program, you need to understand that. You're taking world coordinates, world information, and you're transforming that into something that the rest of the pipeline, the rasterization step, can then use to actually light or color and draw those pixels.

Texture and Fragment Shader, on the other hand, takes that information in, has been clipped, has those pixels all stacked up there, and then does both texture coordinate address generation, which means where in the texture map that you've supplied is that pixel located, and then also does the combination of multi-texturing. So as you saw, like for example, the bump map gloss map, that is a function that requires multiple textures to be combined onto each other to get the shininess in one texture, the bumpiness from another texture, and the base color from even a third texture.

So where are we now? Mac OS X Jaguar, the seed that you all have, The target is, let me back up, Mac OS X Jaguar. The target for that is our vertex program. It's a spec that takes both ATI, NVIDIA and any other vendor's hardware into account and gives a one unified spec that you can write to. You don't have to write to something for ATI, something for NVIDIA, something for Matrox and something for 3D Labs. This brings everything together to give you one environment to write your vertex programs in.

From the texture shader side, we're not as lucky. Really, the hardware is still different enough that there's not a unified specification for these things. So what we've done, we've taken the best vendor specs on both sides and implemented those in the Jaguar. And these are all in your seed. So we have texture shader 1, 2, and 3 from NVIDIA, which is new.

We still have register combiners 1 and 2, which were there from the 10.1. And in ATI, we're looking to put fragment shader into Jaguar. It is not in your seed currently, but it looks like that's going to come along for Jaguar, and that'll be there as a more unified for the ATI hardware way to do fragment shading.

So, what's the hardware support? What hardware do you need? Well, the bad news is, to get the full hardware support you need a GeForce 3, GeForce 4, or ATI-RADI-8500. We can expect that. That's when the shadings come along, in the past year, year and a half. But the good news is, is we have an optimized software solution for vertex programs. So all you guys out there with a laptop, any laptop, don't care what it is, you can pop up the shader builder that James is going to show you later and be sitting outside after the presentation, not during the presentation, after the presentation, and start building shaders.

You don't need a GeForce 3, GeForce 4, or ATI-RADI-8500 Radeon. You can just do it on any machine you want, anytime you want. You can be on the plane flight back home and fooling around with that vertex shader stuff, experimenting and building cool shaders for your next project.

So our vertex program, this is new for Jaguar. As I said, it's a target for Jaguar. It's currently with the arb working group, means it's not a finished spec. So what we did was for Jaguar, we took a snapshot of that. And that's what you see actually represented in Shader Builder. It's not quite what the arb working spec will be, there are some syntactic differences. But the functionality is almost 100% there. The reason we did this is we wanted to get you something today that you could work with and then change the commas and the spacing tomorrow.

We'll be fully compliant with the spec when it comes out. There are 84 issues currently that were brought up. 76 are completely resolved. There are 8 unresolved. We feel real confident that in the next couple of weeks, those will be completely resolved. And you'll have a spec that you can work with.

If you're interested, once the spec comes out, like all other specs, it will be at the OpenGL.org and you can download it and see how it's coming and track it there. And as I said, Shader Builder will be updated in the final Jaguar edition. edition to be updated with whatever the spec turns out to be.

So why do you want to use vertex program? Why is it important to you to learn about this? Why do you want to use it? Well first thing, as we've talked about, one thing we're trying to do is transfer as much load from your programs onto the graphics processing unit. GeForce 4 Titanium has 63 million transistors.

The PowerPC has about 10 million transistors. There's a lot of effort, a lot of specified hardware, specific hardware designed to do this kind of operation in that graphics card. So why don't you let it do that operation? Instead of trying to take the fixed function pipeline and do a whole bunch of manipulations and kind of squeegee your data around and do some strange things to actually make it fit in, just write the vertex program that explicitly does and efficiently does what you need it to do.

That gets to the idea of efficiency. Some things it can do very well is handle non-standard lighting models. Don't like gross shading. I want a phong shaded model. Well you can write a phong shaded model, put it in your vertex program and there you go. Everything in your scene will be phong shaded. Appropriate information down and James will again show you something very similar to that in his demonstrations and I think you'll be pretty impressed. It's also going to do skinning. We saw the skinning part of it. We saw that the chameleon used skinning to manipulate the vertices.

Passed static data down to the GPU so it never had, every frame it's not sending down 40, 50,000 vertices. It passed it down once, put it in, used a vertex array range and then uses the bone structure inside a vertex program to manipulate that on the GPU. Very efficient. Okay.

The last thing that we can really do with it is, one of the things you can really do with it is parametric vertex control. If I want to displace every vertex by a certain amount from a certain point, I can pass that information into a vertex program and it can do that. So I can do a standard transformation, standard lighting, and then do some kind of displacement based on some kind of action inside your application.

Let's talk about the computation model. It's a vector engine. Every component of that is a vector operation. There are four components, 32 bits each. It works on a single vertex. Think of it as a black box that every single vertex that you pass to the GPU is passing through.

No vertex prim- no- no vertex generation is allowed. You can't take one vertex and generate two. And you can't put in three vertices or triangles or quads or whatever other primitives you want. It's a single vertex operation. It's also a state machine. Your application sets the inputs, passes the vertex down, it operates on that vertex. There's no interdependence. I can't take the output of- of the vertex program and feed it back into the next vertex.

That has to be handled by the application. Think about- if you want to think about it in a simple way, think about it as every vertex program works in parallel. If you think about it that way, you realize that there's no way that one vertex program can affect the next one. There's no branching. It's a simple computational model. You can do an evaluate and select.

The output data required by the current rasterization model is required to be output by the vertex program. So if you have two texture coordinates turned on- two texture- two texture units enabled, you must pass that. You can pass texture coordinates down for two texture units or you're going to have an undefined operation. The- the implementation is free to generate random texture coordinates or use zeros or whatever. So understand what your state is and understand what you need to actually pass out of that.

So what's the input for vertex programs? Well first you have conventional input. The conventional input is things you're used to seeing in OpenGL. Things like position, the normal, color fog, and a vertex weight. Also you have texture coordinates. For all your texture units you can pass down texture coordinates. And lastly you can pass material properties. Not normally done in OpenGL as a per vertex operation, but in the RVertex spec it does allow you to do per vertex material properties.

But you can think of it in a totally different way if you like. You can think of those as just a set of generic attributes. You can have a lookup table of vertices and you pass an index in and pass a vertex out. If you wanted to do that you could. There's no limitation that you have to conform to a standard OpenGL model. That's one of the beauties of the vertex program. You can think kind of outside the fixed pipeline or outside the box that defines and do whatever you want.

As efficiently or as inefficiently as you'd like. The maximum number of attributes, because there's kind of a different combination of things you can do here, are defined in the spec, max vertex attributes arb, and the spec defines that there is 16 minimums, so you're guaranteed that you have 16 attributes.

and lastly, the read-only. Also, the other half of the input is program parameters. These are kind of things that would be outside of a begin end. So this is like parametric data that you might define for an entire object, like a displacement. Or if you want to enlarge an object, how much are you going to enlarge that object? Or a point to revolve the object around, some kind of manipulation thing that you would do for an entire object, not per vertex. They can either be OpenGL state, as there's a lot of state defined for these, or they can be user data.

User data can either be local to that program or it can work across a number of programs. And that's defined inside the spec and when it comes out, I think you should look at it. They're very well defined in how this works. In this case you have a lot of these. You have 96 minimum, the spec defines, again it's queryable and implementation dependent. Again, inputs read only. You can't read and write the inputs, you can't feed back into the next vertex program.

Here's some examples of what it looks like. I just wanted to put this up not specifically to make any sense, but to give you a feel of what it's going to look like. It's very assembly language looking. At the top you have a vertex position. Very simple. You have a normal. You have some shininess or material properties. Some diffuse in the specular.

Then the bottom part you have some parameters. Things like the model view matrix. You're going to take the inverse transpose of whatever the model view matrix was when you submitted the vertex and you're going to have that in your program. Obviously this is real critical for that transformation. You can't really transform things from your world space to the viewport unless you're going to have the model view matrix.

Then also you can get the model view projection matrix for example. That's without the inverse. You're just taking it directly. There's lighting state. And then there's program environment variables down there just enumerated. So you can pass your parameters just in program environment variables directly in. And lastly you can use parameters for constants.

So, now that you have the inputs, what are you operating on when you're actually running your vertex program? They're program locals or temporaries. There are four components, 32 bits each, very symmetrical across the whole vertex program engine. And you have to declare them explicitly. You see an example of one at the bottom there? That's a normal temp, a dot product maybe, or temporary color storage. They're both read and write, so you can store-- they're a scratch space for you to work with. Defines as 12 minimum, and they're also implementation dependent. If the implementation can support more, they'll advertise how many they can support to you.

Output is a lot simpler. We've seen like sixteen attributes, ninety-six parameters, twelve local variables. Output is pretty defined. Output has the outputs I list there. Positioning clip coordinates, required output. Doesn't make sense to spit a vertex out unless you have a vertex to spit out. You have to put a vertex out there with actual coordinates there.

Color front, back, primary, secondary. So you can do the back secondary color or the front primary color. You can output all four of those options. A fog coordinate. Point size and texture coordinates for however many texture units you have or have enabled if you want to think of it that way. Because if you haven't enabled it, it makes no sense to output a texture coordinate.

Again, this is why I stated before, understand your rasterization model, understand the texture coordinates enabled, understand what you're doing with it later, whether you have fog turned on or not turned on, and make sure you put out all the outputs you need. These are write only. You write to them, they pass out, and there you go.

So an example of these are result position, result color, color back secondary, result fog. Pretty simple. I mean this is really simple stuff to read. It's pretty readable. You can look through the vertex program and since it is not a looping flow, it's not a branching flow, everything is declared explicitly, you can read through it and really tell what it's going to do.

Let's look at the instruction set. The instruction set, as you may have guessed with my inference to assembly language, it is basically an assembly language instruction set. But it is graphically oriented. It's a limited subset of instructions. Not that many. It's not 100 instructions. It's more like 25 instructions. You also have special properties you can do. You can do a source swizzle in the gate. So you can take four component vector.

You can take x, y, z and w. And I want to use x, y in the opposite order in my multiply. So I can do y.x, z, w. And when the multiply happens, it will swap those two and use that in its final determination of the result there.

You also can do a write mass. You can say I only want to write the w coordinate. For example, it's really good in fog. If you are, for alpha values. If you want to write your position with one set of operations and then write your w or alpha value with another set, you can mask off the w. And then you can write your w. you can just write to the w.

Minimum defined in the spec is going to be 128 operations. But implementations again are free to implement more than this. You can query that. Most instructions are single operation. The spec does note that some may be compound. For example, the cross product instruction could be two instructions substituted in there. And that's noted in the spec. And again, when it comes out, I think you should look clearly at that when you're implementing your program if you're getting near that instruction limit.

So let's look at the instruction set a little bit. Just kind of looking through it, you have an address register load, you have some arithmetic instructions. Things you see that are graphically oriented are dot product, dot product 3, dot product 4 and the cross product. You have some min/maxes.

The set less than and set greater than you can use for the branching and selection. And then there's a whole bunch of math instructions at the end. So it's a graphically oriented assembly language. This is it. This is all that's defined in the current ARB spec as existing. So there's not a page and a page of instructions to learn. It's pretty simple.

So now I'm going to take you through an arb-vertex program example, just to kind of look at it, get some ideas of what it looks like, what you're going to see when James comes up. I want to point out one thing here. Again, with the Jaguar seed that you have, we have a fully functional vertex program. The syntax is slightly different from what you see here because it was evolving and we had to lock it down at some point.

The syntax you see here or something, or whatever the arb decides on, which will probably be almost exactly what you see here, will be in the final arb spec and our shader builder tool, which James will show you, will be updated to reflect that. So if we look at this, this declares a vertex program at the beginning, normal in a position, grabs an inverse transform in the model view matrix, grabs the model view projection matrix, sets some program environment variables for some different stuff, from lighting angles, the half angle.

So obviously if you look at this and you're familiar with lighting models, you can see this is basically going to do a lighting model for you. There's a temporary declaration there, and you can see it then declares the output color and output result, explicitly declaring everything you need to operate on. So you can look at that and say, okay, here's my working set of stuff.

So here's what it does. The beginning of it is pretty simple. It's going to do a transform and output the output positions. It does some of the lighting calculations. And if you look at the end here, you can see it finishes out with output color x, y, z, putting out the x, y, and z parts of that output color using the results of its lighting calculation here, and then ends. So that's kind of what you're looking at in vertex programs.

They're fairly low level, but again, it's an efficient way of doing a lot of the operations you want to do. You don't have to worry about 300 or 400 state variables, setting them up, doing a whole bunch of work on the CPU. You can write a vertex program in these instructions to do really whatever you want to do and ensure that you get the most efficient operation out of it. operation out of it.

What I haven't talked about and I'll spend one slide on it is the OpenGL interface to all of this. Again, fairly simple. You're going to generate a program just like a text for generation. You're going to bind to that program to make it active. One thing I don't have in here is you'll enable the program. The next thing you'll do is, I want to read the program string. Programs are just strings that you can read directly into it as in the program string arb function there. That takes the entire program string and parses it.

To set context parameters that you may have seen, the environment parameters, that would be a way of doing that. And then inside the begin end you would do things like vertex attribute 4f and set whatever that vector is for the attribute and then set a vertex. Lastly, once you're done with the end, let me back up a little bit. Once you call the vertex, that vertex 3f, that actually invokes the program. That call is what says execute this program on that vertex.

So you can actually think about it, as soon as you call that, whatever state you have at that point is passing into the vertex program as needed, and that's going to be executed on. Lastly, when you're done with the programs, please delete them and clean up after yourself.

So, this brings us to Shader Builder. Shader Builder is new for Jaguar. It's a great tool. It's a tool that allows you to interactively debug and build vertex programs. And James McComb is going to come on up, and he's going to show you a demo of Shader Builder. Come on up, James. And he's going to take you through that.

Hi. So, yes, today I'm going to tell you a little bit about Shader Builder. Basically what we noticed at Apple whenever I was working on the implementation of the vertex program specification was a distinct lack of tools for writing these shaders. I mean, it's all very well. Fire up Emacs, start writing a program, save it, and then of course you've got to make all the open-- you've got to write like a GLAD application or something and make all the OpenGL calls to load in the shader.

And then enable it. And there's a whole lot of other work you have to do. And this is just to like test your shader once. Then you might have little typos in it, syntax errors. Not to mention, although there's a limited set of opcodes you can use in these things, they're quite-- the logic is a little difficult to figure out.

And we thought, hey, wouldn't it be really great to have a system whereby we could just start typing them and have things appear in real time and people wouldn't even have to create an OpenGL application framework to start doing their stuff. So that was the sort of birth of the idea of Shader Builder and it's been worked on now for some time.

So we believe it's a world class tool. We don't think you'll see anything like it on any other platform. At the moment. And we think it'll be very helpful for you all to use. So now I'm going to start here and show you some of the features of Shader Builder. Here it is. And I'll just identify the main sort of areas in the user interface.

Basically you can see, this is, I'm selecting it, that is the vertex program right there. That's currently being ran. This is the OpenGL output rendering. This area here, which I'm circling, is the controls to allow you to do interactive debugging with Shader Builder. Basically, I'll talk about it later, but it basically allows you to do step by step debugging using an inbuilt software emulation we have.

We can step through and you can watch the state of the hardware registers and it's very useful for being able to analyze the logic in your program. Also, over here on the right, which I can slide in and out, we have a pop-out reference with all the opcodes used in the arb vertex program spec. And as you select them, you can see here the documentation. This is taken straight out of the white paper, the documentation for all the opcodes. top codes.

and of course I can double click any of these and they will appear at the current cursor position in the text. Also here, if I just select that, there's only one entry there at the moment. Arb Vertex Program. It is our hope further down the line as new shader languages appear, like whenever the arb manages to finalize fragment shader spec.

We'd like to be able to include that in Shader Builder so you could just literally go down here, select arb fragment program, if that's what it's called, and the parser that is used to parse your program will change to that and Shader Builder's behavior will change somewhat so that you can write fragment programs.

We just wanted to have a unified way of being able to do this that was clean and would save you a lot of time. So now I'm going to explain in a little more detail what actually is going on right now. You can see that there's like a white circle being drawn here.

What's actually going on is Shader Builder is submitting a series of vertices to the beginning of the OpenGL pipeline. They comprise a sphere and this program is replacing the transformation, clipping, and lighting part of the pipeline. It is being run once for every vertex going through and then the rasterizer is picking up the output vertices the program's emitting and it is rendering them up here. This is a very simple program.

In fact, this is the simplest vertex program really you can have that would be of any use to you at all. Basically, these three lines, they declare three variables inside the program. They basically bind to attributes of the incoming vertex, both the position, color, and the incoming texture coordinate. This line here binds to the vertex. This line here binds to the model view projection matrix inside of OpenGL.

It's a fairly straightforward syntax. Remember, all the commodities are four component vectors. That's why I've got a bracket four just like the C array syntax, basically declaring a four by four matrix. These three lines right here, they declare three more variables which are bound to the outgoing vertex structure. And that's what I'm trying to fill out. When I fill that out, that is what the rasterizer will render.

These four lines perform four dot products, basically transforming the incoming vertex position by the four rows inside the model view projection matrix. And they fill it out into the output vertex. And this right there is a simple move operation. I'm just moving the incoming vertex color to the output and next line exactly the same with the texture coordinate. So it's simply, it's called pass through. It's just a simple pass through.

If I select this button right here, GL parameters, this allows me to change some of the attributes of the incoming vertices. If I click on this color well right here, basically of course there's a GL color 3F call being made to set the color of all the vertices that are being submitted.

By sliding this around, of course I can change that. And since it's passing it through, you can see the white circle changing. Of course also I have control over, if I select this, it allows me individual control of the texture units on the graphics card. And I can load textures and enable and disable them individually.

So the next little thing I'd like to show you is, I'm going to start making some changes to this program to make things a little more interesting. And also to show you one of the really nicest things. One of the really nicest features of this tool. What I'm going to do is, I'm going to add another binding to an incoming vertex attribute. I'm going to bind to the normal as it comes in. So you're going to notice something as I start typing here.

It's actually, every time I press a character, there's a thread going in the background which is just constantly, as I type, parsing the text and compiling it and returning out the error information from the parser as I type. So you can see down at the bottom, I'll just put the mouse on it, it's giving me exact information of what it thinks is wrong. And it even gives you a hint as to what the parser believes is the erroneous token. So I haven't finished typing, so ATTR is highlighted in red. So I'll just continue to type here.

I've created a binding to the incoming vertex normal. Now I want to be able to see the normals of all the vertices. I'm going to assign the incoming vertex normal to the output color, which will produce an interesting effect. I'm going to go into the instruction reference here on the right, pick out the appropriate opcode, double click it. You can see it just dropped it in there for me. There are placeholders for what variable names to use as the operands. It's of course returning a syntax error because those don't match what's in my program. I'm just going to fill them out right now.

Remove this line here. And as you can see, it's updated just as I've been typing. Now you can see the normals for all the vertices are being mapped to the color. So you can see this rainbow effect. And you can see that Shader Builder actually draws the axes here for you so you can see what's going on.

That's just a basic example, just to let you see that this is occurring in real time. I'm going to show you more interesting shaders than this later, but this is just sort of an introduction to it. I'm not going to show you the debugger inside this tool to allow you to analyze your logic. So I'm going to close down this shader right here.

And I'm going to bring up the next one right here. This is not a very graphically interesting shader. Indeed, so uninteresting that it draws absolutely nothing in the output. Because it binds to none of the outgoing vertex attributes and it writes no data in, although all the vertices for the sphere are going into OpenGL, well, none of them are -- they're getting stuck at the TCL part of the pipeline. They're not being filled out. But I have a different purpose for showing you this shader. It does declare three local variables, sort of three temporaries. I've named them A, B, and C.

I can create new ones if I just type in here. I just created a variable called hello. You'll notice as I've been typing, if you look at the identifier list, over here on the right, as I create new variables in real time, it's actually recording them here on the right. The reason for this is for debugging.

So over here, you'll see that the variable B has got -- it's all four component vectors, as I've told you. And it's got one, two, three, four in it. And C has just got two in it on the X component. I can just double-click here and I can redefine. I can change these.

I can change these and put in new values. Just like that. Now I'm going to start the debugger. The controls are over here on the bottom. If I click on the step button, you can see down here at the bottom, it's saying stepping through the program, press reset to return to editing. The source code is read-only, of course, while I'm debugging. And it grays it out. And you can see that it has highlighted the opcode that it's about to execute in blue.

If I click the step button, pretty straightforward stuff. It's basically, as it said, move B to A. And if you look in the identifier list, the emulator returns information as to what variables were touched upon every cycle of the program counter. And I can see that A has just been touched and it's highlighted in blue there.

So you can very quickly get an idea of what each line of code is touching and changing. So I'm just continuing to step through the program here. You can see it all going through. I can reset the emulator. I can run it just all the way through and see exactly what's going on.

Even if, say, I'm debugging, I'm halfway through the program, I can change the state of any register at any time. You can see here it's about to execute a move operation. I can change the value of A to something else just as I'm debugging. I don't need to reset and change them when I'm starting.

I can do it in the middle of the program. No problem. So hopefully, if you use your imaginations and you do try writing some of these shaders, which I hope you will, I think you'll find yourself using this quite regularly. So the next feature I'd like to show you is the ability... Okay, you've written your shader. You basically have a text file written now with your shader in it, and you want to be able to... You want to be able to take that and have a framework to start writing a game with or doing something useful.

I mean you've got a text file, that's not a lot of use. You need some C files with all the OpenGL entry points being called to load your program in and set it up appropriately. Well, Shader Builder can help you out substantially there to save you a lot of reading basically. I'm going to open up another example right here. This one right here. It's basically a simple lighting model.

I'll show you what's up. Let's just resize this so you can see a little more clearly what's going on. Rotate this around a little. You can see it's a little car. The lighting model, it's absolutely really simple. It's basically just doing a simple dot product with all the vertex normals with the light direction vector. Then of course it's modulating the incoming vertex color with that. Meaning that if I go here and I start changing the vertex color, you can see it updates in real time.

It doesn't seem to be working. Oh well. I don't know. Anyway, it would export a project. It would start a build process. It would basically watch the build log and it would then run the program. It would hide Shader Builder and run the program for you. And you'd be able to open up Project Builder and start working on it. I'll look into that. Anyway, moving on.

The next thing, just to show you some slightly more interesting shaders, just to show you what you can do with these vertex programs. That was a very basic lighting model I showed you. Very similar to the one implemented in OpenGL. It is really, the standard OpenGL lighting model is pretty crude. Here's one that I actually wrote a couple of weeks ago, which I'll open up. It's, where is it here? This one right here.

Okay. Oh, there's a W component. Pardon me. And...

Thanks. So, just any other things I'd like to show here. I'll show one more shader and then I'll be heading over again. This one is a, hang on a second. This is an implementation of a cell shader.

It makes use of texture maps quite extensively. If I go in here, you can see that texture unit one is being loaded up with a one-dimensional texture, making up ramps of color. Actually, about four ramps of color. What's actually being done is the vertex program is setting the texture coordinates to look up within that texture to be able to create a ramped effect. There's another texture right here, which is actually two pixels wide and one pixel tall. It's just a black pixel and a white pixel. It's used for doing edge detection.

It looks like someone's went around the model with a black marker. I'll use the car, but I don't know if you can really see it. On the sphere, it actually looks like there's a black surround on it, and it looks like it's got a quantized look on it. If I rotate the model of the car, you can see the cell-sheeted effect it has.

That's just being done in one vertex. There's no performance hit for doing this, because it's all running on the hardware, if you have the hardware. The important thing to note here is that we do have a highly optimized emulator running inside our standard OpenGL implementation. Even on your old measly PowerBook with its Rage 1 to 8, you can still go ahead and you can write these parameters. The performance will be a little slower, but it's actually very usable. I write shaders all the time on my PowerBook, and it's really no problem. ShaderBuilder is on the Jaguar Seed CD that was given out to you guys. Take it home.

The example programs that I've been working with are on the sample code website, which I believe there will be a reference given to that at the end of the talk when I head back to Jeff. Go download the shaders, play around with it. I guarantee you will not need any prior knowledge of this stuff to actually figure it out. It's incredibly easy to learn. Go ahead, play around, and get experimenting and start using these things in your games. I'm going to hand back over to Jeff here to carry on the show. Thanks, guys.

I think the folks here who have used vertex shaders before and know the kind of write a text file, write a C program, load it in, what do I get? Nothing. Go back. Edit the text file, load it in, what do I get? Nothing. And go through this. When you're trying to learn a new technology, this kind of tool is great.

I mean, a small brain like me, I can sit down and fool around with it and actually kind of come up with something. You guys, I'm sure, can experiment and really build great shaders that you can use in your applications for real world things. You can really manipulate the graphics pipeline in an interactive way for the first time. And this tool is not available anywhere else. And as James mentioned, one thing we're looking at is the instructions and the supported languages. Right now, our vertex program. We're looking to expand that to the other ARB specs.

And even non-ARB specs. We're looking to whatever we can put in that we can put as a text language, as a parser, and give you an interactive environment to debug and work with these things to really multiply your productivity. So you're not spending days trying to debug small things so you can step through and find that small error in your math code.

I'm sure we all wish we could go back and spend more time in our math classes in college so we can, you know, so you guys who are in college, listen to the math classes. But this stuff gets hard and when you have a debugger like this, it really helps out. Let me go back to the software. slides.

and we'll move on and talk a little bit about texture and fragment shaders. What I want to do here is not really cover in the in-depth that I covered in vertex shader and texture and fragment shaders, but give you an idea of what they are, what part of the pipeline they control, and how you can, what effects they can do and how you can use them in your applications.

First, that first bullet there really explains what they are. It's a per fragment, texture address lookup and multi-texture combined. So let's break that down. What that means is, you take that texture coordinate in, and the simplest case of something is to take the texture coordinate, use that to directly index into a texture, take the value of the text at that point, and output it directly as the fragment color. Well, that's okay, I mean that's standard texturing, but there's a lot of other things you can do.

Bump mapping, you can do the shininess, the specular reflections, all these are things you can do in a texture and fragment program. What you do is you manipulate the way you're looking up into textures and the way you're combining those together. So first you need to determine a texture address.

You need to go to the texture and look up that value, and then you need to combine those multi-texture lookups into a fragment color. Or use multiple texture lookups that depend on one another. Look up a value out of one texture that you're going to use to look up into another texture that allows you to actually get some really cool effects.

Simple thing here is what do you output? You output a fragment color. That's basically it. All this program, whatever you want to do there, is outputting a fragment color. So this is basically what I was talking about. You have texture coordinates and vertex colors coming in. You do a texture lookup and you get a color fragment out. That box is what you're going to actually be programming when you look at texture and fragment shaders.

So, why isn't this a single extension? Well, first thing is lower down the pipeline and the hardware that the two main vendors, ATI and NVIDIA have are significantly different in the kind of their functionality and how they work. And they're progressing very rapidly, even more rapidly than the vertex programming area.

So we really haven't come to the maturity that allows us to bring a single extension to encompass the functionality in both sets of hardware. So what we have is, NVIDIA's side we have texture shader and register combiners. On the texture shader side, I need a drink of water, I need a shader.

Textures Shader actually does the address lookups and the register combiners, as you would think, combines the various textures into the final fragment color. Fragment Shader encompasses all of this. It's a little bit newer of an extension and will actually take this entire operation and put it into one kind of unified command set.

Let's look at all three of these. First, Texture Shaders are new for Jaguar. We hadn't had them before. Texture Shader 1, 2 and 3 are supported in Jaguar. And they provide, again, texture address lookup in a various set of shader stages. There are three extensions and they provide 37 canned operations. You can combine these in a lot of different ways and get a very powerful shader program.

Some of the things you can do, for example, they have conventional texture lookups. So the simplest shader may be a 2D texture lookup that's exactly getting the textal at your texture coordinate and returning that as the fragment color or using that into the combiner stage to get the fragment color.

But then you have special cases. You could do nothing or you could just pass a value through. Or you could actually color fragments and say, "Hey, we don't need to draw that fragment for some conditional." Then you can do dependent texture reads, which I talked about a little bit. And it's really critical for things like the recolorization you saw in the keynote, the background color removal.

That's using a dependent texture read to actually do a color correction there and remove that color and change an alpha value. Then the last thing you do in a lot of the reflection and bump mapping algorithms is to do a color correction. And you can do that with the dot product dependent texture reads, which also put the dot product into that kind of formula of how do you configure the shaders.

Moving on to register combiners. Register combiners is really an extremely configurable fragment coloring. It's not, again, not a programming language, but it gives you a kind of a box that you can configure in different ways to replace the texture environment control like textureenvadd, textureenvcombine, the fogging and the color sum operations.

The key here is you go from a linear one, if anyone's looked at doing some of the textureenv stuff, you're pretty limited in a couple texture units. The output of one texture unit needs to go into the other texture unit. Only the first texture unit has access to the color.

All those things are very limiting, especially if you're trying to do special effects with this. So what this allows you to do is having all texture units have access to the color, non-linearly combining these texture units to get a final output. So it really helps in doing things when you have to do multiple levels of combining and multiple levels of coloring onto your texture units.

The second extension, there's two extensions. The second one adds additional color. Flexibility, for example, if you're trying to solve the problem of doing planar textures, red, green, and blue, in separate areas and you want to combine those together, having that separate color is really good because you can colorize each texture unit and then combine them together in a final step. Both of these extensions, or all five of these, register combiners two, texture shaders three, are NVIDIA specific.

ATI has brought along fragment shader. Fragment shader basically brings a unified instruction set. It's still in development right now. We're targeting it for Jaguar. It controls both the texture address lookup for texture shader and the color data, which is the register combiner set, into one overarching extension here.

is similar to our vertex program in its layout. So I'll show you an example of it. Really, it's very preliminary, but it gives you an idea that you'll be able to use a tool like Shader Builder to edit a language very similar to what you've been working with with a vertex program when this is complete. This is on ATI Radeon 8500 and later processors.

So I'm not going to really go through the functionality here, but I just wanted to give you kind of a sneak peek at what it's looking like, what kind of things we can see here. This is a bumped cubic environment map shader. So this is the operations you need to do to do a cubic environment map that's also bump mapped on an object. First we're going to do some routing instructions to basically tell which texture coordinate variables are mapped to which texture coordinates. Pretty simple here.

Then we're going to do some color dot products. This is similar to what you saw with a vertex program where you're going to do a dot product and you're going to use the texture coordinates to, which source texture coordinates and any kind of operations or swizzles or, you can see it says mod 2x down there, any modifications to those texture units as they come in so you can do those in a programmatic way. Then you're going to do some of the i-spaced lookups. This is something you would, it looks like a lighting calculation, a bump map calculation that's very similar.

Moving on, you have a dependent texture read where you're reading out of a previous texture coordinate, using a texture map, a texture coordinate, reading into a texture map and pulling it back out and using it later. And then at the end, you're going to see that the final step here is it moves the texture coordinate zero into the result there.

And that's the output. Again, these kind of pixel programs, fragment shaders, what you will, have the color as the output. So in the end, what you're outputting is that fragment color, pretty simple. So let me bring up one more demo. Let me bring up demo machine number two.

This is NVIDIA's launch demo for the GeForce 4 Titanium. It's an interesting demo in the fact that it does a lot of things. You can see there's bump mapping in the facial area. You have self-shadowing. You have the fur. You can see the bump mapping here.

Extremely impressive demo from NVIDIA showing the power of these shaders. How many shells are there, Geoff? There's eight. Why don't we leave the Wolfman up over here and go back to slides on the right side, please.

The Wolfman demo, I was going to go through a little bit about how it's built and what it does in a minute or two. But instead what we decided to do is give you, in our advanced 3D coming up, we're going to take you through the actual how that was built, how the shaders were used, what the technologies are.

It might be a really good session to go to because you really can see some of the advanced things you can do with shadow volumes, with the fur rendering, with the bones and how to do the bone structure on that. So I really recommend you go to that and you'll see exactly how the Wolfman was put together.

We've talked about a lot of stuff here today and I don't really think that you can take away from this session and actually get every nuance of vertex programming or obviously of pixel or texture shaders. So what I recommend you do is when the arb is complete with the spec, which should be very shortly, you get the spec.

It's a large body of work but if you read through it, it's fairly straightforward and you can use the information you have here today and you can look through it and get the nuances of it. Take shader builder, start working with shader builder, build some shaders, understand what the vertex shaders can and can't do and put those in your programs. And finally the ATI and NVIDIA websites have a wealth of information on their extensions and on vertex and pixel programmability in general.

Some other online sources are theopengl.org, always a good place for reference. Liz.apple.com, if you're not on the Mac OpenGL list, I recommend you join that list. We're on that list, we pay attention. While we may not be able to respond to every single question, we'll try and give you good information that you may need. And lastly, developer.apple.com/opengl.

So, we've talked about programmability in general, talked about vertex programs. James gave you a great demo of the shader builder which is on your seed CD. We're looking to continue moving that forward. Any suggestions you have, send them on in. We'll try and put those into our products. And we finished up with texture and fragment shaders.

Let's talk a little bit about the roadmap. We're at the third line down here, so we're at the graphics programmability. Tomorrow morning, Integrated Graphics 1 talks about integrating the OS and OpenGL together. Followed by Integrated Graphics 2, which is like the supercharged version of that. What are some great optimizations, some great things you can do combining the power of Mac OS X, the Jaguar and OpenGL. Colors and Conditional Media. There's some game solutions which some will incorporate OpenGL. Some will have some great solutions to event handling and some other problems that everyone runs into whether you're a game or not a game. Again, game solutions follows with some network solutions.

Followed by that, Advanced 3D on Thursday at 3:30 and that's going to talk about how we built Wolfman and some other surprise demos that we're going to show that show some really advanced 3D techniques. Followed by OpenGL Performance and Optimization. If you need to go, if you're doing OpenGL, go to that session.

It's a great, talks about some more tools we have, talks about some great performance optimization things and it really is a great session to go to, great things coming out of it. Graphics tuning for graphics in general and then followed at the end by the feedback forum. Contact Sergio's information, [email protected]. And let's move into Q&A.