Thank you everyone for coming to session 417. My name is Babak Mahbod and I'm a Senior Software Engineer here at Apple Computer. So today we're going to talk about what it means to actually switch to Mac OS X OpenGL. So let's talk about what we're going to you know, the con10t of this session.

So at, today we live in an age of digital con10t and multimedia, and at the heart of every good multimedia and digital con10t system must lie a great 2D 3D graphics engi=ne. And here Mac OS X, we have, we feel we have the best commercial implementation of that 2D 3D graphics API in OpenGL.

And also with OpenGL it has historical origin in just being used as a 2D 3D graphics APIs. Well you has, that scene has changed a bit, and today it's, really I feel has evolved to become more than a 2D 3D graphics APIs, it's just become a hardware abstraction layer that actually abstracts the graphics cards and your GPUs from you application developers.

So today we going to talk about application development, and what it means to bring your 2D 3D graphics application to our platform. And to facilitate that process, we going to talk little bit about our utility frameworks that will accelerate that process. And then we going to talk about some modern practices. And why is it that we want to talk about modern practices? Well today it's hard to actually separate performance, optimization you know, from bringing your OpenGL application to a, our platform, and it's the same situation actually with our Direct 3D = and Direct3D 9 and 10 developers.

Because when you bring your applications, you know, when you go beyond a fast path, you need to take into a, you need to get off things like immediate mode and fixed-function pipeline. And there is a common theme amongst all a family of APIs. Of course it's a huge difference going from 1 frame per second to 2 frames a second, to 10 frames a second, to 60 frames a second. So we need to talk about fast path best practices on optimization techniques.

And we going to also talk about, a little bit about HLSL to GLSL, and more specifically we going to talk about the concepts like vertex buffer objects, flush buffer range, which is closely tied to vertex buffer objects, render to texture, pixel buffer objects, and we going to talk about shaders.

And as we progressing through our presentation, more specifically with the concept of textures vertex buffer objects, render to texture, and shaders, I'm going to be showing you some algorithmic workflows in Direct3D 9, some algorithmic workflows in Direct3D 10, and OpenGL. So the basic idea is that you going to see that, even though there are really no one to one and onto mapping between the APIs coming from Direct3D 9, Direct3D 10 to OpenGL. The ideas are there.

So let's talk about application development. So some of you are coming from UNIX variety of OpenGL, and you probably been developing your applications using Xt, Motif, CDE, KDE, or gnome toolkit. So what is that you normally do here? Well, you take into account your user inputs, then you want to handle some events, and for that you write some callbacks. Then you get a window, you set a device context, some parameters on a device context, and then you start working with your OpenGL code.

For those of you who do cross-platform development, you probably been using GLUT, you game developers who have probably been using SDL, Q/t from Trolltech, or wxWidgets. Again, the theme is the same, because you take your user inputs into account, you write your callbacks, you get a window, you set some parameters on your graphics device, and then you're off and running. And for those of you coming from Direct3D 9 and Direct3D 10, you of course been working a lot with Windows APIs and graphic device interfaces.

And idea again is the same. Handle events, get a window, set some parameters on your graphics device, and last but not least, in this case when you're coming from Windows, you get a scene and you start drawing into a scene using Direct3D 9 or Direct3D family of APIs.

On Mac OS X we recommend that you use Cocoa for your application development. And why do we recommend using Cocoa? Well Cocoa has its origins from the NeXT days, but it has actually grown into a very mature and very powerful family of APIs and frameworks. That allows you to rapidly develop commercial quality application in a short period of time.

There's truly a seamless integration between your tools and your Cocoa APIs and frameworks, and as you will shortly see, it really comes down to dragging your buttons, your windows, and, from some palettes, you know, one of our tools called Interface Builder, and you initialize some attributes, you connect them to other objects, and you're off and running.

But Cocoa is just really more than that, because it generally takes into account most of the events that a user, well you know, user might be using. So for example, for application, windows and workspace management, Cocoa frameworks handles those events for you. And in addition when you start using Cocoa and you develop a Cocoa application, you get a lot of additional things such as scriptability, comes automatically with it. So Cocoa also has, also defines a model for behavior of an application. And when you generally develop an application, you try to fit the application that you're developing into this model.

So Cocoa applications adhere to this model view controller design pattern. So your model basically will be your intellectual property. It will be your game engine, game physics engine, or your computational engine. Your view for example will be your NSView where you're actually going to be drawing your OpenGL content into. And your controllers and object, as just like a request broker that marshals the messages between your model and a view.

So at this stage I would like to bring to stage Mr. Michael Jurewitz, he's a colleague of mine at Apple. And he's going to run through a demo that's going to show you basically functionality of the Cocoa application. Michael?

Thanks Babak. So as Babak mentioned, I'm a DTS Engineer, so my job is to work primarily with developers like yourselves who are either bringing applications to the platforms, or who have questions on how to use our frameworks. So we've got a couple demos planned for you today. We'll go ahead and start off with our first one, OpenGL basics. So let's go ahead and build and run this application just to get a flavor for what it does.

So build and go. And once our application is launched, you'll see that by clicking these buttons on the bottom we can go ahead and affect the way that we're actually drawing into our OpenGL view right here. You see we have a simple Cocoa application, complete with menus, complete with event handling, that's really actually a barebones application, so let's dive in and see exactly what's happening right here.

So we'll go into Xcode, and we'll take a look first actually at our header file for the OpenGL basics view. And you'll see this is simply a subclass of NSOpenGLView. Now NSOpenGLView is the basic app kit class for doing work with OpenGL in our frameworks. And this OpenGL view is itself a subclass of NSView, which is responsible for all the drawing behavior throughout the kit.

You'll see we've declared some simple state that we want to keep track of for our view, and we've also declared a method down here we call setClearColor. setClearColor you'll see has a return type of IB action, which might seem kind of strange if you're near the platform. So IB action is simply a #define that Interface Builder uses, so for you to be able to annotate on your objects that there are messages that you want other objects to be able to send to it. We'll see this in a second when we actually go into Interface Builder.

You'll also notice that the one parameter that we take in here is a sender. Again, that's customary with IB actions, we want to know who's actually going to be sending this view a message. So let's go ahead and actually take a quick look at Interface Builder, and see how we have this laid out.

So we'll open up our MainMenu.nib, and we'll take a look at our window. And you'll see this looks exactly the way that we had it on screen, only here we have our OpenGL basics view that at run time we render to. We also have some buttons set up here, and if we control click we'll notice we have this inspector.

And you can see that this button is set to send an action called setClearColor to our OpenGL basics view. Similarly, the same thing for our green and our red buttons, those are also set up to do exactly the same thing. So let's go ahead and hop back into the implementation here, and see what's actually going on.

So if we look at our OpenGL basics view .m file, which is the implementation, we have three methods that we want to focus on primarily right now. So we have this method called awakeFromNib. Now what I just showed you there in Interface Builder is what's called a nib file, where we archive off our objects that at run time are going to be reinstantiated for us, and as part of that reinstantiation process, after all the objects have been initialized, and all the connections have been set up, they're all sent the message awakeFromNib. Now this also happens before any events have been handled by those objects.

So in this case we're actually just setting a colorAlphaIndex for ourself, which if we look up, was back in our header, it's just part of an enum called ClearColors. In this case we're just setting up some basic state for our initial drawing. That's really the purpose of awakeFromNib is for you to set up that kind of initial state.

Now the next method that we want to take a look at as drawRect. drawRect is a method that comes from NSView, and this is a method that anything that wants to draw to screen, and is an NSView subclass implements in order to achieve that behavior. So you'll notice that we're simply taking some basic GL state, and using that to then call a method called drawTexture, and then we're simply flushing our OpenGL context. Now the other method that we mentioned earlier was setClearColor, and you'll see we have the implementation for it right here. Again, we take in a sender. Now you notice that we ask this sender for a tag.

Well, all of our senders are buttons, which are controls in the kit. Controls let you associate with them what's called a tag, which is just an integer to identify them. In this case we've used it simply to be an index that we're going to use for our drawing.

Next, since we've actually clicked one of these buttons, we need to give a clue to the kit to know that we need to update the drawing of our view, and that we should go through drawRect again. In this case we say self setNeedsDisplay, YES. That simply tells us that our view is dirty, we need to redraw it to the screen.

So let's look at how we can actually put together an interface like this, and how quick it actually is. So if we take a look at Interface Builder, we'll go ahead and just close out our nib that we had here, and we'll make a new one. We'll just use the application template. And again, you'll see we've gotten most of the same objects that we had before, as well as a window.

So now let's take this window, and we want to add to it an OpenGL view. So we'll use our inspector, and we'll type in NS OpenGL, and drag this window simply to, drag that view simply to our window. We can go ahead and resize it really quickly. Now again, this is just an NSOpenGLView, but we had a custom subclass that we had been using. So we simply go back to Xcode, and make sure that we drag in the header file for that into Interface Builder. It'll parse that and be aware of it so that we can use it in our nib.

We'll simply make sure that we go and we set its custom class to OpenGLBasicsView. I just typed in O, it already auto completed for me. And you'll also see that it's already parsed the file and noticed that we have an action that we can take called ClearColor, called, sorry, called setClearColor.

Now let's go ahead and just add a button to our interface. So we'll search for button, drag that onto our interface, and then we can just control drag to our view, and you'll see that one of the options here is to send the message setClearColor. We simply connect that up, and we're good to go.

We can go ahead and set the tag on the button if we want, in this case we'll just leave it as 0. And that's all you have to do in order to get a simple interface like this set up very rapidly. All right, Babak, back to you.

Thank you Michael. We're going to see Michael a little bit later on today. Slides please.

Okay. So let's talk about some of these OpenGL utility frameworks. Of course for those of you who are coming from Windows, and you're used to developing OpenGL on Windows, most of you are familiar with WGL, or wiggle. What does wiggle do? I mean we talked about earlier on that once you set up the window, you want to set some attributes for that device context. Great, so how do you do that? You do that basically through wiggle. Example of which we'll be setting a pixel format.

If you're coming from UNIX, the same story is true over there. Because GLX again, you have a window already, you have a device context, you want to set parameters on it against let's say pixel format. So you'll be using GLX. On Mac OS X actually we ship with X11, and we also ship GLX with X11, and it's actually optimized for our platform.

On Mac OS X, Michael briefly touched upon it, we have NSOpenGLView. And NSOpenGLView as you saw, can be setup in an Interface Builder, and it's a subclass of NSView, NSView being its parent class, and it just basically gives you all the facilities to display your OpenGL content. And its building blocks, in addition to NSView are OpenGL, NSOpenGLContext, and NSOpenGLPixelFormat.

So basically when you subclass NSOpenGLView, you really don't need to do much. You don't need to subclass NSOpenGLPixelFormat, or NSOpenGLContext. But if after trial and error you found out that perhaps NSOpenGLView doesn't satisfy your requirement, you can directly subclass NSView, and then you can use NSOpenGLPixelBuffer class, and NSOpenGL classes for doing full screen rendering.

But if you're not by chance doing Cocoa development, and you're using Carbon, and you're doing procedural programming, we have actually three utility frameworks that will, two utility frameworks that will aid you in that workflow. One of them is of course is AGL is strictly for Carbon, then there is CGL which is for both Carbon and Cocoa, and in fact this is the heart of NSOpenGL class and AGL. Both of them offer you full drawing to a window. And the advantage of course CGL is Windows system agnostic. We also ship with a version of GLUT, which is actually a native implementation on our own platform.

So basically we take actually quite a layered up approach to design in all levels of our OS, this is just basically our implementation of OpenGL. We have the driver level, which actually has a software rasterizer, the ATI rasterizer, and ATI driver, plug in, in video and in video rasterizer, Intel rasterizer, and plug in.

And a OpenGL engine actually comes in two varieties, the multi-threaded and the non multi-threaded version. And you've we talked about the utility classes, and of course once you set your pixel format, let's say using AGL or NSOpenGL or CGL, you'll be again using OpenGL to do your, all your work.

So as 2D 3D graphics application developers, what are some of the common things that we use? Well we work a lot with textures. But before actually delving a little bit into how to use textures with OpenGL, let's just look at couple of basic workflows in Direct3D 9 and Direct3D 10. So if you want to create an empty texture in Direct3D 9, you just first declare a texture description structure. Then you allocate a structure, and now you have an empty texture.

With Direct3D 10, you know, the APIs are a little bit different, so there's a little bit of different workflow. So first thing you do, you declare a 2D texture description structure, different data structure, then you initialize the structure, and as many of you have done Direct3D 10 programming, you're familiar with this workflow.

And then you set some texture parameters, use a create textures 2D which you got off a device context. And at this case you know, you will have an empty textures to work with. So with OpenGL, you first call, you generate texture, and you get a texture ID. Then you bind the texture, and you tell them well I want to texture rectangle ARB, and you pass it the texture ID, then you build the texture using the information in a memory bucket, a void pointed in this case.

And you just set some parameters, like its width and height, and then you set the texture parameters. So one of the interesting thing here is you might ask where do I get this memory bucket, this data? We'll talk about how to do this shortly. But before doing so, when I was actually generating 2D texture, I set some texture parameters. So these are the recommended texture formats on our platform.

Of course for those of you who are coming from other varieties of OpenGL, namely from Windows or UNIX, all your formats are supported. But again, the optimal formats are up there. And if you're coming from other platforms as I said, other formats are supported, but they may be swizzled by many of the cards that we ship.

So, and then you saw that when I was binding a texture, I was saying well I want a ARB texture rectangle. Well, this is actually a fast path on our platform, and by using ARB texture rectangles, there are some tradeoffs. And the tradeoffs being that you can't use mipmap filtering with ARB texture rectangles, when you're using ARB texture rectangles your textures can't have borders, your textures be using not normal X texture coordinates with ARB texture rectangles. And of course the only wrap modes they can be using with ARB texture rectangles are CLAMP, CLAMP_TO_EDGE, and CLAMP_TO_BORDER.

But for those of you that are in a world of video processing, and for those of you who have gone to some of our session on digital video processing using Core Video for example, there is good chance you probably will be working with non-power-of-two textures. And this is actually prevailant theme in video processing, but a note of caution here is with non-power- of-two texture is that certain hardware, some texture wrap modes, they don't work. And certain hardware mipmapping may not work.

And normally you, if you want to be using ARB texture rectangle, you should check for this extension. And if one of the things that you're looking for, it's not implemented in the hardware, then you know, you can just scale the image for example, using gluScaleImage API, or you can segment the image into a non-power-of-two rectangles.

So let's talk about how you get that memory bucket, that data. So here we have a framework called image I/O. So image I/O actually is a framework that allows you to read and write platuro (assumed spelling) file formats, and allows you to also read and write metadata. It does automatic color management, and if you want to do work with RAW camera formats, you want to be using image I/O. And it actually has incremental loading, and it has a floating point support.

So it supports all the web standards, the HDR formats like OpenEXR, all the camera RAW formats for all your variety of digital cameras. If you have file formats from Windows or UNIX, supports many file formats from those platforms, supports a metadata file formats, and as more and more file data are becoming available, the image I/O team will bringing that also to you under the hood of image I/O.

So where does actually image I/O lie in a greater scheme of things? Well our team of engineers actually have taken libTiff, libJpeg, and libPng that many of you from UNIX are familiar with, and they have actually optimized them, and they gave you this image I/O framework to actually take advantage of those libraries. And you can see some for other technologies, like Quartz, Image Kit, and Core Image leverage off of image I/O.

So let's look at how you can actually read a file. So the first thing that you do is you start with the file path, okay? So you want to covert that to a URL. The next thing you want to do is you want to create an image from the URL, now that you have a URL.

So the first thing is you actually create a image source, which is a image source reference, which is an opaque data structure, okay? Now your image, that file might have more than one image, so you, what you want to do is you want to say well let's say I want to get the image at index zero, or index one. So now you have another data structure called ImageRef. And from ImageRef, what you want to do is just you want to get some of its attribute, for example, its width and height, and of course you want to get device's color space.

So now next step is you create a bitmap context. That's where you get the data from. But we're not quite done yet. Because we want to call ContextDrawImage API to actually fill that memory bucket. Now you're almost done, because you note that a lot of the APIs here have create in them. That means you'll be holding the memory, you'll be controlling that memory. So what you need to do in order to avoid memory leaks, you want to actually release all those extra references, okay? So now we know how to read data into a texture.

So what's the other things that's common to all the 2D 3D graphics programmers actually working with vertex arrays. So let's talk about vertex buffer objects. So in general, vertex arrays actually exist on clients' side of things, and your graphics card wants to access that vertex array, it actually has to go across a bus and fetch those vertex arrays for you. Of course, because of the limited bandwidth of a bus, there'll be a slowdown.

So that's where vertex buffer objects come in handy. So in case of vertex buffer object, OpenGL actually caches the vertex arrays on GPU, and allows, you'll be kind of, you don't, no longer actually need to worry about that limited bandwidth going across a bus. And when the OpenGL caches a vertex array into VRAM, your buffer objects will be resident on a VRAM, and your vertex processor can actually take advantage of that fat pipe that exists on your graphics card.

And of course once you start using vertex buffer object, what you need to be doing is go through your code and look at where you're using display lists and vertex arrays, where you're you know, applying your techniques that you learned long ago, you know, using immediate mode graphics, and adopt a vertex buffer object.

So let's see how we do this in Direct3D 9. So first we define a buffer and vertex data, then you create a vertex buffer object, and then you need to actually at this stage set out the layout so the pipeline knows what kind of data is coming to the pipeline.

But we're not done yet with Direct3D 9. So the next thing that you need to do is you configure the vertex and buffer layout, then you're ready to draw. With Direct3D 10, as you know, it normally involves initializing multiple stages of your Direct3D pipeline, and= you always do that with Direct3D 10 by initializing some data structure. So let's first start here by defining a buffer and vertex data.

Then you initialize a buffer description object, then you initialize a subresource structure, you create a vertex buffer object, then you basically have to configure input assembler stage, and then you configure a data topology in the input assembler stage, and then you're ready to draw. I'm sure as many of you have, who have been working with Direct3D 10, it most of the time involves a lot more coding than Direct3D 9. In OpenGL you first define a buffer.

You then, you create a buffer, then you modify the memory in the buffer by first mapping from buffer object into client memory. And then once you're done modifying that vertex buffer object data, you let go of it. And then you're ready to draw. One thing I want you to remember is that the buffer offset macro that we're going to be using throughout this session.

So that's vertex buffer objects. So here the theme was again, optimization technique, a fast path. But a concept that's actually closely tied to vertex buffer object is actually flush buffer range. So let's talk about what flush buffer range is. So basically OpenGL commands in first in first out fashion. And buffer objects usually block the data, until the data is processed through the OpenGL pipeline. And in order to avoid this blockage, GL usually creates extra copies. So for partial flushing of vertex buffer objects, you want to be using APPLE_flush_buffer_range extension.

So of course inherent advantages are that the theme is asynchronicity, and also if you want to minimize the data having to get copied back and forth, you want to be using this extension. And of course as application developers, you know your application better than anybody else. So you can manage this synchronization point using fences, and then you can modify the ranges of the data that you know, your buffer object contains. And while all this magic is going on, OpenGL might be just using part of a buffer object.

So here the first step is of course you switch off flushing on GL on map buffer, you modify the memory in the buffer, you modify the vertex buffer object, and here you explicitly flush the modified bytes. So this is flush buffer range, and again, the theme is optimization, fast path.

So the next thing, we create an empty texture. But really, creating an empty texture and setting some parameters on them is just kind of not terribly interesting. You won't always be able to do something with textures. So let's talk about render to texture. So one of the applications of render to texture, and it's really a tightly coupling with this concept of multipass rendering. So for those of you who come from CG world, you've probably seen multi pass rendering.

And for those of you who don't know what multipass rendering is, it basically involves, you have separate images, and you want to render these separate images for different attributes that comprise a scene. And then these images, you want to take them, and then load them into some image processing app, and then filter them together to create a final image.

So in reality what you're doing, you're really breaking up this complicated workflow into actually, to really a discrete number of steps, and into multiple rendering passes. And then what you can do is you can combine these passes into the framebuffer using blending, and then you can use these multiple passes to sort textures that you'll be using in all your future rendering passes.

And of course on our platform, OpenGL implementations, this idea is basically realized through the use of framebuffer objects, or FBOs. So basically again, it's being on a fast path, optimization technique, it really doesn't require you flushing to synchronize pixel buffer object, which we'll talk about a little bit later, and vertex buffer object can use the rendering results in your vertex arrays.

And this is actually modern replacement also for pbuffers. Why use these FBOs instead of pbuffers? Because you know, number one pbuffers will be a you know, a required additional OpenGL context, FBOs don't, and actually switching between frame buffers is actually faster than pbuffers. And of course you get a low memory footprint, you get some savings here because your render buffer and texture images can now be shared amongst your various framebuffers. So at a high level, what's the idea? Well the idea is let's say, many of you have seen this now famous Stamford bunny.

So you have this geometry and some texture, and you want to render this in FBO, and this result will reside, will be resident in an FBO. And then you take that result that you have in an FBO, and you have some arbitrary geometry, and you do render everything normally. And then you have, in this case you have a bunny on a teapot.

So in Direct3D 9 how do you do this with render to texture? Well first thing we have seen how to create a render target texture, and then we declare a structure describing a region that's, we want to be locked. And then we load the data into a texture, then we start manipulating that memory bucket, and then we let go of the texture.

With Direct3D 10, we create a render target texture, we already seen how to do that again. And we then create a render target views, because we want to describe to the pipeline how to access our target texture. And then we create a shade of resource views so the render target can be bound to the pipeline as input. And this is again a recurring theme for all you people who've been using Direct3D 10. Because at one stage or another that's what you'll be doing, initializing some structure, and binding it to one stage or another in your pipeline.

With OpenGL, well you first start defining a framebuffer, then you bind framebuffer. Well you attach a texture to a framebuffer, we already seen how to, basically we know where this texture ID comes from. And then you do a test, you make sure that a frame buffer is valid, and then you draw to the texture.

You rebind the default, to default framebuffer, and then you use a, using a texture ID, that texture ID that we got, you use it as a source for your rendering. So let's now talk about, a little bit about pixel buffer objects. Now pixel buffer objects, there's really no analogous idea with Direct3D 9 or Direct3D 10. So in order to talk about pixel buffer object, you really need to see how pixel buffer objects are used.

So, basic idea of pixel buffer object's main application is if you want to do fast copies of pixel data between various buffer objects. And some of its main application of pixel buffer objects is streaming texture updates, stream draw pixels, asynchronous read pixels, and render-to-vertex array. Let's examine each of them just briefly.

So let's look at why pixel buffer objects are ideal candidate when you're doing the streaming texture objects. So if your applications are using the glMapBuffer and glUnmapBuffer combination, and you're using the data for texture sub image, and you're writing that into a buffer object, you want to eliminate an extra texture that is created during the download process. So by doing so, by using pixel buffer objects, you'll be significantly increasing your performance. Again, that's same here being performance.

For streaming draw pixels, well when you issue a glDrawPixel, glDrawPixel sources a client memory. So client memory then can be modified right after glDrawPixel command returns, because you don't want to disturb the image that you're drawing into. And this basically necessitates basically unpacking and copying the image prior to glDrawPixel call returning. So by using a PBO in conjunction with glDrawPixels, with pixel pack buffer object, your image will return prior to unpacking, again an optimization technique.

Asynchronous glReadPixels. So in this case your application, let's say your application needs to read back a number of images, and process each of those images on a CPU. So this, because of the architects, general architecture OpenGL, this is really difficult to pipeline. So what you, what will typically happen is driver typically usually sends a hardware, a ReadPixel, and then it will wait for all this data to be available before returning the control to your application.

And your application at this stage can also issue another glReadPixel, or they can process the data immediately. And in neither of these cases there will be a lack of parallelism, and read backs will basically, will overlap with the process. So here again using pixel buffer objects with GLReadPixel, your application can issue multiple read backs into multiple buffer objects, and map each one of the processes on that, and you'll introduce a measure of optimization parallelism into your application.

And last but not least, render-to-vertex array. So here you have your application that is going to be using a fragment program to render some image into one of its buffers. And then you want to read this image into a buffer object using your ReadPixels, and then use this buffer object as a source of your vertex data. So let's see, we'll just concentrate only on a couple of these things. Let's first look at render-to-vertex array. So first let's create a buffer object, our usual for a few of our vertices. And then let's render the vertex data into a framebuffer using a fragment program.

And then we going to read the vertex data back from the framebuffer, and we'll bind the vertex, or a binding point. And then we draw, so we're done with that. So let's talk about asynchronous read pixels. The first thing you want to do, you want to create and bind a pixel buffer object as a destination for reading pixels. Then you render to framebuffer, you bind a pixel buffer object, and store the read back pixels asynchronously. Now here the glReadPixel call will return after DMA transfer, then you process the image.

And of course here we'll call the glMapBuffer, and we get a memory bucket back. And then you unmap the image buffer. So we've talked about some optimization techniques here. Now in order for us to further optimize our code, and basically introduce some stunning visual effects into our application, into our digital content, let's talk about using shaders.

So why use shaders? So for many years graphic pipelines operated using a fixed-function pipeline. And this introduced a certain level of determinism in each stage. And, but unfortunately with having this kind of deterministic behavior, you had really little control over how geometry was rendered or processed. But like anything else, graphics hardwares evolved. And you know, most developers required a higher level of flexibility. And initially this problem was solved by introducing extensions, and extensions kind of took away that element of determinism, by just a tiny bit of a degree.

So, but graphics vendors, because it was such a competitive market, they wanted to actually introduce changes later in their hardware development cycle. So they introduced this concept of microcode programs. And as soon as developers heard about this, they wanted to actually utilize this functionality. So this initially, this kind of flexibility made its appearance in DirectX 8, and little bit later in, it became available in OpenGL.

So shaders then basically changed the entire landscape. And first version of the shaders were basically low level shaders, in case of OpenGL the concept really converged under our vertex programs, our fragment programs. But later on, as again the shared languages evolved, the high level shaders made their appearance, and they made their appearance in the shader program languages, Cg, HLSL, and GLSL. So let's look at how actually we do these things, we use shaders in Direct3D 9.

So let's assume that we already written your shader in HLSL, you already have that going. So first thing you want to do, you want to compile a shader. And then you create a vertex shader, and you pretty much, well you're almost done. So then you set some parameters for your vertex shader in your device, you initialize some shader constant, it could be a, your float constant, your integer constant, or your bool constants. And then you tell the device where the input data's coming from. And you basically repeat this process for your pixel shaders under Direct3D 9.

With Direct3D 10 shaders, the landscape has changed, because versus when you have access to low level shaders and high level shaders with Direct 3D 9 and HLSL 3, with Direct3D 10 you no longer have that luxury. You can only target Shader Model 4. But of course the offline compilation of shaders is in a bytecode format still supported.

And of course the stages for Direct 3D 10 are you compile a shader, you create a shader object, and you set the shader object. And basically you repeat this process for your pixel shader, but you also here have you know, you do this also for your vertex shader, but here you also have the geometry shaders as well.

So how do you do this? Well you do, let's say that you've written your HLSL for shader, and then you, first thing you want to do, you want to compile your shader. Then once the shader is compiled, you want to create a shader object, then you want to set the shader object. By now you should have a ID3D10Device interface, you have instantiated that, and you want to, once you have the shader object, you want to just pass it here. And you're ready to use your shader object, and it's right now bound to a stage in your pipeline.

With OpenGL shaders there are really three things that we need to talk about. Let's talk about first the vertex shaders, and what vertex shaders are. So vertex shader is bound to the vertex processor stage of the pipeline. What you can perform in the vertex shader is you can do vertex position transform using model view and projection matrices, normal transformation, and if you needed normalization, texture coordinate generation and transformation, lighting vertex or computing values for lighting per pixel.

You can do color computation, or you can do vertex. And the thing that I want to impress on you here, that remember that in case of a vertex shader you'll be working on per vertex basis. And you definitely have here in this case, if you remember the OpenGL pipeline architecture, you have no access to the framebuffer. So how do you do this? Well you create a vertex shader object first of all, and now that you have a vertex shader object, you have a source, which could be a CString or a text file.

And then you compile the vertex shader. The next thing we want to talk about is a fragment shaders. So fragment shaders are bound to the fragment processor stage of the OpenGL pipeline. They can perform some functionality, and those functionality being if you're working with computing colors and texture coordinates per pixel, if you do texture applications for computation or you're computing normals for lighting per pixels.

And fragment shaders has access to your OpenGL state, and they can't change pixel coordinates. And of course in this case, again fragment shader has no access to the framebuffer. The workflow is similar for creating a vertex shader, so you first create a fragment shader object. Again, you have a fragment shader object, you have a fragment shader source that comes from your C file, sorry, comes from your CString, or a text file, and then you compile your fragment shader, okay? So let's say you've successfully compiled your fragment shader and a vertex shader.

Then you want to bind these to a program object, okay? So let's first create a program object, then you first attach your vertex shader to your program object, then you attach your fragment shader to a program object. And here you link your program object, and if you've successfully linked your program object, now it's time for you to use your program object, and apply it to your various geometries.

So in review, what did we do? Well we first created a vertex shader, then we created a fragment shader. And if we were successful in those stages, we attached it to a program object, and then we linked the program object when we were ready to use the program object.

So of course each of these things, if you're coming from Direct3D 9 or 10, and an OpenGL site, they need your shaders, and your shaders are basically authored in the high level languages like HLSL or GLSL. But instead of going through you know, a detailed dissertation on what are the various things that you need to do to go from HLSL to GLSL, let's just talk about some common themes.

So to that end we just going to concentrate on some simple data structures that you probably will run into in both HLSL and GLSL. Here I'm going to concentrate on HLSL shader language four. So what are some of the common data types? Well you have, you can have a bool, a float, int or double. And this, these can be, in this case they're vectors.

They can have one component, in which case you're talking about scalers, or up to four components. But in HLSL shader language four you can also use the SDL style notation. And vectors, you can access their members using either position set or a color set. But you know, as usual, you know, you just can't mix and match the color set or position set.

And with HLSL you also have matrices. They can have, they could be one by one matrix, which is not terrible interesting, up to four by four, and they can be int, float, or double. But you can also use again, SDL style notation to actually instantiate a matrix object.

You can access their, members of a matrix using a zero base column position, or a one base row column position. And a matrix element can be accessed using array notation, and I have some examples down there for you where I have a two by two float matrix, and I have a vector V, which is this case is a one vector, or a scaler. And I have a float too, which is, has two elements. And then I'm just copying from my matrix, two by two matrix, into my scaler, or in my two vector.

So in GLSL again, the simple data types, the scaler types are float, bool, and integers. And we have texture sampling types, which are sampler {1,2,3}D, or samplerCube. And just like C, you can initialize variables when you declare them, so here I have a float, I have two scalers A and B, and I have an integer, I'm assigning a value two to it, and a bool that I'm assigning a value true. And GLSL again, heavily relies on using constructor for initializing and its type casting. Here I have an integer, a scaler, and then I have a float and I want to convert that integer to a float. So I can use constructor to do that.

So here vectors are also available, so we have a two vector, a three vector, and a four vector. In the first case there are floats, and then B vectors are of course your booleans, and I vectors are just basically integers. They can have up to two, or from two to four elements.

And you can use constructors for vectors, and here let's look at some simple examples. Let's start with a declaration called U which is a two vector, and V which is also a two vector. And here I want to just copy that using my constructor into a four vector. And again, you can use the, mix and match that with the scaler types as well.

And accessing the members is again, very similar to HLSL. But here we have both the position, if you want to use access, the individual members of a vector, you can use, we have texture coordinates available, as well as color coordinates. And in GLSL you know, arrays can be declared just using identical notation as in C. We have matrices, there's two matrix, three matrix, or four matrix.

Again, you can use constructors here as well. Here on the second example I have a two vector U, and a two vector V, and then I'm using a constructor to just construct a four by four matrix. And you can actually specify all the elements of a four by four matrix. And then you can use your basically row and column selectors to select individual members of a matrix.

And structures are very similar to C, and you can actually use constructors against, with structures here. And you can access elements of your structure, members of your structures using the dot notation like you would normally in C. And you have qualifiers like const, attributes which are global variables that may change per vertex. And they're actually passed from your OpenGL application to your vertex shaders, and they are really only variables for vertex shaders.

You have uniforms, which are basically your global variables that may change per primitive, and they're passed from your application to the shaders, and they can be used by both vertex and fragment shaders. And again, this is a read-only variable, and there's also varying qualifier, which is used for interpolating data between vertex and fragment shaders. So I'm not, I'm not you know, not talking about the four loops, Y loops, if then elses, because you know, if then elses in HLSL all if then elses on GLSL same with your loop constructs.

So at this stage I want to bring Michael Jurewitz one more time to the stage, and he's going to go through with you on a second demo. Michael?

I want to go ahead and build and run that for you so we can take a look at what's going on. You can see, we've got a simple OpenGL view here. We're actually at this point now responding to mouse events. So here I'm actually dragging the mouse back and forth, and I can reorient the drawing. I can also go ahead and scale this in and out.

So let's go ahead and dive into the code, and take a look at what's going on. So we'll go ahead and take a look at our header, just like before. And again, you'll see we're a simple NSOpenGLView subclass, just GLSL basics view. And we set up some basic state for our drawing. We haven't defined any custom methods here, we're keeping things very, very simple. So let's go ahead and take a look at the implementation.

So the first thing we want to take a look at is initWithFrame. So for any NSView subclass after they are instantiated by the nib at nib load time, they're sent the message initWithFrame. So this is the designated initializer for NSViews. Now NSOpenGLView gets a little more specific, and so we actually have init WithFrame pixel format that we end up calling. This is a designated initializer for NSOpenGLView.

So in object oriented programming, a designated initializer is this idea that you can have multiple initializers that'll take different parameters, or different numbers of parameters, but that you really only want to have one that's doing the work for proper encapsulation, and to make sure that you have one entry point for your class.

So in this case we use initWithFrame pixelFormat. So let's go ahead and just scroll up here, and we'll take a look at what's actually happening in this method. So in initWithFrame pixelFormat we simply want to make sure that we're actually, that we get in a valid pixel format, and if we don't, we're going to go ahead and set up our own.

Now as with any object initialization in Cocoa, the first thing we want to do is give our super class a chance to set up its state. So you'll see we have this statement here, which is super initWithFrame pixelFormat, and we simply assign that back to self with our own instance. In this case we've wrapped it in an if statement, because we want to make sure that we actually are getting back a valid object, and not nil.

So we go ahead first, and we set up some basic OpenGL state for ourselves. We're going to load our shaders in, we're also going to be at that point setting up our frame size, and also setting up a timer for doing the animation that we're actually doing to the screen. We're also setting up some further state later on, and I'll get to that in a second. So let's take a look at our load shaders from resource method first.

So you'll see in load shaders from resource, we have a couple of methods that are really doing all the work. We have getVertexShaderSourceFromResource, and we also have getFragmentShaderSourceFromResource. In this case we're just passing in the input to our function, which is the shader's name, or in this case for our function which is plasma. So let's go ahead and look at what these methods are actually doing.

So getVertexShaderSourceFromResource actually is just calling a helper method that we have, and passing in the extension of the file that we're after. If we take a look at our, if we take a look at our project here, go back to here, we have a couple of shaders that we've defined, Plasma.frag and Plasma.vert, our fragment and our vertex shaders. So now we're go, so what we're doing is we're actually loading these in at run time.

So when you're doing work with Cocoa, it's very common that you're going to have resources that you want to load in, such as these shaders. So we do here is down in the targets phase we can see for our application we have a copy bundle resources. We've gone ahead and included our Plasma.frag and our Plasma.vert files in there. What this'll do is when the app is actually compiled and built, these are going to reside in our main application bundles resources directory, and we'll be able to get to it at run time.

So back to the code here, you can see that we're doing exactly that. We ask the NS bundle class for the main bundle, which is just the application bundle, and then we ask it for the pass the resource that we want, whether that's Plasma.vert or Plasma.frag. Once we have that, we go ahead and read this file, and make an NSString out of it, and convert that string to a CString before passing it back on to OpenGL. So let's head back to our initWithFrame pixelFormat.

So you'll see one of the next things we do is we set up our FrameSize. We set our FrameSize, as you see one of the arguments to initWithFrame is a frame. Frame in Cocoa is simply a rectangle, it defines where on the screen we're going to be. So we have an origin, a width, and a height. We want to make sure that we're just, we set that up for consistency.

So let's take a look at new update timer, which is where we're actually setting up the callbacks to do the drawing. You'll see what we do here is we use NSTimer to create a timer that's going to happen a periodic, at periodic intervals, and we pass in a selector for heartbeat. You'll see up here that heartbeat is simply a method that we use to say we need to draw back to the screen by calling self set needs display YES.

So you can think of this purely as a callback, that's exactly the function that it fills. So we want to take this timer, and we actually installed this on our run loop so that it'll, so that we can get it called. And then we want to make sure that we're going to, that we avoid screen tearing, so we've set up some further state with OpenGL.

If we head back to initWithFrame pixelFormat, you'll notice that after that we simply do a, we want to make sure that if our FrameSize changes, that we let things that might be interested in that know about that event. So we simply set up the state here to make sure that we post a notification if the frame size is going to change.

So now again, you remember that we were actually handling mouse events in this view. So let's go ahead and take a look at some of these mouse event handling methods. Now every NSView is actually a subclass of an abstract class called NS responder. NS responder is a class that sets up all the event handling mechanisms throughout the app kit. In this case, the message that we've defined here, such as mouse down, mouse up, these are all built in methods for NS responder.

So let's go ahead and take a look at mouse drag, cause that's actually the most, one of the first interesting methods. You'll see that we take this parameter called the event, which is simply an NS event. It has many characteristics about what exactly has happened as part of this mouse down.

So once we get this mouse down, we want to check to see is it the left mouse button or the right mouse button. If it's the right mouse, we want to go ahead and dispatch that event to our right mouse dragged event. Otherwise, we're just going to find out where in the window that event occurred, and we're going to update the orientation of our shape as a result, and we'll call self setNeedsDisplay to make sure that we display that to the screen.

Similarly, in right mouse drag we do, we find out where that event occurred on screen, and based on the movement, we're going to go ahead and change the zoom, and set whether or not we need to redisplay to the screen. And that's all you have to do to set up a very basic OpenGL view where you're at, where you have full event handling in a Cocoa application. Babak, back to you.

Thank you Michael. Slides please.

Okay, so at this stage if you want more information, we have our evangelist for you, who's Allan Schaffer, and he's going to take care of all your 2D 3D graphics needs. And we also have documentation, sample code, and many other resources that you can go to developer.apple.com/wwdc2007/ site.

We have couple of interesting sessions coming up for you tomorrow. And in fact the last demo that we did was actually an homage to two of our OpenGL engineers, that geometry that you saw in the last demo was initially used by our own Geoff Stahl, and he's going to be talking to you about leveraging the OpenGL Shading Language. And the shader, the plasma shader was actually an homage to Kent Miller, and he's going to be in the session Tune Your OpenGL Application.

But before letting you go, I want to impress on you some final thoughts. And basically what did we talk? Well we talked about a lot of optimization techniques. And we also talked about lot of common themes. You saw that while there were, you know, you didn't see any one to one or onto mapping between APIs from Direct3D 9 and 10 to OpenGL. But you saw the ideas were common.

So how can you utilize these two concepts? Well they are really tightly coupled, and they're really you know, married together under the concept of refactoring. So if you have refactored your code, you can just bring your Direct3D 9 and Direct3D 10 application to our platform rather rapidly. And when, with that -