My name is Ian Ollmann and I'm the first speaker in a series that will talk to you about OpenCL. OpenCL is Snow Leopard technology which was just released a couple of months ago, this fall. And, I wanted to give you an overview of what OpenCL is all about before we get started on the more in-depth talks. So, first I want to discuss OpenCL design and philosophy, and then explore some common developer questions about, that have emerged over the last year.

Some of you may have seen them on the developer forums. And then, offer up a few debugging tips about how we go about to tune our OpenCL kernels. Now, since OpenCL has been out for awhile, I'm going to assume that many of you have already played with it.

Maybe you've downloaded some of the sample code and looked at it, read through the spec, maybe even tried to write a few applications of your own. And so, what I wanted to do is provide an overview to kind of tie it all back together, because there are a lot of APIs and a lot of spec to read, and sometimes seeing the forest for the trees is a little tricky. So, when we set out to build OpenCL, what we wanted to do is bring programmability to all of the devices on your system. One API to bind them all; CPU, GPU. If we had them, you could also program an accelerator.

And, this brings in new problems into the programming paradigms. Some of these devices may have their own memory attached to them, which sit off in a separate address space and do not fit together contiguously with the RAM you're used to using on your system in your ordinary C or C++ or Objective-C program.

Many of these devices run on a different instruction set than the Intel instruction set that you're used to. So, we have to overcome these problems when we build OpenCL. So, OpenCL, in a nutshell, is sort of the minimum set of objects that you would need to encapsulate this information and make it all work.

We have an object which represents a device. This might be a CPU or a GPU. We have a context, which does not do very much; it's just a sandbox to hold everything else. This serves two purposes. It allows you to keep the damage localized if something goes horribly wrong, which we hope won't happen. And also, if you're doing data sharing between OpenCL and OpenGL, it acts as a counterpart to the CGL share group.

You'll want to put your data into something. Because we have to copy the data up to a device sometimes, we need to know how big your data is, so a simple C pointer is not enough; we need an extent. So, the object's designed to encapsulate your data. There are two kinds. There is a buffer, which is almost exactly like what you'd get out of malloc .

When you call malloc, it's essentially a range of bytes; what you put it in is up to you. And then, there's image data type, which is useful for sample data in a regularly spaced grid, and images are designed to be used by the GPU texture unit, which has hardware to make sampling out of the image much faster. Collectively, these things are called MemObjects.

You also want to write code, so OpenCL provides a C like programming language. We essentially started with C99, and then we tagged on a few extra features, like vectors and vector intrinsics, and we stripped out a little bit of the C standard library that didn't make any sense for GPUs.

And then, so you'll build a program against multiple devices, and then you'll need some sort of function pointer-like thing to go find the functions within that program or compilation unit, and these things are called kernels. And, you can make many kernels for the same function if you want, and that actually turns out to be quite useful.

Finally, we have to have some verbs in this sentence, and those are the command queues. You can queue commands into the queue to make all of these objects start doing things. They might be to copy data from one place to another. They might be to run a program on some of your data. And, that's basically it in a nutshell.

The important thing with the command queues though, is that these are fully asynchronous queues. So, when the device is finished with command one, it's going to want to go straight to command two, and if there isn't a command two, then it's going to sit and be idle, and you're not taking full advantage of the computational horsepower of your entire system when they're sitting idle.

So, what you really want to do is enqueue a pile of stuff into the command queues and let it take off and run in asynchronous fashion. As an example of how these things work, here I have, at top left, some data that you might have allocated, called My Buffer or My Image. And then, in your program somewhere; I put it in Main, but it can be anywhere, you'd start enqueueing commands to then operate on that data. We might start with an enqueue to write a buffer, and what this does is copy data from your buffer into OpenCL's counterpart.

You might call EnqueueWriteImage, which will do the same thing for images. And then, you can call EnqueueNDRangeKernel, which will copy all of that data automatically up to whatever device you enqueue it for. The device will run your program, and something will happen. You could have some results. And then finally, you might enqueue a read buffer to copy the data back to your thing, and proceed in that way.

The OpenCL API itself is very consistent. We have about eight object types and they all have the same set of functions that are used with them. They have a creation function; it's used to create the object. We turn the object out the left side and enter out of the right. These things follow Core Foundation reference counting semantics, so you'll find that familiar probably with retain and release. When the reference count goes to zero, the object is then destroyed in the background by OpenCL.

We have getters and setters in the standard object paradigm to get information in and out of the object. Almost all of the objects have getters. Only a very few of them have setters, because that introduces mutable state into the object, which makes it harder to make a thread safe back in. So, and then finally, if you want to queue a command, we have a clEnqueue and then some command type that--a bunch of functions like that.

So, the next question is; OK, I've got this object infrastructure, but I don't really understand, like how I write code for it. I mean, what does the code I actually write look like? And, this is the code I mean that you run on the device, not the thing running in your regular C program.

So, we have to find some way to split up lots and lots of parallelism in a way that you write your code where it still makes sense to you. So, a traditional way to do that would be to invoke task level parallelism. This is something you might have done yourself, you know, using Pthreads or NSThread or something, where you divide the work up into different tasks.

So, if we take a mail reader app as an example, we might have a thread to get the mail from the server, another one to scan through the results and identify junk mail as it comes in, another one that might run mail filters, a thread to draw the UI, a thread to get keyboard input, a thread to play the audio if you want to make a little beeping noise.

You know, we can divide up many such tasks like that and, you know, usually you come up with five or ten different things you can do, and that's great when you've got five or ten different processors to work with, because you can get a five- or a ten-way parallelism if you can manage to stack those things all up concurrently.

But, in OpenCL we're really targeting a much more parallel system than that. We're going after many core systems, you know, that might have hundreds or thousands of cores, so how do you break up your workload in a way that's more amenable to those kinds of systems? And, what we can do is just learn from standard shader languages, like OpenGL shader language, and take advantage of data-level parallelism; that is, rather than breaking things up by different kinds of tasks, we break things along data boundaries. So, if we continue our example with the email reader, you might have a separate thread for each email.

So, if you're like me and you come in in the morning and you have 200 emails waiting for you to waste a good part of your morning, then you know, this would be a great way to get 200 way parallelism out of your system. And, hopefully these kinds of computations that you're doing on the different data elements are largely independent. You know, in my case, what happens to one email probably doesn't influence much what happens to another one. And, assuming you have enough data, then you can presumably get your 1,000 way parallelism potential.

Or, better yet, you know, find a million way parallelism in there. And, this is exactly the sort of problem that OpenCL is set up to go after, is where you can just get massive parallelism in your computation.

So, as an example of a way to break things down, let's take this image. This is an old OpenCL logo. And, I've broken it down so that you can see each pixel outlined in the grid, and each pixel we would call a workitem, and we would run a single function against that pixel.

So, for even a very small image of, you know, one kilopixels by one kilopixels, which is much smaller than you can get out of most cameras today, you still easily see I could get a million workitems out of this and you can get a great deal of parallelism.

And, what OpenCL will do is essentially implement the outer loop for you and go through and call the function in turn for each one of these workitems. And then, in your function, when you get called, you call a little get_global_id function, which tells you which one you are, and then you go operate on that one.

And then, just to be complete, it's not really required that your dimensions of your problem map directly to the workitem dimensions that you've called with OpenCL; it can be another dimension heading off in a orthogonal space and you can map it back, and I'll talk more about that in a minute.

However, there is a little bit of a complication here. It turns out that modern silicon is generally not a vast array of very simple scalar processors all running in parallel. Over the last few decades, as you know, we've been adding all sorts of ways to get instruction level and data level parallelism into the single instruction stream. We have super scalar processors, symmetric multi-threading. You can do a lot of work at the same time in a SIMD fashion using vector units.

And, it turns out that if we actually look at the possibility for doing concurrent work on general modern cores, these things are capable of running many, and sometimes a great many, workitems concurrently. So, what we do is we take our giant grid of workitems, and we break it out into little groups, called workgroups. And, here I've outlined them in yellow, in one way of breaking down the problem.

And, we run a workgroup on a particular core, and all of the workitems in that workgroup will run as roughly concurrently, more or less, on that core. And this is actually a very convenient motif, because it means that you have spatially adjacent data working together at roughly the same time, which means you get great cachetilization. You can share resources, like local memory.

If one workitem accesses a piece of memory and another one right next to it accesses the piece of memory right next to it and so on down the line, the compiler might be smart enough to spot; hey, you just loaded a continuous range of data, and turn that into one big load rather than doing it in smaller loads.

So, that's called a coalesced load, or you can do it with storage, too. It's also very cheap to synchronize. It could be as cheap as a NoOP to get all of those things to work together, because you're essentially doing it all in software. Or, it could be a little hardware interlock within a single core. What's really hard to do though, is to synchronize between workgroups, because they're running on different cores, which might be far apart on the silicon. There's actually sort of speed-of-light information travel problems to get from one core to another, so communication can be slow.

It would add complexity to the chip to have each core capable of being interrupted by every other core, in sort of an N-squared fashion. And then finally, you run up into sort of a memory limit. If we imagine that we have a relatively small image, say a million pixels, and we make a little stack for each workitem, which is four kilobytes, again very modest. Multiply that and quickly you just realize you've just exhausted your address space in a 32-bit process. So, we can't actually have all of those stacks living at the same time.

So, it's not possible to say, put a barrier in the middle of your code, have all million workitems come up to that barrier, then once they get there they all continue. It just doesn't work. So, how do we map workitems directly to hardware? There are varieties of ways to do this.

One way is a simple, direct model, and this is what we do on a CPU today in Snow Leopard, where essentially one workitem is one hardware thread running there, just like a Pthread. And, in this model, you have to vectorize your code if you want to get full use out of the hardware available, and then there might be other sources of parallelism on the core that you might want to take advantage of, like superscalerism or try to use the reorder buffers to get more to happen concurrently.

But, there are other ways to do it in a more GPU-like fashion. We can parallelize our work items through the SIMD engine, and here we run a separate workitem down each lane of the vector register in the vector unit. We can go further than that. We can write a software loop to do multiple vectors at a time; meaning, in this case, 32 workitems at a time.

And of course, since it's a loop, we can turn the loop over many, many times, and pretty soon you can see how you can end up with a workgroup that's hundreds or thousands of workitems in size, all running on the same piece of parallel hardware. And finally, we can parallelize this in a different way, using SMT on each one of these vector things down through a symmetric multi-threading engine, and use hardware to do all of the scheduling. So, these are just simple examples of how it might happen.

There are some ways to synchronize within a workgroup. The simplest is mem_fence. That actually synchronizes within a single workitem. And, it's mostly there just for hardware that has really weak memory ordering. This is largely unnecessary on Mac OS X, because all of our hardware is more strongly ordered than that. So, you should not need to use mem_fence. Barrier is very important, however.

If you are working with local memory, you'll find that you will want to copy data over. Make sure all of the copies are done, issue a barrier, and then proceed forward. Now you can use the data that's in local memory. But, barriers only work across a single workgroup for the reasons I mentioned before. Then finally, there's a call, wait_group_events, which is a barrier-like synchronization, but it works with an asynchronous sort of switch that copies data around, called async_workgroup_copy.

There is also a problem with figuring out how many workitems to put in your workgroup. You can pick just about any number, as long as the hardware will swallow it. So, what will the hardware swallow? Well, it depends on the hardware. OpenCL provides quite a diversity of different interfaces to try to figure that out.

You can see I have eight of them here. Six of them are APIs and there's also some constraints in the standard; the dimensions of your workgroup have to divide evenly into the total problem size and it might turn out that you need so much local memory per workitem in order to do your work.

There's only so much local memory on the system, so that kind of can limit the size of your workgroup, too. So, that's the bottom entry. So, how do you wade through all of this and try to figure out how big your workgroup can or should be? Well, the first solution is to give up. You can pass in NULL with the workgroup size. We'll swallow that. You can hopefully, you know, do some magic, which we'll try.

We'll do the best we can, and magic is wonderful stuff, but unfortunately it's not real grounded in reality, so we may not get it right for reasons I will describe in a minute. So, in cases where you have more information then you guess we do, it's often best to handle this yourself.

So, another approach is divide and conquer, which is the standard technique, of course. And, you just go through and sort the information you're getting back from OpenCL according to dimensionality. So, you won't need to call all of these interfaces. You'll quickly realize that only some of them really apply to your particular problem.

But, you'll get a number of one-dimensional limits that you might have to take the minimum of; we can't have any more workitems than this. And then, there's a couple of APIs that will give you a 3D shape, which will constrain the size of your workgroup, and this is required because there are devices out there that are only able to vectorize in one dimension and not all three. So, these APIs can be used to do that, and then, of course, the overall size of your global problem set will constrain your data shape because it's got to divide evenly into your workgroup size, or by your workgroup size.

You can run into problems when the global work size is a prime number. What divides into a prime number? Well, the only thing that divides into it is one and the prime number. Prime number's not going to work; it's too big. One means we're only running one workitem per core; you're not going to get any parallelism that way, so it's going to perform terribly. So, what do you do? Well, one thing you can do is just make your problem demand a little bigger.

And then, in your kernel, when you go write your code and you go get, you know, what problem am I, if you have to be outside the original problem size, then you just have an early out. So, that will let you solve the local size must divide into the global size problem and run on a prime number workgroup of global size. Another thing you can do, as I mentioned earlier, is enumerate your workitems out into some abstract dimension.

And then, you write a little function such as this where you take in my global ID in the abstract dimension and map it back to something more real, like your X or Y position. So, these are all things that you can just do yourself in code and, you know, it's limited only by your imagination.

So, I wanted to work through a few developer questions we've had over the years. They come from a variety of topics. Many of you have noticed that we've added half precision to the spec. This is a 16-bit floating point number. And, it's easy to get quite excited about that; ooh, what is this thing? You know; ooh, I have a new float to work with. It's not quite that. It's a storage-only format, which means that it only exists as a 16-bit floating point number in memory.

As soon as you pick it up and start trying to work on it, the first thing OpenCL does is convert it to single precision number. You do all of your arithmetic in single precision, and then when you go to store it back out somewhere, then it gets converted back. Which interface you use to load and store your data depends on whether you're working with buffers or images. Buffers will use vload_halfn/vstore_halfn; images will use read_imagef or write_imagef, as for other pixel formats.

There is an extension you'll see in the end of the spec, which is cl_khr_fp16, which actually specifies half precision direct arithmetic. That's not supported on our platform. There's a couple of good reasons for that. The native hardware doesn't do it; we'd have to emulate it in software, which would be a lot slower than doing it in single precision.

And, the other problem is, of course, half precision only has about 11 bits of precision if you do enough in multiplies and adds and whatever else. You're going to start losing bits and you'll be down to eight, seven, six bits of precision and for most algorithms, that's just not enough.

Many people see that; oh, OpenCL has four address spaces, which are sort of disjoined places to put memory. What's that all about? We have global, which is akin to the main system memory used to use. We have local, which is a little, user managed cache tied to your compute unit.

We also have a constant memory space, and private, which is just local storage for your particular workitem. And, the confusing one is, what is local memory? It's just a user managed cache, and the way you use it is you either explicity or using a convenience function, like async_workgroup_copy, just pick up the data from global memory and write it over there.

So, it's as simple as, you know, just doing an assignment from A to B in your OpenCL C code. So, what you want to do is have all of your workitems work together to copy in the data from global to local, then issue a barrier to make sure everyone's done so that we don't try to read any of it before any of them are done. And now, the data you know is resident in local memory and you can read that out much quicker than it would have taken to access it from global.

Now, a key point about local memory is that it only really works if you touch the data more than once. If you just touched it once, then you would essentially be reading it once, copying it over here and then reading it back from over there. You didn't save yourself any time.

You want to be in a situation where you read it once here, put it over here and then all sorts of different people use this over time. So, if it turns out that you only plan to use your data once, let's say in My Image, I'm just converting RGB to YUV, so each pixel is largely independent and I only touch it once.

Then we wanted to use a variety of different approaches, depending on what kind of hurdle you're on. On a GPU, there's a texture cache designed to accelerate that kind of read-once access, and that is backed up by the image data type, so you'd want to use that. On the CPU, there is a global cache backing up buffers, so as long as your data has good spatial locality, then you should get some acceleration out of the caches.

I should note that local memory, while it seems like a predominantly GPU technology, we have found that it's actually quite helpful on the CPU. It can make vectorization easier. It also can avoid polluting your caches, and what I mean in that sense is, let's say your global data structures are an Array of Structures, data type. Here I have an AoS strapped with x, y and z in it and then some of these telephone numbers, some unrelated piece of data. And, I know many of you just love doing code like that. I've seen it everywhere.

And, this might be an array in your buffer, but when you go actually work on it in your kernel, it can be quite useful to then transpose that around into an array of x's followed by an array of y's and an array of z's. If you only intended to work on x, y and z, and you didn't care about the telephone numbers, then you end up compacting the data down into a much smaller space. Also, because it's plainer in orientation, it's much easier to vectorize. The GPUs have a watchdog timer.

And, what is a watchdog timer? It's somebody looking over your shoulder to make sure you don't monopolize the GPU for too long. And, the reason for that is the UI will not interact as long as you're busy on the GPU. If you use the GPU for more than a few seconds at a time in a single kernel, you'll probably get a message in the console, such as this one shown here. You may see a flash on screen and you definitely will not get the right answer out of OpenCL. Your contacts might be invalidated.

So, if you start running into this, the simple solution is just divide your task up into smaller chunks so that it runs faster and doesn't use up quite as much time. You can enqueue them one after another in the queue; you know, the second one will start as soon as the first one's done, so you won't waste too much time.

You want to be careful though, because the breadth of capability between sort of a low-end GTU and a high-end can be quite large in order of magnitude, maybe. So, if you are doing this, be sure you're testing out a low-end system to make sure that it works everywhere.

Some of you have noticed that OpenCL provides a way to get out of your kernels compiled as a binary. There's this interface, clGetProgramInfo (CL_PROGRAM_BINARIES). And, the intention of this thing is to give you a cache, or a way to create your own cache, to avoid having to compile your kernels every time your app runs.

Some people want to use it for code obfuscation, but it's not suitable for that use right now, and the reason for that is that Apple has not committed to an API for the kernels. And so, that means that on some future OS we might change the API; your binary will not work anymore.

When you go try to load it, you have to call clBuildProgram before you can actually use it, and that call will fail. If the only thing you shipped with your app was the binary, you're now in deep trouble because you have no code to run. So, what you need to do is ship your source. If this happens to you, then you rebuild your source fresh and override the cache that you had set up for yourself, and continue on.

Some developers are curious; when should I use buffers, when should I use images? I think if you're familiar with OpenGL, it should be obvious. But, if you're from CPU land, like me, then it's not so clear. Buffers are sort of native territory for a CPU. We have caches to back them up; they're very fast.

On a GPU, on current generation there's no cache to back up global memory accesses, so you're taking essentially a big, long trip; several hundred cycles out to get uncached memory. GPU, you'd want to use, copy that data in the local memory, which is very close and much faster to use. Or, use coalesced reads wherein multiple workitems are reading data from largely contiguous regions of memory.

Images are great on the GPU. There's a texture unit to accelerate those accesses, if they have great spatial locality. But, on the CPU there's no such hardware, so we have to emulate the whole thing in software. So, these things look a lot more complicated than you would think just looking at the spec. But, at least the CPU is extremely accurate, so it's good for debugging to make sure you're doing the right thing.

But, as you can see, this is the implementation of a single pixel read using linear sampling; it's 168 instructions. So, I would only want to use the read image feature on the CPU if you've budgeted ample CPU time to go through and do all of that work. Some developers are wondering; how do I use OpenCL in my multi-threaded app? It says it's not completely thread safe. The intended design is to use a separate queue from each thread that you intend to enqueue work into OpenCL. The other thing you have to do is make sure you're not getting reentrant access into individual objects.

And, you can end up in some patterns also where you can step on yourself. For example, on this one, I might set the kernel argument to be a value and then get interrupted, have another thread which is using the same kernel come along and set it to a different value and queue its kernel, then finally I wake up and queue my kernel, but it has the wrong argument. So, you want to make sure that you don't get these sorts of races happening in your app.

Now, you could implement some very fancy locking schemes to try to guarantee this, but it's going to be heavy, it's going to damage your concurrency, and you're not going to like it. What we are actually thinking that you would do is, if you intend to call the same kernel for multiple threads, make multiple kernel objects, all pointed back to the same kernel function. That's cheap to do. You don't have to do any locking, because each thread will have its own copy of the kernel object, and it's safe to do.

Probably the number one thing that I've seen developers do is block too much when they're enqueueing stuff into OpenCL. A number of the enqueue APIs have the capacity to block until the work is completely done in OpenCL before returning control to you. But, that's extremely insensitive because you don't get to do anything while OpenCL's doing stuff, and then once OpenCL is done, then it has to wait for you. And, so you end up losing a lot of your concurrency, and I'll show you an example of that a little bit later.

So, there's an API entirely intended to block your queue, clFinish. You should almost never need to call that. The only time I've seen where it was a good use case of that is where somebody wanted to shut down OpenCL completely, wanting to make sure all of the work was done and all of the reference counting had resolved itself and all of the objects were freed and all of the memory was released. So, that's great use for clFinish, but you should otherwise almost never need to do it. Some people just seem to instinctively put it in there proactively after every single call, and it's killing them, I'll tell you.

There are calls to read and write data in and out of OpenCL. These can be made blocking if you want, but you'll only need to be blocking on these some of the time. And, you can probably figure out for yourself when this is, but often you'll see, like for example; I need to enqueue multiple reads to read back results from multiple buffers after my computation. It turns out because the queue is in order, which means each job is finished before the next one can start; you only actually need to block on the last one, because you know that the other ones have already completed.

Likewise, when you're doing writes, the typical pattern is write data into OpenCL and queue a bunch of kernels, and then read back the results; the last one is blocking. Well, OK; I know my write finished a long time ago, way up here, so no need to block on that either. So, there really, in any, like giant sequence of calls, you probably only need one block at the end.

People also run into some performance pitfalls using half-full vectors. This can come in two forms. On the CPU, the vectors are fixed width, they're all 16-bytes on SSE, and so if you write, like a float2, which is only an 8-byte type, you're essentially issuing a 16-byte instruction work on 16 bytes of data, but you've only populated it half-full, and so we end up doing extra work on some, who knows what's in the rest of the register.

So, this is bad for two reasons. Obviously, you've wasted half of your potential to do work. But, in floating point, if those lanes in the vector happen to get any NaNs or infinities or denormals, you might set yourself up to take a hundred cycle stall for each one of those, and that can hit you operation after operation after operation after operation, and make your code run, like orders of magnitude slower than it should. So, you want to be sure, when programming for the CPU, on the direct model like we have, that you try to make sure you use 16-byte vectors or larger.

Larger will actually do a little free unrolling for you, in the compiler is at times a little faster than just using the 16-byte vectors. The GPU has sort of the reverse problem. When it's revectorizing your problem along a different dimension in the way the GPU would like to do it, you might have a float4, but you've only put data in the first two elements and have a bunch of garbage after that because you couldn't figure out what to do and you declared it a float4 somewhere.

Well, when the GPU vectorizes that, it will make a big vector full of x's and a big vector full of y's, and then two big vectors full of junk, which it will then go do arithmetic on needlessly, so that just wastes time. Finally, we've noticed that people often will make objects, use them once, then delete them; make them, use them once, delete them. And, that can be kind of wasteful. Many OpenCL objects are heavy; they're intended to be reused a lot. Like a program, you'd have to compile it each time, which can take a big chunk of a second, sometimes.

Images and buffers have a big, giant backing store, megabytes in size; a bunch of driver calls to set it up, and then there's some state associated with who used it last, so we can track which device the actual data lives on right now. And then, finally, on any buffers that you make that are new on the system are subject to the usual zero full activity that the kernel will do the first time you use it. So, if you reuse them, you save yourself this cost the second and later time.

However, it's only really useful to reuse things if they're about the same size, or in the case of images, exactly the same size as the previous use. Otherwise, OpenCL has no concept of only copy part of this buffer up there; it'll copy the whole thing up to the device.

So, you only really want to reuse them if they're about the same size. So finally, I'd like to talk about a few debugging tips. These are standard techniques we use back in Infinite Loop. Pretty much all of us run with the environment variable CL_LOG_ERRORS set all of the time. I just put that in my bashrc.

And, you can set this to either standard out or standard error or console, depending on where you want the error messages to go. And, what it does is, whenever you call an OpenCL API and you manage to miss some little gotcha in the spec, and OpenCL returns an error out, you also get spewed to the console or standard error or whatever; some hopefully human understandable English message about what exactly you did wrong.

So, if you're encountering any problems with the API, getting that to do what you want, then CL_LOG_ERRORS is very much your friend. There's also a way to programmably hook into it. You can roll your own function and pass it in when the context is created. Finally, when you're working on the CPU, we make heavy use out of Shark and Instruments to see what's going on in there, and what I'd like to do is give you a quick look at what that process looks like.

So, this is an iPhoto. It's unmodified, and as it turns out, iPhoto on Snow Leopard for certain things will use OpenCL on the CPU. So, we can take a look at what it's doing. So, we call up our little Adjust panel and we can start, you know, adding little filters onto the image, and these are all processed in real time, as you can see.

And then, we can go and start jiggling stuff. So, if we run Shark; Shark has a bunch of different ways to run samples, but the two most interesting are Time Profile and System Trace. I'll run a Time Profile first; I'm sure many of you have done this. So, while I'm getting OpenCL to do something through iPhoto, here I'm jiggling this thing around.

I can hit option escape to get it to record, and what it's doing is it's taking a sample server so many milliseconds or microseconds, and records where the CPU or CPUs, what instruction they were on. And then later, it puts it all back together, backtraces it against which functions they were there, and you can get a breakdown as to where your time was going using this stochastic technique. So, you can see it was spending about 12 percent of its time in this function. This is an OpenCL library.

OpenCL provides a number of libraries which are strangely named; they look a little bit more like pixel formats than libraries. So, if you see this, this is a cost accrued to you due to the cost of going to use readImage on the CPU. You will also see lines that say unknown library in them. Well, why doesn't it say OpenCL? Well, this is your code. It doesn't actually exist on disk anywhere.

We just built this code, stuck it in the memory and run it. So, Shark doesn't really know what it is, so it says unknown library. But, you can still drill down and see all of the code that we prepared, and if you're good, you can figure out what parts map onto your kernel and see if you got the code you wanted, and if there are large stalls in here, you can generally figure out what went wrong in your kernel.

I can also show you a system trace, which is very useful for understanding how your interactions with the OpenCL queues are progressing. So, if we go back and play with the image a bit more, I'll now record a system trace for a second or two. This is a 16-core system, or 8-cores with 2ASMT. So, we have here a bunch of iPhoto threads. I'll just limit it down to iPhoto.

And, I'm looking at the timeline, and what you can see here; these are threads, horizontally. Regions that are amber is time during the timeline, which progresses to the right, when the CPU was active. And, what we can see is that it's single-threaded a lot, but then there are these little windows where we're multi-thread.

And, these are when OpenCL is running, and we can zoom in on these things. These little telephones are system calls, and here we can see; here is the main thread on the top and it's running through, making various system calls, and we can go look at this and track this back to OpenCL.

This is a release_mem_object call to release some memory object. This one is enqueue a kernel, and you see a little while after a kernel we get a little blip of something happening. And here we enqueue another kernel, and then this one's a big one. And, it seems like there is some serial process here to kick off each CPU as it goes along, so you can see them all firing up. But, sometime before we manage to get them all fired up, we're already done with the work.

So, maybe the kernel you enqueued is too small, because we didn't actually get all 16 of the threads up and running, the hardware threads, before we ran out of work to do. Well, why did that happen? Well, we can go and look at this one. What's this thing? The main thread wasn't doing anything during this time; that's kind of a little strange. We can go track that back.

Oh, look! It's a clFinish. Apparently the enqueued a kernel and then issued a clFinish to wait for it to be done, and then did some more work and then repeated the process again. You can see another clFinish over here. Well, this is quite costly for a number of reasons. For example, let's see if I can zoom in here.

While we weren't doing anything here, we probably could have been doing this much work in the main thread. It only would have cost OpenCL one thread out of 16. So, it probably wouldn't have slowed down too much. So, you could have gotten this much work done, which meant all of this dead time, all down here, would have gone away and we would have compressed the launch-to-launch time from here to over here by about that much time. So, you can see, just that clFinish, that's what it's costing you.

Another thing is, all of the CPUs go back to sleep after we're done with this, and I only have to wake them up again. If you didn't have that Finish in there, it's possible we would have just picked right up where we left off, and then started running these things, but, except all of the CPUs would be awake now. So, rather than getting a little, tiny bit of work out of these threads, you might have gotten a full width. So, the whole thing, you know, might have been about twice as fast if we go by one-half base times height on this little triangle here.

So, you can use Shark to dig right in to how much residency you're getting on the CPU. And, the same techniques apply in the GPU. You won't see these little triangles, because the work is happening on the GPU, but you will certainly see time that's dead time in your main thread, when you could have been doing something else.

So, that's Shark with OpenCL in a nutshell. And, what I'd like to do now is invite Abe Stephens up to tell you all about how to integrate your workflow between OpenCL and OpenGL, and that allows you to quickly and seamlessly, and with a minimum of cost, share data between the two.

Hi. My name is Abe Stephens and I work with the OpenCL Group at Apple. Today I'm going to talk about OpenCL and OpenGL sharing, which is a mechanism that allows us to create objects in OpenGL and then import them into OpenCL in such a way that the actual data that is operated on by both APIs is the same. So, we can avoid copying data or making duplicate copies of the data, and accelerate our programs.

The motivation here is that we'd like to combine these two APIs. Now, OpenCL and OpenGL are similar in many ways. A lot of programs that are written for both APIs run on the GPU. It's possible to write programs for the CPU in OpenCL, and that's a little bit less common for OpenGL, but really, we're looking for a mechanism that allows us to move data efficiently between these two interfaces.

Let's take a look at a simple example; a case where we have an application that's going to perform some kind of physical simulation and then visualize the results. The physical simulation part of this example; for example, a bunch of objects bouncing around the screen, is a very OpenCL oriented kind of task. It might involve collision detection, computing maybe Newtonian mechanics or something like that. And then, it also might involve rendering.

We could compute our position and velocity in our compute side of the application, and then take that data, move it to graphics, and actually render the scene. Now, this type of task could also be performed without OpenCL. We could produce data on the CPU and then transfer it to the GPU, and then in the next frame, repeat that process.

Alternatively, with a sharing API, with a sharing mechanism, we could produce the data in CL on the GPU, and then move it from the CL side to the GL side, and use the same data over again. So, let's take a look. In OpenCL, we'll produce a list of vertex data and then move that data from CL into OpenGL, where we can provide, or we can implement, some kind of sophisticated shading operation. In this example, we've rendered these spheres with a refraction shader and some other special effects.

That shader operation is obviously best suited to graphics, and the physics operation in this case is very well suited to OpenCL. We can also perform the opposite kind of operation. Instead of producing data in OpenCL, we can consume the data in OpenCL and produce something in OpenGL. For example, in that previous slide, OpenGL could produce data about surface normals and surface positions of fragments, and then that data could be passed into OpenGL for some type of post-process.

So, for example, in our physics application, we could render the spheres using OpenGL to a vertex buffer or to a fragment buffer object, and then take a surface of that fragment buffer object, transfer it to OpenCL, where the CL program might use that data as input to a ray tracer, trace a caustic effect, and then move that caustic effect back into OpenGL for final display and compositing.

Now, under the hood, under the hood both of these applications, or both of these APIs, operate using similar data structures in the driver, and are implemented using shared structures. As you might be familiar, OpenGL selects the devices that are used for computing using a pixel format. So, a developer or a programmer sets up a pixel format and that format is matched with the rendering devices in the system, GPUs and CPUs.

Matching the pixel format to system devices and then passing that format to a create function generates an OpenGL context. At this point in setting up a OpenGL system, the programmer is done. We can use this OpenGL context to start sending draw commands to the system. But, if we wanted to take our program a step further and add an OpenCL application or an OpenCL task, we have to take the context and convert it into OpenCL, and this is done by extracting the ShareGroup from the GLContext, and then taking that ShareGroup and moving the ShareGroup into an OpenCL context. And then, if we looked inside that OpenCL context, we'd see that the CLContext contained all of the devices that were originally in that pixel format.

Now, the CGL structure, the CGLShareGroup, and the OpenCL context both contain very analogous types of structures. The ShareGroup has a list of vertex buffer objects, textures and render buffers, and the CLContext has data objects that are wrapped by the cl_mem object type. When we convert the ShareGroup into a CLContext, we're making all of those GL data structures available to the CLContext, and then we have to obtain references to those structures from the CLContext to use in our program.

There's some relationship between the CGLContext that's used by OpenGL, and the list of CL devices, although in CL, we have a very explicit representation for the device. In OpenGL, we select a virtual screen, or we use another mechanism to select a rendering device that the system will use. So, although there's a relationship, the devices aren't exactly analogous.

There are five steps for setting up a sharing process between OpenGL and OpenCL. We've already looked at the first step, which was to obtain that CGLContext. From the Context, we obtain a ShareGroup and then use that ShareGroup to create the CLContext with which we'll send commands on the CL side. After the Context has been created, we can import data objects from OpenGL to OpenCL. In the first example that we looked at, those data objects consisted of a vertex buffer and then in that second post-process example, the shared data object was a GL render buffer.

After we've imported the data objects, the set-up phase of our application is complete, and now we can concentrate on executing commands, and there's a specific flush and acquire semantic that we have to use to coordinate between the OpenGL side of the program and the OpenCL side of the program. And then, finally, when we're done, we have to tear the whole system apart and clean up by making sure to release the objects safely in CL before destroying them in GL.

So, let's take a look at some source code. The first step is to obtain the CGLShareGroup for an application from the CGLContext. And, the example that we'll look at will focus on a Cocoa application. In Cocoa, the NSOpenGLView is commonly extended to add OpenGL to an application. Within the initialization function of our Cocoa program, we can obtain first the CGLContext associated with that OpenGLView, using an accessor function. And then, we use the CGLContext to obtain a CGLShareGroup, which is essentially what we'll use to create the CLContext.

Now, the second step in our five-step process is to take that ShareGroup and use it to create a CLContext. We do this using OpenCL; an OpenCL function with a special enumerant, the CL_CONTEXT_PROPERTY_USE_CGL_SHAREGROUP_APPLE enumerant, which is really hard to forget. And, this is passed in a property list to the CLCreateContext function and in this case, without any other arguments except for the error argument.

Now that we've created a CLContext, we have to obtain a list of CL devices, and if you remember from the slide at the very beginning of the talk, those devices will reflect the devices that were used, that were passed in the pixel format when that CGLContext was originally created. In a standard Cocoa application, the runtime has already taken care of creating a pixel format, and we simply obtained the CGLContext that was provided by the runtime.

If we wanted to obtain all of the devices that were associated with our CGLShareGroup in our CGLContext, we could simply use the existing or the standard CL accessor method, clGetContextInfo, passing it the correct enumerant, and get a list of all of the devices. However, in that case, we might obtain a GPU device that doesn't correspond to the GPU device our Cocoa application is currently using for display. If we want to obtain the device that's currently, the virtual screen that's currently being used for display, we have to use another special enumerant and another special function, clGetGLContextInfoAPPLE, to obtain the CL device that matches the current virtual screen that our application's running on.

So, let's take a look now at what we've done. We've made our way around this entire figure, and we're back to the point where we have a list of CL devices inside of our CLContext; however, if we look at those devices, we'll notice that the CL API has removed the CPU device from that initial list of GL devices.

If we want to add the CPU device back in; for example, if we wanted to run some CL programs or some CL kernels on the CPU, and others on the GPU, when we create our CLContext we have to explicitly add the CPU device, as if we were creating a normal CLContext. So, this involves getting a list of device IDs with the CPU device type, and then passing that to CLCreateContext.

OK, now here's the fun part. Now that we have our Context, our CLContext, we have to move data objects, or tell the runtime which data objects we'd like to move from OpenGL into OpenCL. We have to import the shared objects. OK, so here are the two ShareGroups we'd like to end up with. If we'd like to move that vertex buffer object into OpenCL, we use the function clCreateFromGLBuffer, passing it both the CLContext and the GLBuffer, and then memory access flags that tell OpenCL how we plan on using that data structure.

Now, if we look under the hood, what we've really done by sharing the vertex buffer object between OpenGL and OpenCL is we've created a structure within the OpenGL runtime and the OpenCL runtime, but those two structures, the VBO and the cl_mem object, actually refer to the same piece of driver storage. This piece of driver storage, called a backing store, is the actual memory that contains that vertex data, or texture data, or render buffer data, and is the piece of memory that is moved between devices as we execute our program.

Now, if we execute a command either in GL, like a draw command, or if we execute a kernel in CL, the runtime, the driver, will take care of moving a cached copy of that data to the device that the command is executed on. So in this case, when I created that mem_object in OpenCL, it's not as if I was allocating memory on a device.

I was really allocating memory inside of this driver storage pool, and then if I execute a command using a device, that memory is cached, in this case in Device VRAM, on the GPU. Now, if I execute a command in either API on a different GPU, the runtime will take care of moving that data back to the host and then to the other device.

Now, OpenCL is a little bit different than OpenGL in this respect, in that unlike the GPU, which has; each GPU has its own piece of device VRAM. In OpenCL, the CPU device shares VRAM with driver storage. So, when I execute a command on the CPU in OpenCL, this copy operation doesn't occur. The CL kernel is able to use the same backing store as the runtime.

Now, the operations we've looked at so far, and the example in the demo, involve vertex buffer objects or pixel buffer objects. But, there are three other functions we can use to create shared data between OpenGL and OpenCL. The first two functions involve textures; 2 and 3D textures. And, the third object allows us to manipulate render buffers, which was the structure we used for that post-process example that we saw earlier.

After these structures are created in OpenGL and then imported into OpenCL, we can do a lot of things with them. In either API we can modify their contents, execute commands that use them; but, one thing we can't do is we can't modify their properties. So, after an image is imported from OpenGL into OpenCL, we can't change the width and height of the image. We can't change other properties of the image. We'd have to create a new copy in GL and then move it back into CL in order to make those sorts of changes.

OK, now that we've created the objects and imported it into OpenCL, as we're launching commands and executing commands in either API, we have to use a specific type of semantic, called Flush and Acquire, in order to coordinate access between the two APIs. Let's take a look at the standard queueing system that's used by OpenGL by itself. As I execute commands, they're enqueued in a command queue on the host, and at a certain point, those commands are flushed to the device and executed.

Now, the order of commands as I call functions is sort of maintained by that command queue, and those commands will be executed in the same order when they get to the device. And, since I have a single command queue, there's no chance of commands being executed in an order other than the one I've specified.

However, if I add the OpenCL command queue alongside that OpenGL command queue, and then execute a bunch of OpenGL commands in my thread, and then a bunch of OpenCL commands in my thread, without any explicit synchronization, I don't have any control over the order that those commands get submitted to the device.

So, it's very possible that even though I sent the two OpenGL functions first, they might be submitted to the device by the runtime in an interleaved order. Now, if there are data dependencies between OpenGL and OpenCL such that I had to execute all of those GL operations because I was producing that render buffer in OpenGL before I consumed it in OpenCL, this type of unsynchronized execution would cause a problem.

Therefore, before we move data, or before we move commands between those two APIs and execute work on one side or another, we have to make sure to flush the GL side and then that flush operation sends those commands out to the device, and then acquire those shared objects on the CL side, then execute our CL commands.

Once we then call clEnqueueRelease, our CL commands will be flushed to the device after the GL commands. This explicit type of Flush and Acquire semantic ensures that those GL commands are in progress on the device before the OpenCL commands have a chance to be submitted to the device, and the order between the two APIs will be maintained.

OK, now that we've gone over how to create the ShareGroup, move the GLShareGroup into OpenCL, then create data objects and import those data objects, and then safely execute commands, the very last step is how to safely clean up this system. The key here is that we always release objects in the opposite order that we created them. We always release the objects in OpenCL, and then destroy them in OpenGL. This ensures that in OpenGL, the OpenGL driver won't take objects out from underneath the OpenCL implementation.

Now, as you might be aware, OpenGL automatically takes care of retaining objects for you, so if you pass a kernel into the runtime, or a data object into the runtime, the runtime will make sure that the reference count of that object reflects that the runtime is holding onto a pointer for it. So, it's necessary to make sure that after you've enqueued commands that use memory objects, those commands have been executed and have been completed before OpenGL has an opportunity or might accidentally delete or destroy an object.

So, the objects have to be completely released by OpenCL before they can be destroyed by OpenGL. And, releasing a mem_object is simple. Essentially, there's one function, regardless of whether the mem_object was created on the GL side, whether it's a buffer or an image. We simply call clReleaseMemObject. OK, now I'd like to show you an example; a live demo of the example I showed earlier.

This is the case where a vertex buffer object is created in OpenCL and shared with OpenGL, and then a GLFrame is created, or a GLFBO is created and then shared with OpenCL to perform some post-processing. In this example, In this example OpenCL is rendering, OpenCL is computing the physics interaction between the spheres that are bouncing around the screen, and then OpenGL is rendering the refraction and the reflection on each individual sphere.

As OpenGL renders that effect, it also produces a buffer which contains the surface normal and position on the surface of various fragments for the spheres. That buffer is passed back to OpenCL using a shared render buffer. OpenCL then provides or performs some photon tracing to compute the caustic effect that you see towards the bottom of the screen. This is a simple example.

The physics that's computed in OpenCL isn't particularly sophisticated, but this allows us to perform all of the computation on the GPU, instead of having to coordinate between the CPU and the GPU for each frame. So, both the physical simulation and the rendering of the spheres can happen on the GPU, and then the photon tracing and the rendering of the caustic highlight can also occur on the GPU.

OK, so the three steps for that demo were simply to update vertex positions, perform the photon trace, and then render the scene. And, by using the sharing API, it was very easy to perform all of those operations on a single device. If we had some type of application where we wanted to perform, say updating the vertex positions or rendering the photons on the CPU or on another device, the OpenCL runtime would have allowed us to automatically move the data back and forth between those devices. And so, even if we're not running applications that do all of their work on the GPU, it's still possible to use OpenCL and the sharing mechanism to handle moving data between the two devices.

OK, so five easy steps to shared data between OpenGL and OpenCL. The first step is to make sure that we select our pixel format and the devices that we're going to use for OpenCL, using that CGLPixelFormat function, and using that pixel format to create our CGLContext and ShareGroup.

We passed that ShareGroup to CLCreateContext in our second step, to produce and initialize a CLContext containing those devices. Then we create objects in GL, import them into CL. We use a GLFlush and CLAcquire pattern to handle coordination of commands between the two APIs, and then lastly, when we're done, we release in CL before destroying in GL.

Now, this concludes the CL session. For more information, you should contact Allan Schaffer, who's our Evangelist, and of course, look at the Apple developer forums. There's a CL Dev forum and also a GL Dev forum that are great ways of getting in touch. There are a number of other sessions this week that will address OpenCL and OpenGL.

Immediately following this session in this room, we'll hear from a number of vendors that will describe how to maximize OpenCL performance for different devices. Tomorrow there's a session on OpenGL for the Mac, and then later in the day tomorrow is a session that will describe multi-GPU programming, both with OpenGL and OpenCL.