Good morning everyone. My name is Karol Gasinski, and I am a member of GPU Software Architecture Team at Apple. We will start this session with a brief summary of what is new in [inaudible], in terms of VR adoption. Then, we will take a deep dive into a new Metal 2 features, designs specifically for the VR this year. And finally, we will end this session with advanced techniques for developing VR applications.

Recently, we introduced new iMac and iMac Pro that have great GPUs on board; iMac is now equipped with [inaudible] based GPUs and have up to 80 gigabytes of video memory on board. While iMac Pro are equipped with even more advanced and even bigger based GPUs, with up to 16 gigabytes of video memory.

That's a lot of power that is now in your hands. But we are not limiting our service to iMac on their own. With recent announcement of extended GPU support, you can now turn any Mac into powerful workstation that gives you more than 10 [inaudible] of processing power. And that's not all.

Today we are introducing plug and play support for HTC Vive head mounted display. It has two panels with 1,440 by 1,600 and 650 pixels per inch. That's 78% increase in the resolution, and 57% increase in pixel density compared to Vive. And with support for better panels comes support for its new dual-camera front-facing system, so developers will now be able to use those cameras to experiment with pass-through video on Mac. And together with Vive Pro support comes improved tracking system.

So, you might be wondering how you can start developing VR application on Mac OS. Both HTC Vive and Vive Pro work in conjunction with Valve SteamVR runtime, that provides a number of services including VR compositor. Valve is also making open VR framework that is available on Mac OS, so that you can do the map that works with SteamVR. We've been working very closely with both Valve and HTC to make sure that Vive Pro is supported in SteamVR runtime on Mac OS.

So, now let's see how new Metal features that we're introducing can develop a [inaudible] Mac OS Mojave can be used to fiber-optimize your VR application. As a quick refresher, let's review current interaction between application and VR compositor. Application will start by rendering image for left and right eye into 30 multi-sample textures.

Then it will resolve those images into iOS surface back textures that can be further passed to VR compositor. VR compositor will perform final processing step that will include [inaudible] distortion correction, chromatic aberration, and order operations. We can call it in short warp. Once the final image is produced, it can be sent to the headset for presentment. It is a lot of work here that is happening twice, so let's see if you can do something about that.

See, now with VR application it wants to benefit from multi-sample [inaudible], it needed to use the dedicated textures per i, or single shared one, for both. But none of those layouts is perfect. The dedicated textures require separate draw calls and passes, as we just saw. While straight textures enable rendering of both eyes in single rendered and results pass, they are problematic when it comes to post-processing the effects.

[Inaudible] textures have all the benefits of both dedicated and shared layouts, but currently they couldn't be used with MSAA. This was forcing app developers to use different rendering part-time layouts, based on the fact if they wanted to use MSAA or not. Or use different tricks to work around it. So let's see how we can optimize this rendering.

Today, we introduce new texture types to the multi-sample already textured. This texture type has all the benefits of previously mentioned types without any of the drawbacks. Thanks to that it is now possible to separate from each other rendering space, which simplifies the post-processing effects, views count, so that application can fall back easily to monoscope rendering and control over anti-aliasing mode. As a result, application can now have single rendering files that can be easily adopted to any situation, and most important, that can be rendered with single draw and render pass in each case.

So here we see code snippet for creation of mentioned 2D multi sample [inaudible] texture. We set up sample count to 4, as it's an optimal tradeoff between quality and performance, and at the same time, we set up our other length to 2 as we want to store each image for each I in separate slice. So let's see how our pipeline will change. We can now replace those 2D multi-sample textures with single 2D multi-sample [inaudible] one.

So now application can render both I in single render pass and if it's using instancing, it can even do that in single draw code. So that already looks great, but we still need to resolve those 2D multi-sample array texture slices into separate iOS [inaudible] faces before we pass them to compositor.

So let's focus on our way, application shares textures with compositor. So now, for sharing textures, we use IOSurfaces. They are sharing textures between different process spaces and different GPUs, that we've got [inaudible] comes a price. IOSurfaces can be only used to share simple 2D textures, so if you have a multi-sampled one, storing [inaudible] or having [inaudible], they couldn't be shared. That's why today we introduce shareable Metal textures that allow your applications to share any type of Metal texture between process spaces, as long as these textures stay in scope of single GPU.

This file features [inaudible] advanced view of these cases. For example, sharing depth of your scene with VR compositor. But, of course, it's not limited just to that. Now, let's look how those textures can be created. Because shareable textures allows us now to pass complex textures between processes, we will create 2D array texture that we will pass to VR compositor. As you can see, to do that, we use new methods, new shared texture with this creator. And while doing that, you need to remember to use private storage mode, as this texture can be only accessed by the GPU on which it was created.

Now, we see a code snippet showing us how our VR application would send IOSurface to VR compositor in the past. We will now go through this code snippet, and see what changes needs to be applied to switch from using IOSurfaces to shared Metal textures. So we don't need those two IOSurfaces anymore, and those two textures that were backed by them can now be replaced with single shareable Metal texture that is over 2D array type.

We will then assign this texture to both texture descriptors from open VRSDK, and change its type from IOSurface to Metal. After doing these few changes, we can submit image for the left and right I to the compositor. Compositor will now know that we've passed shared Metal texture with advanced layout, instead of IOSurface, and if we check, if its type is 2D array or 2D multi-sampling array.

If it is, then compositor will automatically assume that image for the left i is stored in slice 0, and image for right i is stored in slice 1. So your application doesn't need to do anything more about that. And of course, sharing Metal textures between application and compositor is not the only use case for shareable Metal textures. So here we have simple example of how you can pass Metal texture between any two processes.

So we start exactly in the same way. We create our shareable Metal texture, but now from this texture, we create special shared texture handle that can be passed between process spaces using cross-process communication connection. Once this handle is passed to other process, it can be used to recreate texture object. But while doing that, you need to remember to recreate your texture object on exactly the same device as it was originally created in other process space, as this texture cannot leave scope of GPU.

So now let's get back to our pipeline and see what will change. Application can now replace those separate IOSurfaces with one 2D array texture, storing the image for both i's. This allows further optimization as original 2D multi-sample array texture can be now resolved in one pass as well to just create it shareable through the array texture.

But that's not everything. Let's look at the compositor. Once we have simplified rendering parts on application site, there is nothing preventing compositor from benefiting from those new features as well. So compositor can now use those incoming 2D array textures and perform work for both i's in single render pass as well.

And as you can see, we've just simplified the whole pipeline. So let's do recap of what we've just learned. We've just described two new Metal features. Shareable Metal textures, and 2D multi-sample array texture type. And the way they can be used to further optimize your rendering pipeline. Both features will be soon supported in upcoming SteamVR runtime updates.

So now, let's focus on techniques that will allow your application to maximize its CPU and GPU utilization. We will divide this section into two subsections-- Advanced frame pacing and a reducing free rate. We will start with frame pacing. And in this section, we will analyze application frame pacing and how it can be optimized for VR. So let's start with simple, single-threaded application that is executing everything in serial monitoring. Such application will start its frame by calling WaitGet pauses, to receive pauses, and synchronize its execution to the frame rate of the headset.

Both Vive and Vive Pro has refresh rate of 90 frames per second, which means the application has only 11.1 milliseconds to process the whole frame. For comparison, blink of an eye takes about 300 milliseconds. So in this time, the application should render 50 frames. So once our application receives pauses from WaitGet pauses, it can start simulation of your trial [inaudible]. When this simulation is complete, and state of all objects is known, application can continue with encoding command buffer that will be then sent to GPU for execution.

Once GPU is done, an image for both i's is rendered, it can be sent to VR compositor for final post-processing, as we talked about a few slides before. After that, frames scanned out from memory to [inaudible] in the headset. This transfer takes additional frame as all pixels need to be updated before image can be presented. Once all pixels are updated, [inaudible] and user can see a frame.

So as you can see from the moment the application receives pauses, to the moment image is really projected, it takes about 25 milliseconds. That is why application receives pauses that are already predicted into the future, to the moment when photons will be emitted, so that the rendered image is matching the user pause. And this cascade of events overlapping with previous and next frame is creating our frame basing diagram. As you can see, in case of the single-threaded application, GPU is idle most of the time. So let's see if we can do anything about that.

We are now switching to multi-threaded application, which separates simulation of its visual environment from encoding operations to the GPU. Encoding of those operations will now happen on separate rendering threads. Because we've separated simulation from encoding, simulation for our frame can happen in parallel to previous frame encoding of GPU operations.

This means that encoding is now shifted area in time, and starts immediately after we receive predicted pauses. This means that your application will now have more time to encode the GPU [inaudible] and GPU will have more time to process it. So, as a result, your application can have better visualize.

But there is one trick. Because simulation is now happening one frame in advance, it requires separate set of predicted pauses. This set is predicted 56 milliseconds into the future so that it will match the set predicted for rendering thread and both will match the moment when photons are emitted. This diagram already looks good from CPU side, as we can see application is nicely distributing its work [inaudible] CPU course, but let's focus on GPU.

As you can see, now our example application is encoding all these GPU [inaudible] for the whole frame into a single common buffer, so unless this common buffer is complete, GPU is waiting idle. But it's important to notice that encoding of GPU operations on a CPU takes much less time than processing of these operations on the GPU. So we can benefit from this fact, and split our encoding operation into a few common buffers while a few common buffer will be encoded very fast, with just few operations, and submitted to GPU as fast as possible.

This way, now our encoding is processing in parallel to GPU already processing our frame, and as you can see, we've just extended the time when GPU is doing its work, and as a result, further increase amount of work that you can submit in a frame. Now, let's get back to our diagram, and see how it all looks together.

So as you can see, now both CPU and GPU are fully utilized. So [inaudible] application is already very good example of your application, but there are still few things we can do. If you will notice, rendering thread is still waiting with encoding of any type of GPU work before it will receive predicted pauses. But not all [inaudible] in the frame requires those pauses. So let's analyze in more detail to pick our frame workloads.

Here, you can see a list of workloads that may be executed in each frame. Part of them happen in screen space or require general knowledge about pause for which frame is rendered. We call such workloads pause-dependent ones. At the same time, there are workloads that are generic and can be executed without knowledge about pauses immediately. We call those workloads pause independent ones. So currently, our application was waiting for pauses to encode any type of work to GPU. But if we split those workloads in half, we can encode pause independent workloads immediately and then wait for pauses to continue with encoding pause-dependent ones.

In this slide, we've already separated pause independent workloads from pause dependent ones. Pause independent workloads is now encoded in [inaudible] common buffer, and is marked with a little bit darker shade than pause-dependent workload following it. Because pause-independent workload can be encoded immediately, we will do exactly that. We will encode it as soon as the previous frame workload is encoded.

This gives CPU more time to encode the GPU work, and what is even more important, it ensures us that this GPU work is already waiting for being executed on GPU so there will be exactly no idle time on GPU. As soon as previous frame is finished, GPU can start with the next one.

The last subsection is a multi-GPU workload distribution. We can scale our workload across multiple GPUs. Current Mac Book Pro has two GPU on board, and while they have different performance characteristics, there is nothing preventing us from using them. Similarly, if each GPU is connected, application can use it for rendering to the headset while using Mac's primary GPU to offload some work.

So we've just separated pause-independent work and moved it to a secondary GPU. We could do that because it was already encoded much earlier in our frame, and now this pause-independent workload is executing in parallel to pause-dependent workload of previous frame. As a result, we further increased the amount of GPU time that you had for your frame. But, by splitting this work into multiple GPUs, we now get to the point where we need a way to synchronize those workloads with each other.

So today we introduce new synchronization parameters to deal exactly with such situation. MTL Events can now be used to synchronize GPU work in scope of single GPU across different Metal cues and MTL Shared Events extends this functionality by allowing it to synchronize workloads across different GPUs and even across different processes.

So here we will go through the simple code example. We have our Mac, with attached eGPU through Thunderbolt 3 connection. This eGPU will be our primary GPU driving the headset, so we can use GPU that is already in our Mac as secondary supporting GPU. And we will use shared event to synchronize workloads of both GPUs.

Event initial value is zero, so it's important to start synchronization counter from 1. That's because when we would wait on just initialized event, its counter of zero will cause it to return immediately, so there would be no synchronization. So our rendering thread now starts encoding work for our supporting GPU immediately.

It will encode pause-independent work that will happen on our supporting GPU course, and once this work is complete, its results will be stored in locker memory. That's why we follow with encoding brief operation that will transfer those results to system memory that is visible by both GPUs. And once this transfer is complete, our supporting GPU can safely signal our shared event. This signal will tell eGPU that now it's safe to take those results.

So our rendering thread committed this [inaudible] common buffer, and supporting GPU is already processing its work. At the same time, we can start encoding command buffer for a primary GPU that is driving the headset. In this command buffer, we will start by waiting for our shared event to be sure that the data is in system memory, and once it's there, and the shared event is signaled, we can perform a brief operation that will transfer this data through Thunderbolt 3 connection, back to our [inaudible] GPU and once this transfer is complete, it's safe to perform pause-dependent work, so a second command buffer will signal lockout event to let pause-dependent work know that it can start executing. After encoding and submitting those two command buffers, rendering thread can continue as usual, with waiting for pauses, and later encoding pause-dependent work.

So now we have a mechanism to synchronize different workloads between different GPUs. But as you can see, our secondary GPU is still a little bit idle. That's because in this example we decided to push through it, pause dependent workloads that have dependency with pause dependent ones. Excuse me.

But of course there are types of workloads that have no dependencies, and they can happen at lower frequencies, the frequency of the headset. One example of such workloads can be, for example, simulation of physically based accurate [inaudible] or anything else that requires a lot of time to be updated.

Such workload can happen in the background, completely asynchronously from rendering frames, and each time it's ready, its results will be sent to primary GPU. It's marked here with gray color to indicate that it's not related to any particular frame. So, of course there are different GPUs with different performance characteristics, and they will have different bandwidth connections.

And your application will have different workloads in a single frame with different relations between them. So you will need to design a way to distribute this workload on your own, but saying all that, it's important to start thinking about this GPU workload distribution, as multi-GPU configuration are becoming common on Apple platforms.

So let's summarize everything that we've learned in this section. We showed multi-thread application to take full benefit of all CPU codes. And split your command buffers, to ensure that GPU is not idle. When doing that, if possible, try to separate pause-independent from pause-dependent workloads, to be able to encode this work as soon as possible, and even further, splitting workloads by frequency of update so if your application will execute on multi-GPU configuration, you can easily distribute it across those GPUs. And while doing that, ensure that you drive each GPU with separate rendering threads to ensure that they all execute asynchronously. Now, you switch to reducing fill rate.

Vive Pro introduces new challenges for VR application developers. To better understand scale of the problem, we will compare different medium fill rates. So, for example, application rendering in default scaling rate to Vive headset, produces 436 megapixels per second. And most advanced [inaudible] against that [inaudible] HD TVs have fill rate of 475 megapixels per second. Those numbers are already so big that game developers use different tweaks to reduce this fill rate. Now, let's see how Vive Pro compares to those numbers.

Vive Pro has a normal fill rate of 775 megapixels per second, and if you add to that four times multi-sampling [inaudible] or bigger scaling rate, this number will grow even more. That is why reducing fill rate is so important. There are multiple techniques there and new techniques are made every day. So I encourage you to try them all, but today we will focus only on a few still as they are the simplest to implement and bring nice performance gains. So we will start with clipping invisible pixels.

Here, you can see image rendered for left eye. But due to the nature of the lens work, about 20% of those pixels are lost after compositor performs its distortion correction. So on the right, you can see image that will be displayed on a panel in a headset before it goes through the lens.

So, the simplest way to reduce our fill rate is to prevent our application from rendering those pixels that won't be visible anyway, and you can do that easily by using SteamVR Stencil Mask. So we've just saved 20% of our fill rate by applying this simple mask, and reduce our Vive Pro fill rate to 620 megapixels.

Now, we will analyze implication of this lens distortion correction in more detail. We will divide our field of view into nine sections. Central section has field of view of 80 degrees horizontally by 80 degrees vertically, and we have surrounding sections on the edges and corners. We've color tinted them to better visualize the contribution to final image.

So as you can see, corners are almost completely invisible and edges have matched less contribution to the image than in the original one. In fact, if you see this image in the headset, you wouldn't be able to look directly at the red sections. The only way to see them would be with your peripheral vision.

So this gives us great hint. We can render those edge and corner sections and a reduced fill rate, as they are mostly invisible anyway. We render the central section as we did before. But then we will render vertical edges with half of the width and horizontal sections with half of the height.

And finally, we will render corner edges at one-fourth of the resolution. Once our expensive rendering pass is complete, we will perform cheap upscaling pass that will stretch those regions back to the resolution at which they need to be submitted to compositor. So you are wondering how much we've gained by doing that.

In case of 80 by 80 degree central region, we reduced our fill rate all the way down to 491 megapixels per second. But you remember that we just talked about clipping invisible pixels, so let's combine those two techniques together. By clipping pixels combined with multi-resolution shading, you can reduce your fill rate even further to 456 megapixels per second, and that is not a random number.

In fact, that's a default fill rate of Vive headset, so by just using those two optimization techniques, your application can render to Vive Pro with much higher resolution using exactly the same GPU as it did when rendering to Vive headset. Of course, you can use those techniques when rendering to Vive as well, which will allow you to bring visualize of your application even further and make it prettier.

There is one caveat here. Multi-resolution shading requires few render passes, so it will increase your workload on geometric [inaudible], but you can easily mitigate that by just reducing your central vision by a few degrees. Here, by just reducing our central vision by 10 degrees, we've reduced fill rate all the way to 382 megapixels per second. And if your geometry workload is really high, you can go further, and experiment with lower fill rate, lower regions, that will reduce fill rate even more.

In case of 55 by 55 degrees central region, 80% of your [inaudible] eye movement will be still inside this region, but we've reduced our fill rate by more than half, to 360 megapixels per second. So of course there are different ways to implement multi-resolution shading. And you will get different performance gains from that. So I encourage you to experiment with this technique and try what will work for you best. So let's summarize everything that we've learned during this session.

We've just announced plug and play support for Vive Pro Headsets, and introduced new Metal 2 features that allow you now to develop even more advanced VR applications. And I encourage you to take advantage of multi-GPU configurations, as they are becoming common on other platforms. You can learn more about this session from this link, and I would like to invite all of you to meet with me and my colleagues during Metal 4 VR Lab, that will take place today at 12:00 p.m. in Technology Lab 6. Thank you very much.