Good morning. Welcome to the introduction to the Shake SDK session. My name is Angus Taggart, and I'm an engineer with the Shake team. I do some development and I also do a lot of work with the SDK. Our objectives for this morning are really to provide you with a general introduction to Shake, what it does, our customer base, you know, who's using Shake, some of the people who developed for Shake, and then I'm going to spend some time talking about Shake's architecture, some important concepts. And we want to take a high-level overview of Shake's architecture, and then we're going to move into some of the code building blocks.

And Shake is a very complex application. It's a very sophisticated piece of software that solves a lot of very interesting problems for our customer. Hang on, for our customers. So let me just get started. Let's introduce Shake. First of all, I want to see a show of hands. How many people here are familiar with the Shake product and what it does? Okay, fantastic. That's great. I was expecting to see only a couple. So we're going to do a quick intro to Shake.

We're going to talk about the customer base, third-party developer community, and the SDK package, just to get started. First of all, as it sounds like most of you know, Shake is Apple's high-end digital compositing and visual effects software solution. New in Shake 4, an exciting new feature is the ability to do digital compositing in a 3D environment.

A lot of our customers are more and more working or implementing workflows that require 3D solutions for compositing, and Shake 4 has a very exciting new capability to support that. It's got a lot of underlying capabilities to do image processing, effects. It's just a very powerful application. One of the things I'd like to do really quickly is introduce the kinds of problems that our customers are solving.

We're fortunate enough to get an effects build-up clip from our friends at Weta Digital. They're based in New Zealand, and they did, obviously, all the Lord of the Rings work. In this shot, what you'll see is they take the original footage, and then they do a huge amount to it to create the final shot.

They paint in shadows using mattes. They do color correction. Now they need to put in a background so that they create a matte for the background. At this point, we're going to start introducing 3D digital elements. We're going to have horses, which are 3D digital elements. There's a big dragon that swoops in that's a 3D digital element.

This is really the core of what Shake does. It takes a number of different sources of imagery from live footage to CG elements, and it combines them all to create a seamless final image, what you see on the scene. That's the core of what Shake does. It's a very powerful product.

One of the things I wanted to go over is the Shake customer base. As plug-in developers, you have to make an economic decision. You're going to put in so much work to build your plug-ins, and you want to know, who am I selling to? How big is the customer base? First and foremost, Shake is entrenched at the high-end film visual FX market.

I've listed, basically, in the United States, in Europe, and also in the South Pacific, Shake is used by the top-tier high-end visual FX companies. It would be surprising if you went to see one of the blockbuster movies this summer, and it would be very surprising if Shake hadn't been used to help produce some of the visual FX for those movies, Batman Begins, that sort of thing.

It's becoming a required courseware at a number of film schools. I think a very exciting story for Shake, and this is something that's really happened over the last two or three years as Shake has been brought into the Apple family, is that with the port to OS X and very aggressive pricing and marketing, we're seeing a rapid adoption of Shake in what you would consider to be the broadcast video market.

That is a very large customer base. It's a very large customer base that certainly as a plug-in developer would mean your return on your investment would be greatly enhanced. Since Shake has been around for, I guess, since the late '90s, mid-late '90s, we actually have a fairly established third-party developer community in place, and we're very fortunate to have support from some of the top motion graphics plug-in developers in the world. We're very fortunate to have support from some of the top motion graphics plug-in developers in the world.

We're very fortunate to have support from some of the top motion graphics plug-in developers in the world. And the reason that we have this kind of support is that Shake is being used in some of the most exciting projects, post-production work going on now, and so people naturally choose Shake as a way to get their technology out to high-end users. And the reason that we have this kind of support is that Shake is being used in some of the most exciting projects, post-production work going on now, and so people naturally choose Shake as a way to get their technology out to high-end users.

Okay, one of the things that it's really worthwhile doing is spending a bit of time talking about Shake's architecture. One of the great advantages of Shake is that the SDK exposes the same internals, basically the same base classes that the internal development engineering team uses. And so as an SDK developer, you're going to be able to get very deep into the application. You're going to be able to very tightly merge your technology in with Shake, and that's very exciting. But to do that, you need a conceptual base.

You need to be able to sit there and go, okay, I understand what I'm looking at. I understand why Shake's doing that. And so that's why I like to spend a little bit of time talking about Shake's architecture and just reviewing some of the key things that Shake does. So there's essentially three main points we're going to look at. We're going to look at dependency graph.

Shake is a dependency graph-based application. The rendering engine, we're going to look at how the rendering engine works, at least at a high level. And we're also going to look at the workspace. You know, everything exists inside of a 2D coordinate system, but we want to know a little bit more about how Shake tracks things inside of that 2D coordinate system.

So first of all, I guess I'll ask, how many people here have worked on an application or developed an application that has a dependency graph architecture? Okay, so there's a couple of you. Good, I'm glad I've got these slides then. So first and foremost, let's just take a look.

If you open up Shake and you start putting together a composition, one of the things that you'll see is a very simple processing tree. So we've got some input image data, we've got a color wheel, we've got an input file, and you'll see that the image data flows down, and that as it flows down, it goes through what we call nodes.

And the two key conceptual underpinnings of a dependency graph are nodes and plugs. A node is essentially an entity where data processing occurs. So it's within nodes that actual computation occurs. If you've got an image processing algorithm, you're going to be implementing it inside of a node. Now, plugs are the entity within Shake.

They're basically containers for data, and they allow the movement of data through the dependency graph. So if we take a quick look at that image in Shake, what you'll see--let me see if my laser pointer skills. So what you'll see is we've got some image data that's flowing down through the tree, but essentially that data is transferred in and out of nodes through entities called plugs. So if we take a look at--in the right, I've got a very simple blown-up diagram, and what it does is it basically shows you a couple of input plugs where data moves into the node. The node then processes that data.

Once it's done processing the data, it sort of places the answer or whatever value or values that it's computed in an output plug. So that's the basis, the basics of a dependency graph. Let's talk a little bit about nodes. So nodes are connected via plugs. That's the way we move data. Key things--plugs must be of the same data types.

So if you've got a floating point value, if you want to connect it to another plug, that other plug has to be a floating point value. And the--one of the key efficiencies in a dependency graph is you might have 30 plugs all connected, but they all share a reference to a single data item.

So that's something that makes the dependency graph very efficient. There's not duplication of data and also the state of the data. Basically, all the plugs will inherit or all the connected plugs will inherit the same state of the data. Another key concept with nodes is the fact that they're hierarchical.

And this is very important because you can imagine that Shake actually has a lot of very simple data processing nodes that do things like maybe a bytes conversion, you know, from 8-bit to 16-bit. You can imagine that there's nodes in Shake that do things like, you know, crop data, all kinds of things. Well, you want to be able to reuse that functionality. You don't want to have to rewrite it.

And, so, one of the things that you're able to do is you're able to take advantage of the hierarchical nature of the dependency graph and you're able to create what we call composite nodes or macro nodes. And, so, you can rapidly create your own customized image processing node, but at the same time reuse some of the existing functionality that Shake's already implemented for you. And it's just a question of wiring up your own internal node tree to manage that processing. So, that's an important concept.

So let's look at plugs. Some points with plugs. We know that they're containers for data, but an important point with them is that they must always be owned by a node. There's no such thing as just a plug dangling outside somewhere or in some sort of outside of a node. We basically always have to have an owner node for that plug.

Plugs have polarity, meaning certain plugs are dedicated to bringing data into the node, and other plugs are dedicated for taking data out of the node. And when you define a plug, you're going to say right up front you're an input plug or you're an output plug. And finally, the plugs in Shake actually only represents five basic data types. We've got ins floats, double strings, and also a plug that can store a pointer.

So that's something that you might want to look at. And then there's the plug that you might find kind of surprising, but that's all we need. And the reason for that is that plugs similar to nodes can have hierarchical relationships. And so one of the things that you can do is you can actually create fairly complex plugs that have a lot of children plugs. And basically it allows you to pass in a complex data structure. In this particular example, we've got a composite plug or a macro plug, and ten of its children are input plugs.

Six of its children are output plugs. And so it allows sort of an interesting movement of data in and out of the node. And the analogy I always like to make is that nodes that have hierarchical relationships are basically supporting more complicated processing. So when you see a node with a lot of children, it's basically performing more complex processing.

If you see a plug with a lot of children, what it is is it's more complex data structure. I always consider a plug with children to be like a C struct. Basically you've got a number of data members that are part of it, and it allows you to move complex data in and out of your node.

So where does the dependency part of the dependency graph architecture come from? It comes from the concept that your node produces values and places them in their output plug, but you need to register with Shake what dependencies your, what input plugs your output plug depends upon. So to produce a value, usually you can write out a simple equation like C equals A plus B, and then you can derive a very simple dependency relationship which says, hey, plug C is dependent upon both plugs A and plugs B.

And the reason that this is important is because Shake needs to know whether the value stored in your plug is up to date or not. Now let's say that you've got a plug that computes A plus B equals C, and it's computed at once, and so plug C is storing some value.

Well, when Shake goes back to your node to request the value of plug C, the first thing that it's going to do is it's going to look at plug A and plug B and see if the values have changed since the last time that it requested the value of plug C. And if plug A and plug B's values haven't changed, then guess what? Shake isn't going to ask you to do any processing at all. It's going to use the existing value that's stored in plug C.

And this is something that makes a dependency graph very, very efficient so that there's only recomputation when it's necessary. So if somebody does come in and change an input value such as, okay, I'm going to change the input value stored in plug B, Shake's dependency graph architecture tracks that, and it'll say, hey, guess what? You need to recompute C.

Another very important point that makes Shake's dependency graph efficient is that the Shake node engine or dependency graph engine is only going to call your plug-in for output values. I don't want to say it doesn't care about your input plugs, but at the end of the day, Shake is only going to ask your node to produce an output value.

And what that means is that somebody can go in and be fiddling with the value in plug B and plug A, but your node isn't necessarily going to be called at that point to compute a value. It's only going to be called when somebody else out there needs your value stored in output plug C.

So when somebody requests the value stored in plug C, at that point, Shake does its dependency graph logic, looks at the value stored in plugs A and B. If they've changed, then you're going to get called. And then you're going to have to do some computation. So both lazy evaluation and dependencies are very key aspects of dependency graphs that make them very efficient.

So let's take a look at SHAKE's rendering engine. First thing, when SHAKE came out, you know, back in the middle, late 90s, one of the things that people just loved about SHAKE was that it supported both 8, 16, and 32-bit image data. You know, what you'll see is it's 2005 now, and if you go to some of the computer graphics, trade shows, NAB, SIGGRAPH, somebody will come out and say, hey, we support HDR data now, or high dynamic range data.

Well, SHAKE's been supporting high dynamic range data by supporting 32-bit image data since the late 90s. That's no big deal for us. We've been able to manage that for a very long time. Resolution independence, that means that when you're working on a composition in SHAKE, you can bring in 4K images, you can bring in 2K images, you can bring in 640 by 480 images, and you can bring in all those images into the same composition. It's not like I'm going to sit down and I'm going to do a composition and this one's going to be a 720 by 486 composition, and everything coming in has to be cropped to that size.

And actually, if you take a look at the little graphics that I've got at the bottom, it just shows we've got a color wheel. That color wheel is 8-bit image data. We're reading in a--that's actually an OpenEXR file, and it's just a sample image, and that's 32-bit image data.

You can see that the sizes of the images are different, and all we're doing is a simple layering operation, and you'll see that SHAKE manages that without any problem. So it can put A on top of B without any problems whatsoever. And you can choose whether you want the bit depth promoted to the background image value or the foreground image value.

So that's a very powerful aspect of SHAKE's rendering engine. The reason I bring that up is, okay, you're a plug-in developer, and you've got an image processing algorithm that you want to get working inside of SHAKE. The thing that you need to do is you need to make sure that you have an 8-bit, a 16-bit, and a 32-bit implementation of your algorithm, meaning-- 'cause when you've got your pointers to the data, it's got to be a pointer to a data of a certain, you know, certain bit depth. And so most image processing nodes in SHAKE support all three of those--all three of those bit depths. So that's something to keep in the back of your mind when you're developing your algorithms.

Um... Okay, so that's sort of the first piece of information. The second piece of information is that how does Shake actually generate pixel values? Well, what it does is, I don't know if you're familiar with like a scanline renderer. A scanline renderer essentially just renders a line of pixels and, you know, just sequentially does one line after the next until it builds up an entire image. Shake uses what we call a hybrid block scanline and tile-based renderer.

And the reason why it's a block scanline is that Shake will basically process as many scanlines as can fit in the CPU's L2 cache. The reason is that, you know, the processing nodes are constantly updating the pixel data stored in memory, and if you've got that pixel data sitting in your CPU's L2 cache, the retrieval of that is very fast. It makes the Shake rendering engine very, very fast.

So that's an important thing. The other thing is that, so tile-based rendering. Tile-based rendering basically is what it sounds like. It takes an image and breaks it up into individual tiles and can work with individual tiles. And there are certain circumstances that Shake will internally convert to being a tile-based renderer.

And the final point on that is by rendering just a few scan lines at a time instead of working with full frame image buffers, it really reduces Shake's memory footprint. A lot of our customers will be working on composites that might have, like, 40 or 50 input files. These things look like a map of L.A.

I mean, you've got a huge number of input nodes. You've got, you know, tons of a lot of image processing operations. And so, it's very important that image processing nodes in Shake are able to work on just a few scan lines of data at a time so that the memory footprint doesn't become ridiculous.

So in tile-based mode, you can imagine that if, let's say you're a transformation operator or a warp operator where you've got to rotate an input image. Well, to do the rotation, you're actually going to need to retrieve pixel data from other regions of the image. And we're trying to just work on a few scan lines at a time, right? And so underneath the hood, what Shake will do is when you request essentially image data from outside of your scan line, Shake can actually break up the incoming image into tiles that it caches. And so it'll give you that image data. It'll keep the tile around. And then if like a subsequent pixel that's being processed down the scan line needs image data from one of those tiles that's been cached, it's right there and very fast.

So that's sort of the basics of the rendering. And one of the things that we always encourage our developers to do because sometimes developers will come from doing plug-ins from an application that they get the full frame image buffer. And so they're used to having all of the pixel data, all of the input image pixel data at once. And so sometimes developers need to refactor their algorithms or image processing algorithms so that they can actually work within a block scan line environment.

And a lot of times... It's very easy to do that. It's just a question of sometimes developers will have image processing libraries and those image processing libraries expect the full frame input image and they just need to do some refactoring to work on just a few scan lines at a time.

Okay, so within Shake, all the pixels live in a 2D coordinate space. And one of the things that we like to do right up front is, you know, where is the origin of this 2D coordinate system? And as this graphic illustrates, if you open up Shake, there is what we call the viewer window. And in the viewer window, you actually see the final, what the dimensions of the final image are going to look like. So I call that the image window.

And the 2D coordinate system inside of Shake is always anchored to the lower left-hand corner of the image window. So if you've got a 720 by 486 image, your coordinate system is going to be anchored to the lower left-hand corner of that. You know, you've got a 640 by, you know, 486, whatever it is, it's always going to be anchored to the lower left-hand corner of the image window. And even within a composite, let's say you've got a composite. The input image is 720 by 486.

But at some other point in the composite, you change the size of the image. Maybe you've done a resize to make it larger or smaller. The 2D coordinate system still stays anchored to that lower left-hand corner. Everything else sort of adjusts itself with respect to that. So that's the first piece of information.

The second piece of information, and this is sort of another one of the things that's a bit of a first in Shake, is the notion that the composition workspace is actually an infinite workspace. There aren't hard boundaries where image data can be lost in most circumstances. And so if you transform pixels or resize them, do a corner pin, whatever you're doing, even though that pixel data might fall outside of your image window, as you see, for instance, in this graphic down here, the color wheel has been transformed to the right. And even though some of its image data is outside of the image window, Shake is still tracking it.

Shake, under the hood, actually tracks pixel data. And so if you want to track pixel data, you can track pixel data everywhere in this 2D space. And a quick point that I want to cover, because it comes up in plug-in development, is there is a color to Shake's canvas, and it's important to track this color. It's called the background pixel. And in this case, you can see that the background pixel color is black, and it's almost always black. It rarely changes. But it is what we call the color of the canvas, the background pixel. And there's a couple of key bounding boxes.

And the first one is that Shake defines to enable you to track where pixel data is and where processing should occur. The first thing is the domain of definition, or we just call it DOD. And the DOD is a bounding box that actually bounds all valid pixel data.

And by valid pixel data, you say, oh, it's valid pixel data. It's a pixel value that is not equal to the background pixel value. So even though in this graphic, it's not equal to the background pixel value, it's a pixel value that is not equal to the background pixel value. And so in this graphic that I've got here, what you'll see is the DOD is represented by the ground or by the green rubber band box.

And actually, that green rubber band box extends off the screen to encompass. Let me use my laser pointer again. The green bounding box will actually extend off the screen to track where the edge of the valid pixel data for the color wheel is. And so that's the first thing that Shake tracks. The second thing.

The third thing that Shake tracks is what we call the region of interest. And the region of interest is a very powerful tool within Shake because essentially, you might have a 2K image coming in. But at the end of the day, the user might only be interested in a small 200 by 200 region inside of that 2K image.

The region of interest allows you to tell Shake or basically allows you to tell Shake, we only want to process these 200 by 200 pixels. I don't want to perform image processing on the full 2K plate. So the region of interest is a very important bounding box. In this case, the region of interest is actually-- well, we'll take a look at it really quickly. Next slide.

You're going to be using plugs to get data in and out of your node, and so you're going to be using the NRI plug class. You're rarely going to need to derive from NRI plug. As it turns out, we've got 26 example plug-ins in the SDK, and I don't think any of them actually end up coming up with a specialized plug that derives from NRI plug, so you'll just be using the NRI plug class as is.

And is everyone, I mean, if you're Objective C programmers or C++ programmers, you're probably comfortable with the idea of an interface, a functional interface. Is everyone pretty comfortable with that? So essentially a functional interface is a set of functions or a suite of functions that, you know, fulfill a certain contract or task.

And in Shake, we've got a set of functions in the NRI node base class that allow you to communicate with Shake's dependency graph engine, basically, and rendering engine. And I think you'll be surprised. There's actually not a lot of functions that you'll need to override. It'll be fairly straightforward. The first and most important one is the eval function. And your eval function is where you compute the value of output plugs.

Remember, with lazy evaluation, Shake isn't going to ask you for the value of your input plugs. It's not going to come to you and say, "Give me a value of your input plugs." Shake is only going to come to you and say, "I need this value for your output plug." And when it does that, it's going to call your eval member function.

It's going to pass in a pointer to the plug that it's interested in. And then essentially what you've got inside of your eval is something that says, "Ah, if you're interested in the value of this output plug, then I need to do this set of computations." You do those computations. And you update the value in the output plug. A second one, which is one that ends up being used quite a bit, is the notify. And the reason for this is that the concept of lazy evaluation is you're only going to be asked for your output plugs.

But sometimes you want to get notified when one of your input plug values changes. You know, a user might do something and it might change the mode that your plug-in is going to work in. So you want some kind of notification. So there is a way around this. With an input plug, you can say to Shake, "Hey, call me when somebody changes the value of one of my input plugs." Because there's some stuff I need to do, reset up, whatever it is.

And so on an input plug, you can register that plug. Say, "I want to be notified when it changes." When its value changes, Shake is going to call you through the notify callback function. At that point, you can take any action you want to take. You can't actually do image processing. Or core rendering. But there's a lot of things that you can do in there.

Another one that developers might override is the begin exec. That essentially gets called right before render starts. So if you've got things you need to set up before you do your rendering, implement that in the begin exec. Because it's just called once right before frame renders. It's called on a per frame basis. So each frame that renders, your begin exec will be called. And then you can allocate some memory, set up some flags, whatever you want. whatever you need to do.

I think we covered that, didn't we? End exec, that's just sort of the other side of the render from the begin exec. So when the frame finishes rendering, if there's something you need to do to clean up or reset, override the end exec member function. One of the things that Shake sort of does for free for you is that it'll serialize in sort of a standardized format all of your input plug values for you. So Shake actually manages serialization of your plug-in's input values for you.

If you need to customize that, if there are some actions that you need to take, then you'll need to override the serialize member function. Most plug-ins don't have to, but it's something that you have the option of doing so that you can override the default serialization logic that's built into the NRI node base class. And a final one for computationally intensive image processing algorithms, I think there's a check interrupt function.

It's not something that you override or implement, but it's just there for you to call so that let's say you've got this big loop, maybe in the outer loop you can put in a check interrupt so that if the user hits escape, the check interrupt will tell you that the user's hit escape and then you can do the right thing and stop computing. Because if you've got something that takes 15, 20 seconds or a minute or more, you've certainly got to give the user the option of canceling the processing.

Wow, we got duplicate, I see it lit up when I pressed it, okay. So code building blocks, let's get into plugs cuz one of the things you're gonna need to do is you're gonna need to get values out of plugs cuz plugs are containers, they store the values. But you need to get those values out. So for each of the data types, you'll see that there's the associated give me the data as an int, as a float, as a double, as a pointer and as a string. And Shake has its own string class.

Most applications end up doing that for whatever reason, unless they use STL or something like that. And so if you need to get a const char*, it's really easy. The shake string class is nri_name. If you need to get your const char* pointer out of that, just call get_string on the nri_name object. So that's how you get data out of your plugs. And most of the time you're pulling data out of your input plugs cuz you're gonna need those input values to compute your output values.

Finally, there is only a single member function for setting a value in a plug. The logic inside of the NRI plug base class will manage, let's say you want to set a floating point value and you set it as a quoted string 4.0, there is logic inside of the NRI plug base class to do the necessary conversion based on the data type of the plug.

And a key thing in a dependency graph, the only way that we can move data in a dependency graph is by connecting plugs. And so one of the things that you'll use a lot is the connect call. As we said earlier, connections can only be made between plugs of the same data type. And there is sort of an order or a getter-setter relationship.

Basically, if I'm a plug and I say connect me to, I'm plug A, and I say connect me to plug B, that means that plug B is going to be setting my value. I'm not setting plug B's value. So if I'm plug A and I say connect me to plug B, that means I'm getting my value from plug B. And I'm sharing a number of other things, like the state of the plug.

Like in the dependency graph, we know that input plugs, if their value's changed, will have a state that we call marked as dirty, meaning that its input value is has changed since the last evaluation. Connected plugs all share that state. So if I've got 30 connected plugs, the plug at the end of that chain shares the same value and up-to-date state as the plug at the end of the chain.

And finally, how do you add a plug to your node? As we said, plugs must be owned by nodes, and so it makes sense that the NRI node base class actually has the function necessary to add a plug to itself. So when you add a plug, you set its name.

It's just a string, and it's actually the name that will show up in the UI when somebody's editing your plug-in. So it's usually good to choose a name that's intuitive to the user. So that's the first argument. The second argument is the type. Is it going to be an int, float, double, pointer, string? The next argument is the IO. As we discussed, a plug is either an input or an output plug, so you'll be setting that.

An internal plug is one that won't show up in the UI. It's just something that stores a value internally for you. So you can set a plug as being K internal. And one of the things that plug-in developers come back with is, how do I change the order of how my plugs appear in the UI? Well, your node actually stores a list of all the plugs you've added, and you can control that ordering. And within this add plug, you can actually say, hey, add this plug as the first one in the list, which means when the user's editing your plug-in, they'll actually see that plug first in the list.

Okay, so I think those are the basic code building blocks. There's obviously a lot more classes that are part of the Shake code base, but the key ones that you're going to be using over and over and over again is nri_node and nri_plug. Making the connection, setting values, that's the core things that the plugins in Shake do. So what I want to do is I want to revisit the C equals A plus B plugin example and actually Take a look at some source code for that.

So let's take a look at the class definition for this node. The first thing is that this one we're just going to derive directly from NRI node. We're not doing anything fancy in it. So we derive from NRI node, and we've got three plugs. We've got A, B, and C. A and B are input plugs. C is the output plug. And so we declare pointers to them in our header.

In the constructor in Shake, and this is an important point, is that in some object-oriented environments, it's encouraged that you don't do too much in the constructor. You know, that you basically have a very lightweight constructor, and then maybe you'll have an initialized member function where everything happens in the initialized. Well, Shake doesn't follow that.

Actually, it's just the opposite. In Shake, you do everything in terms of setup must happen in your constructor. So in terms of adding the plugs to your node, registering dependencies, making connections, anything that you're going to do, you're going to do in the constructor when you're building a plug-in in Shake. And in this, the only dependency graph interface function that we're going to override is eval, because the only thing that we're going to need to do is eval the value of C, our output plug. So that's what your header file looks like.

Fairly simple. And let's take a look at the constructor for this plug-in. As you can see, we're using the add plug member function. We've just decided to name the plugs capital A, capital B, capital C, and that's the way they'll appear in the UI. The first, all three plugs are integer plugs. And the first two, A and B, we tell Shake that they're input plugs. And the third plug, we tell Shake that it's an output plug. So that's the first thing.

Add all your plugs. Now that you've got your plugs added, remember this is a dependency graph, and Shake needs a way to know when the value that you're computing for plug C is up to date. So what you've got to do is add dependencies. So on plug C, we call add dependencies, and we tell Shake, by the way, the output value of C is dependent upon A and B being up to date. And here, I just wanted to give you a demonstration of actually setting a plug's value.

You'll see that for plug A, I just set it using a standard integer value. But you'll also see with plug B, I actually passed in a quoted string. And as we discussed before, the NRI node base class, or the NRI plug base class, knows how to handle the conversion necessary based on your plug type.

Okay, so let's go into our eval function. So this is how, this is the function that Shake's going to call when it wants the value of plug C. So all we're going to do is, we've defined pointers to our input plugs and our output plugs, and so what we're going to do is we're just going to see whether Shake is calling us for the value of plug C. And if it is, then you'll see that we're going to retrieve the values from plug A and plug B, so we've got our two integer values.

And then we're going to do a simple computation, A plus B, and we're going to take the resulting value of C, and we've got to actually set it into the plug. So remember, a plug is a container, and it holds a value, so we've actually got to set that value into the C plug. And one of the things that happens is that since Shake, actually Shake has quite a deep class hierarchy for doing image processing, it's always good form to call your super class or your parent class.

And because you might not evaluate that plug, it might actually be a plug that your parent class defines and has logic for evaluating, so it's always good form in Shake to call your super class with a plug. Now, in this case, you could put something in the evaluation section for plug C where it just says return zero, and that would be fine. Zero is the success code in Shake or the error code that indicates that nothing went wrong. And so, okay, we've covered that.

Cool, so one of the main things you're gonna do with Shake is you're going to develop image processing plug-ins. I would say 90% of the plug-ins that people develop for Shake are image processing plug-ins. The other ones, if I were to list them, you'll see people developing asset management, you know, plug-ins that enable Shake to work with an asset management system. Render queue submittal plug-ins are another variety of plug-in. But for the most part, our developers out there are developing image processing plug-ins.

And so that's what I would guess is, like, out of everyone there, do you guys do, if you do do plug-in development, do you do image processing plug-in development? Okay, what other kinds of plug-in development do you guys do? Render queue? Okay, we've got a plug-in type that supports that.

Any others? Okay. Anyhow, let's talk about, first of all, plug-in types. Here they are. Image processing plug-ins, on-screen controls, like for instance, is everyone familiar with an on-screen control? Basically, it's a reference geometry that sits right on top of the image, and what you can do is the user can actually interact with that reference geometry to make parameter changes. So instead of actually going over to a parameter editor and tweaking a slider, there's reference geometry drawn right on top of the image, and by interacting with that reference geometry, you're actually modifying values and changing the image.

One of the things that we've just introduced in Shake 4 is the ability to get spline data or shape data in and out of Shake. There's a number of applications out there that support rotoscoping, and that's the kind of thing, and we need to get shape data in and out.

As I mentioned, render queue interface. Also, Shake's widget toolkit is open to developers, so let's say you want to create custom dialogues and custom parameter widgets, you can do that. And we've also got custom file format readers and writers. So let's say you've got a custom file format, you need to get it into Shake, you can actually write something that's able to read that file format.

Okay, so let's just dig into image processing plug-ins and just focus in on this. One of the key things to understand when developing an image processing plug-in in Shake is the image plug. So if we take a look at this image up here, you'll see that we've generated a color wheel and that image data is flowing down through the tree. And you'll see that there's a connection made between the node that generates the color wheel and then an image processing node.

And the connection's indicated by something in Shake called noodles. And actually what a noodle is, is it's a connection, a plug connection. And it's the image plug. This is the thing that the image data and the information associated with the image data moves from one processing node in Shake to the next processing node. That is the primary connection between image processing nodes in Shake. It's the image plug.

And one of the things that makes the image plug a little bit tougher to sort of get a handle on is that it's a composite plug. It actually has 16 children plugs, and 10 of those children plugs carry data down the processing tree in Shake, and 6 of those plugs actually carry data up the processing tree. So there's some information, you know, let's say you've got a node, let me go back a little bit, let me see what the next slide looks like. Okay, let's go back.

So there's a certain amount of information in SHAKE that is defined by what we'll call a leaf node. It's a node right at the top of the processing tree. And that information will flow down. But there's also information that will be defined at the bottom of the processing tree that will flow up. Basically some information, I'll cover it, but you can imagine that in terms of what time do you want the image data.

Well, that's defined by something near the bottom of the tree. But in terms of what size the image data is, that's something defined by something near the top of the processing tree. So there's this flow of information both up and down the processing tree that as a plug-in you've got to track.

Okay. Now, this is a chart that I'm not gonna go into a huge amount of time going through each of the items, because when you develop, you know, when you start working with Shake Plugins, you'll get all this information. But these are the 10 plugs, or child plugs, that carry information down the processing tree.

So the information that flows down, the width and height of the image, that flows down. The bit depth of the image, that flows down the tree. The domain of definition, which is that bounding box that tells us where valid pixel data is, that flows down the tree. Let me see. I better click again.

Some of these The active color channels. That's another very important one. One of the things that Shake can do, or that plug-ins and Shake can do, is they can say, guess what, I've only got valid image data or luminance data. I've only got valid image data and luminance and alpha channels.

And that information propagates down the tree, and that means that plug-ins can limit their processing to only the active color channels. It might only have RGB and not alpha. As you guys are probably well aware, the kind of image data that can come in and the channels that are being processed can change quite a bit, and it's a big efficiency gain for a plug-in if it only processes the active color channels. The background pixel we discussed earlier, that's the color of the canvas.

You can attach some blind data and pass it down the tree, which is really cool, because let's say you're an input node and you've opened up a data file, and that data file or that image file has some metadata that you're going to want further down the tree. Well, you can take that metadata and you can store it all in the blind data plug, and that'll be passed down the tree, and then a node further down the tree can actually retrieve that data and use it.

The time range is something that flows down, because an input node usually knows that it's going to be passed down the tree. It knows, okay, I'm opening up 100 frames of data, so it can set the valid time range. And the key thing, the image buffer itself, we call it the O-buff, and that gets passed down the tree. So you'll actually retrieve that O-buff, which comes in from the node above you, and that's processed image data, meaning it's been processed by the node above you. You bring it in, you do your processing, and then you pass it to the node below you.

So the final one is the iBuff. And actually what happens in Shake in the processing tree is that the node at the bottom of the tree actually ends up allocating the image buffer. So because the node at the bottom of the tree knows the region of interest, it knows how much pixel data the user wants to look at, it allocates the buffer. That buffer then is passed all the way up to the top leaf node, and it gets to the top leaf node, and then the top leaf node starts filling in pixel values, passes it to the node below it.

He does his processing, and then it goes like that. So you can sort of get this image in your head or in your mind of an image buffer being allocated at the bottom of the tree, being passed all the way up to the top leaf node, and then the top leaf node fills in pixel values, and then it goes back down to the processing tree.

So the next two slides, if you're developing image processing plugins, the next two slides are worth just looking at because as the SDK engineer and support guy, I get more questions from this aspect of plugin development, image processing plugin development, than any other. And that is the fact that you've got an input image plugged to your node and you've got an output image plugged to your node. And tracking the polarity of the child plugs is key. Once you get this slide, these kind of plugins, very, very easy to do. So okay, let's look at the top of the node. At the top of the node, I've got image data coming into me.

Well, those ten downstream plugs are now inputs to my node. It's the data flowing into my node. And we know from dependency graph concepts that Shake-Only asks us for the output values from our node, the output plug values. So all the child plugs that are green in here, Shake isn't gonna ask you for those because they're inputs.

Those are values that you take into your node and you use those values to compute some value. So you can see in the input image plug, it's actually the data flowing up the tree that are the outputs to your node. So if you wanna change the ROI or the region of interest, then that's something that's gonna flow up the tree.

Shake will ask you for that. Let's go down and take a look at the output image plug. The polarity's flipped now cuz it's now an output plug to the node. And you can see that all of the downstream quantities are now outputs to the node. So anything associated with the width, the bytes, the DOD, the active color channels, Shake can potentially be asking you for that information cuz it's outputs to your node. And you'll see that the information flowing up the dependency graph or up the processing tree, the current time, the region of interest, etc., those are inputs to your node. So Shake won't ask you for those on that plug.

So how do you actually do image processing? Like how do you implement your image processing algorithm? Well, a lot of the time, let's face it, you're not going to change those values. Like for instance, a lot of times you're not going to change the width and height of the image. I mean, you're going to stick with the same width and height. So all you need to do is you need to take the output plugs.

So these are the output plugs to my node. I don't want to change them. I don't want to change the width, the height, the bit of the DOD or the active color channels. So I simply connect them to the inputs. And once I form that connection between the inputs, those input values, and the output values, Shake won't call me for the output values.

Because it'll look, it'll say, how do I get that value? And it realizes that it can just go up through a connection tree and retrieve that value somewhere else. So that means that you might have thought, oh, crap, I've got 16 plugs I might have to evaluate and do all the bookkeeping on in my node.

Well, actually you don't. Most of the time, you've only got a couple plugs that you're actually going to change the values of. So you leave the others connected so that that data just flows straight through your node. So you can see the data moving down the processing tree, flow straight through your node. And the data going up the processing tree flow straight through your node.

Two key ones you've got to cut, though. The first one is the O buff. You're going to be doing some kind of processing on the buffer itself. So you need to get that buffer of pixels from the node above you, implement your algorithm, modify their values, and take that pointer to the buffer and put it into the output O buff. So you're always going to do, oops.

So you're always going to do this. You're always going to snip the connection between the incoming obuf and the out obuf. The other one is this cache ID. Shake has a very sophisticated caching system so that it'll make up its mind to cache some of your image data. So that, let's say you've got a node that does some, like an FFT, some expensive computation to produce an image. Well, Shake can, under certain circumstances, cache your output image data.

And if it decides that all of your inputs haven't changed at all since the last time that it cached your image data, when your node is requested to process image data, it'll just use the cache value. The cache ID is actually, it's simply a string that's used to track the up-to-date state of your node. Basically, the cache ID is the thing that enables Shake to determine, has anything changed since I last cached your image data? If it hasn't, Shake will reuse image data that it's caching. You won't get called to process. and that's a big efficiency gain.

So, as you can imagine, Shake's a very deep, sophisticated application. There are a lot of base classes that we've defined that are dedicated to doing different kinds of image processing. Sort of the parent, the mother of all image processing base classes, you'll find out is we call it NRIFX. All it does is simply define an output image plug. And you can understand that that's the most basic thing that an image processing node can do, is generate some image data.

Not even worry about taking any image data in, just simply generate an image. And so the NRIFX base class, which is the mother of all image processing base classes, provides that functionality. And at this point, what I wanted to do, since we've got just a little over 12 minutes left, is I can keep going and review some of the additional image processing base classes, so that as a developer, you'll say, oh, I'm developing this kind of node.

That's the base class I want to start deriving my class from. Or would you like me to open up to Q&A? Because we've got 10, 12 minutes. If you guys have questions, I can spend the time answering your questions. So any vote, yay or nay? Keep going. Keep going.

Okay, we'll keep going, unless you've got questions. Stop me if you've got questions at this point. Okay. Not yet. Yeah. Fair enough. I can answer that question if you'd like right now. There are image processing base classes in Shake that support concatenation. The concept of concatenation is, imagine if you've got five image processing nodes that do transformations, and all of those transformations use a 4x4 transformation matrix.

Well, if you have all of those four nodes basically stacked up together, Shake has logic that takes each of the transformation matrices and concatenates them, and basically takes what looks like four image processing operations and concatenates them so they're actually executed as a single image processing operation. This is done with lookup table-based operators. Certain color-based image processing nodes do support that.

Just arrived from the right class. Yeah, so let me actually go through the classes really quickly. So a very popular one is NRI Monadic. It takes a single input image and produces a single output image. It doesn't cache image data, though. So if you've got a very expensive image processing operation, look at the NRI Cmonadic base class, because that actually has an internal node that actually supports the caching of image data. Like we talked about right at the beginning, nodes can have children nodes, and if you derive from NRI Cmonadic, that is a node that supports caching of image data. but it still takes a single input image and a single output image.

Some image processing nodes, they know ahead of time that they want four input images. And that's never gonna change, it's always gonna wanna work on four input images. You're gonna wanna derive from the NRI an attic base class, so you can preset how many input images there's gonna be. Shake will call you then with the fill output buffer member function, it'll basically prep all the input image buffers.

They'll be there in an array for you so you've got all your input image data. And this is a really popular base class, because a lot of people like getting the full frame input image buffer. In this class, you can set a flag and say, give me a full frame input image buffer.

Guess what, my algorithm, it's doing FFTs, I need the full frame image buffer. You can set a flag, and with NRI and attic, it can work with one input, just as well as it can work with ten inputs. So some people will just use NRI and attic simply because they want, they love the full frame image buffer functionality.

If you're a generator node, one of the things, if you want to generate an image, you'll want to derive from the NRI input member function. The nice thing about it is you might think, well, I'll just start with NRI effects because it defines a single output image. But the things that you get for free from NRI input is how to manage the valid time range and that sort of thing. So there's a lot of logic built in there so that you can pass information down the tree about how to handle once you're outside of a valid time range and that sort of thing.

And let me just wrap up at this point. We've got a little bit, we've got some more information. Documentation, sample code and other resources, the WWDC website is one good place to look. Who to contact? Patrick Collins is the track manager, so you'll want to, and technology manager, you'll want to contact him. And also myself, angus at apple.com. And actually, you know, an address that I want to give you guys is shake-sdk at apple.com. If you're interested in getting a hold of the SDK package, drop us a line there so that we can start talking to you.