My name is Matthew Badin, and I'm an engineer at Apple. Welcome to the Using Accelerate and simd session. So my colleague Luke Chang and I are pretty excited today. We get to talk to you about all the great APIs that are available in Accelerate and its associated frameworks, and we're going to start by taking a high-level overview of Accelerate and some of the high-performance libraries that are contained inside it.

We're then going to take a deep dive into a few of libraries, and we're going to start by taking a look at vDSP. We have two examples for you. The first one, we're going to show you how to extract signal from noise, and then we're going to show you how you can remove certain types of artifacts from an image. We're then going to take a look at simd, and we're going to show you how you can use quaternions to represent rotations in three dimensions.

I'm then going to pass off the presentation to my colleague, Luke Cheng, who's going to show you some of the interesting things that you can do with vImage. So with that, let's get started. So you're probably asking yourself exactly this. What on Earth is Accelerate? So the primary purpose of Accelerate is to provide thousands of low-level math primitives, and we provide these primitives across all of Apple's platforms. So this includes not only iOS and macOS but watchOS and tvOS as well.

Most of these primitives are hand-tuned to the microarchitecture of the processor. So this means we get excellent performance. And this performance translates directly into energy savings. So if you're an app developer and you use the Accelerate framework, not only will your application run faster, but you'll also use less battery life. This means your users are going to have an overall better experience.

Now, because we provide so many primitives, we find it useful to organize or group them into domain-specific libraries. So, for instance, we group all the signal processing primitives under vDSP. So, for instance, this would be your FFTs or your DFTs and your DCTs, your fast Fourier transforms, your discrete cosine transforms. VImage contains the image processing primitives. If you're doing color space conversions, this is the library for you. VForce contains vector versions of the transcendental functions. So for instance sine and cosine.

We also have support for dense linear algebra as well as sparse linear algebra, and we have a specialized library for neural networks called BNNS. It stands for Basic Neural Network Subroutines. Not strictly part of the Accelerate framework but very closely related, and we find these libraries very useful, include libraries like simd, which is a vector programming aid for the CPU, and Compression, which contains several different lossless data compression algorithms. So let's take a look at our first library. We're going to take a look at vDSP. VDSP is a state-of-the-art signal processing library, and it contains a wide range of signal processing primitives.

These range from basic arithmetic operations on arrays such as add and subtract as well as more complicated operations like convolution and FFTs. If you're a successful app developer and maybe in the past you have avoided FFTs, I want to show you how we make that easy with Accelerate. With just a few lines of code, you can make that work. And I have an example. I'm going to show you how to extract signal from noise.

So what we have here is an audio signal. This is the base signal. We haven't added any noise to it yet, and you'll notice I have two sliders at the bottom. In the lower left I have a slider that lets me add noise, and you can see that.

And I also have a second slider that let's me remove this noise called Threshold. You'll also notice I have a toggle switch in the lower right. This lets me look at this under a different domain. So currently we're looking at this signal under the time domain. We're going to do some analysis and see this signal in the frequency domain.

You can see all the spikes on the left. Those are the frequency components of the signal. You'll also notice the blue bar. The blue bar is the threshold slider, so you can see I can move it. So what I'm going to do now is add some more noise to the signal. Actually, let's add quite a bit of noise.

So you can see the signal that I'm still interested in is represented by the spikes on the left or another way of looking at it is the spikes with the tallest height, and what we've added, the type of noise we're adding to this, is background noise. So it's kind of evenly distributed spikes everywhere, but they're low-level spikes.

So now I'm going to remove this noise, and the way I'm going to do this is I'm going to move this threshold slightly higher. As I do this, what's happening is, behind the scenes, is we're identifying any frequency component less than this threshold and removing it. We're saying effectively if there is any spike with a height less than this blue bar, we're going to pretend it's noise and remove it. So if I keep sliding this higher, eventually I've removed all the noise.

And if I go back to the time domain, you can see that I removed the noise. And in case you don't believe me, we can remove the threshold, and this is what this looks like with all the noise still added. So let me show you how we do that.

At a high level what we're going to do is first perform an analysis on a signal. That's what the switch let me do. We're then going to identify the frequency components that represent the noise and remove them. After we have done that, we're going to reconstruct the audio signal. So let me show you some code.

What we're going to use here is a discrete cosine transform or a DCT. You can see here the DCT CreateSetup. This context is going to describe the type of work we're going to do as well as give us space to perform this work. In this case, we're going to use a type 2 DCT. We're then going to pass this context to an execute function.

This will actually perform the work. Once we've performed the analysis, we want to remove the noise. This right here is where the magic happens. This routine is going to identify any frequency component less than that threshold and make it zero. It's going to zero it out. After we've done that, we want to reconstruct the audio signal.

Again, we're going to need to use CreateSetup to create context. In this case, we're going to use a type 3 DCT to reconstruct the signal. We're then going to pass this context DCT execute, and this will actually perform the work of reconstructing the audio signal. So what we've shown you is an example of how you can remove certain types of noise from an audio signal using vDSP. I want to show you an example of how we can remove certain types of noise from an image also using vDSP. In this case, we're going to restore an old newspaper photograph.

So what we've done is we've taken this image, and we've applied a two-tone screen to it. So this could represent an old newspaper photograph. And what we're going to try to do is remove this screen. We're going to try to remove the artifacts that you see. Currently, we're in off, so we're not doing anything. What we're going to do is actually take a sample of this screen, and we're going to create a mask from that sample and then apply that to the image to try to remove it. So let me show you the first attempt.

So what we're doing is we are identifying a frequency component at a certain threshold, and any frequency component higher than that threshold we're going to remove. And you see, if we set the threshold too low, not only do we remove the artifacts, but we also remove quite a bit of the image as well. If we set it too high, you can see that we didn't do much of anything. Medium seems about right. Medium seems too correctly identify the artifacts in the image without damaging too much of the image. So let me show you how we do that.

At a high level what we're going to do is we're going to perform an FFT on the image and the sample. We're then going to create a mask from that sample and apply it to the image. Once we've done that, we're going to reconstruct the original image. So let me walk you through some code. We're going to do an FFT. This means it needs to be a power of 2. That's why you see the log2 call. The 1024 x 1024 is the size of the image.

We're then going to pass it to fft2d zrop. This is quite a mouthful. The important part here is the op stands for out of place. So we're going to have to create some temporary space just for the result. We're going to store in this complex structure, and effectively this says we're going to store the complex number in two arrays with the real component in one array and the imaginary component in the second. We also need to specify a direction. In this case we're going to do a forward FFT.

Now the artifact removal is a little bit more advanced, so I'm only going to go through this at a very high level. I recommend you download the sample application that's available online right now, and each one of these routines is also documented under vDSP, and we have excellent online documentation.

A high level what we're going to do is we're going to identify the magnitude of the frequency components, in this case with the sample. We're then going to identify the components we want to remove. We're then going to create a mask from that. And once we've done that, we're going to apply that mask to the image. Effectively what we're doing is we're multiplying by 0 the components we want to remove and by 1 the components we want to keep.

We're going to use zrop again to reconstruct the image. Because this is an FFT, we get to reuse the context. In this case, one key detail is we're going to store the image in two arrays because this is a complex structure. So the even pixels are going to be in the real array, and the odd pixels are going to be in the imaginary.

And again we're going to specify a direction. In this case, we're going to use an inverse FFT. So now I want to shift gears for a moment. Previously we showed you two examples, and we had two working examples for vDSP, and then we worked backwards, and we showed you how we constructed these examples. We worked backwards and showed you the moving pieces.

For this next library, what I want to do is start at the basic components. I want to build up to a result. So we're going to take a look at simd, and we're going to start at the basic low-level primitives, and we're going to build up to rotations. In this case, rotations of 3D objects.

At a high level, simd is an abstraction for the vector processing unit. So what this lets you do is declare vector and matrix objects, and you get to perform operations on these objects, and this will map directly to the vector hardware of the processor. So let me show you a coding example. So what we're going to do here is we're going to take two arrays, and we're going to average the components. So we're going to iteratively go through each of the scaler components, add them together, and divide by 2.

This is going to be really slow. Instead, you can declare these arrays as simd float4 vector types. So then we can perform basic arithmetic operations on these objects. So not only can you express the computation more naturally, but this will also run about as fast as it can. And this will work across all of Apple's platforms.

Simd has a tremendous amount of functionality. In addition to vector and matrix objects and allowing you to perform arithmetic operations on these objects. It has extensions. So for instance, dot products and clamp. It also has support for the transcendental functions, so for instance sine and cosine as well as quaternions. Quaternions are very useful for representing rotations in three dimensions, and I actually want to talk a little bit more about those. So I'm going to walk you through a coding example.

There's quite a bit to unpack here. So we're going to start on the right. We have a unit sphere. That's what this gray sphere is. And you'll notice this red dot. And that's actually the tip of this vector. We've declared a simd float3 vector. We've set the x and y components to 0 and the z component to 1. So it's coming out at us. And that red dot is represented by the tip of that vector.

We're now going to perform a rotation on this vector using a quaternion. Technically, we're rotating the entire scene, but for purposes of illustration, we're going to say we're rotating this vector. When using a quaternion for rotations, you need to specify an axis and angle of rotation. Or to put it another way, what are you rotating about and by how much.

So we're going to rotate about the x axis, and we're going to rotate pi over 3 radians up. You apply the rotation by calling the simd act function. This applies the action of the quaternion on that vector and returns a rotated vector. So let's take a look at that now.

So normally you're not interested in rotating along a single axis. You usually want to rotate along multiple axes, and if you're already familiar with rotation matrices, this is going to seem very natural. Like rotation matrices, you can combine the rotations using multiplication, and also like rotation matrices, the multiplication is not commutative.

So this means if you change the order of the operands, you will change the order of the rotations. So effectively what we're going to do here is rotate pi over 3 radians up, and then pi over 3 radians to the right. And we're going to combine that into a single rotation.

Some of the more interesting things you can actually do with quaternions and simd is interpolation, and we support two types of interpolation. The first is Slerp. It stands for Spherical Linear Interpolation, and there are actually two variants of it. We have simd slerp, which will find the shortest arc between these two points, in this case between the blue and the green, and we have simd slerp longest, which will find the longest arc. So you'll actually see it go behind the unit sphere.

The second variant is Spline. Spline is really more useful if you have more than two rotations. So, for instance, here we're going to interpolate between an array of rotations, and there's quite a bit of boilerplate code here, so I want you to focus just on the Spline call.

Effectively what we're doing is just iterating through all the individual rotations and applying Spline. So the thing you have to specify with Spline is not only the two rotations that you wish to interpolate between but also the previous and the next as well. And this is what this looks like.

So if you're a game developer, you're probably not interested in rotating individual vectors. You're probably interested in rotating objects. So we have that for you. We have a cube. It's represented by multiple vectors, and it's going to go through a series of eight rotations. On the left, we're going to trace those rotations using Slerp, and on the right we're going to use Spline. So let me show you what Slerp looks like.

And you can see because it's a linear interpolation, every single time it changes direction you get these sharp corners. Whereas if we look at Spline -- Because it's aware of the previous and next rotation as well, you end up with these rounded corners. So let's see that again.

So now I went through all those topics pretty quickly, so as a quick recap, we started by taking a look at vDSP, and we showed you two examples. The first was how to extract a signal from noise, and the second was how to remove certain types of artifacts from an image. We then took a look at simd, and I showed you how you can represent rotations in three dimensions using quaternions. I'm now going to pass off the presentation to my colleague, Luke Chang, who's going to show you some of the interesting things you can do with vImage.

Thank you, Matthew. Hello everyone. My name is Luke Chang. I'm an engineer in Vector and Numerics group. Today, I'm going to talk about vImage, what vImage offers, and how easy it is to use vImage in your apps. With just a few lines of code, you can create engaging video effects in your app. Let's get right to it.

VImage is our image processing library. It has several components. The first component is the conversion function. Conversion function help you move image between different image formats. Different image formats have different advantages. For example, RGV format matches the pixels on your display, so it's best for the display. On the other hand, we have YCbCr image, which is similar to how human perceive the image. Human eyes recognize brightness, which is the luminance channel. Also, the color, which is the chrominance channel. Also, camera uses YCbCr format to capture images. So converging function helps you easily move images between these formats.

Next, we have geometry function. Geometry function changes the size or orientation of the image. We have vImage scale that can do enlarge or shrink the image. We use Lanczos algorithm, so we'll have high-quality output after the operation. We also have vImage rotate that can rotate image clockwise or counterclockwise.

Next, convolution function. The most notable effect of convolution function is the blur effect. You see blur effect all the time, in UI, in photography. If you want to phase something into the background, you can apply the blur function, blur effect. Next, transform function. Transform function is basically a matrix multiply. It lets you operate on the data channel of each pixel. Let's say you want to increase red or increase green, you can do that with transform functions.

Morphology. Morphology changes the size or shape of the objects in the image, not the image itself. We have vImage erode, vImage dilate to make the object smaller and bigger. If you're feeling adventurous, you can actually provide a custom shape of kernels to these functions, and vImage erode and dilate will make the object smaller or bigger according to the kernel you provided. Those are the five things in vImage. Now, I want to show you a demo app that we wrote based on vImage and show you what kind of effect you can get from vImage.

What I have here is a lab that captures images using the back camera and put the image onto the screen. And we're doing in real-time. This is a live stream, so you can see the drinking bird doing its motion next to the roses. All right, the first effect I want to show you is the color saturation effect. You can see this effect in a lot of photo editing software. So I want to bring out the color. What I can do is move this slider to the right to make the red really red and green really green.

And on the other side, I have white roses. I feel that the color of white roses is not that interesting to me. I would like to direct the focus of my audience to the composition and the contrast of this image. I can slide to the left to desaturate the image to the point that this becomes a black and white image. So now color is no more distraction in this image, and the viewer can focus on the composition and contrast.

Okay. So, how do we do this? There are several steps we need to take. First, of course, we have to get the image from the camera, and then we want to use vImage to process, to apply the effect, so we have to prepare the input and output buffer for vImage. Then we actually calling vImage functions to apply these effects, and we display the output to the screen. Let me jump ahead and talk about how do we apply effects using vImage functions.

The effect I show you is a color saturation effect, and here is the formula to do color saturation. Basically, we want to remove the bias from the pixel, and using multiplication to apply the saturation effects, and then we put the bias back to the pixel. VImage has exactly the function to do this operation, which is vImage matrix multiply.

VImage matrix multiply takes the preBias, in this case minus 128, to remove the bias, and because the saturation is [inaudible] point and the image is an integer, we want to convert this saturation value first into fixed point format. We chose Q12 as the fixed point format, hence a divisor of 0x100. And then we have the postBias 128 times the divisor, just to put the bias back to the pixel.

And the matrix itself is really, really simple. All we want to do is just doing a scaling of CbCr channel. So the matrix itself is just a scaler, want to multiply CbCr channel with this scaler. We have all the information, so let's call vImageMatrixMultiply, and with just one line of code, one function call, you can achieve the saturation effect.

Now let me come back to other steps that we need to take. We need to take the image from the camera. How do we do that? We need to write a delegate method, and what camera gives us is a CV image buffer. So we get the buffer. We have to make sure this buffer is accessible to CPU. That's where vImage lives. After we apply the effects, whatever effects that may be, and we have to unlock the base address of this pixel buffer so that the camera can reuse this piece of memory.

The second step, we have to prepare the vImage input and output buffer. We already have this image in CV image buffer. All we need to do is just get the information such as height and width, and then we can package this into a vImage buffer object so it can be consumed by vImage library. We do this for luminance and chrominance channel.

Now we need to prepare an output buffer. Remember, we don't have a piece of memory allocated to the output image yet, so we need to do that, and vImage has a convenience function, vImage buffer in it, to do just that. Given this height, width, and bits per pixel, vImage buffer in it will allocate a memory that's large enough to hold this image and then also create a vImage buffer object so it can be consumed by the vImage library. Last step is put this process image to the screen.

Like I said before, RGB is really the best format for display, so let's use the conversion function to convert YCbCr image into a RGB image. And then because the UI expect the cgImage object, we have to create one. There is a convenience function in vImage. vImageCreatesCGImage FromBuffer that helps you to create a cgImage based on the buffer you already have in vImage.

One thing to note is that we're not actually copying the large data buffer in the image from one place to another. We're simply creating a cgImage object that adds a container to this image buffer. So we are only filling in the information that cgImage needs, create a cgImage objects, instead of copying data around.

Once we have that, we can send the cgImage object to the image view, and it will be displayed on the screen. So it's that simple. Four steps and you can create your own effects, we show you the saturation effects. Now, there are other effects we can do with vImage.

We can do a rotation, like I said before, rotate the image clockwise, counterclockwise. We can do blur or phase something into the background. And you feel, if you feel like you want to add some retro feeling to your images, you can do dithering for black and white images, and color quantization for color images. Let me show you how they look like in the app.

So, again, I have the slider here to control the rotation. I can do rotate counterclockwise or rotate clockwise. Now I want to try the blurring effect. Let me click on the one here. And I can apply more blurring or slide to the left, bring the roses back to the foreground.

For black and white, I can use dithering. Now this black and white image, the gray scale is represented now by the density of the dots, that's the dithering effect. And we use the accents and dithering algorithm for this, and I will show you how to do it later, and for color quantization, we have lookup table.

And I can move this slider to increase the quantization level. As I move the slider to the right, fewer and fewer color is available in this image. That's sort of creative, that's how your computer screen looks like in the '90's or in the '80's. All right. So let me show you how we do that.

For rotation effect, you can call vImageRotate and given the angle of rotation, it will do counterclockwise or clockwise to better align your images. For the blur effect, we use TentConvolve. The blur effect is controlled by the size of the kernel. The larger the kernel, the more blur you'll get. Dither effect is basically converting an 8-bit image into a 1-bit image. At the same time, you can specify a dithering algorithm. In this case, we used the Atkinson dithering algorithm.

Color quantization, we used the quantization level to create a lookup table for the RGB channel, and we call vImageTableLookUp to apply this table lookup to RGB channels to limit the number of colors on the screen. Those are the four additional effects I wanted to show you, and I think now is a good time to move onto the next topic.

LINPACK Benchmark. We talked about the functionality of Accelerate. We talk about how easy it is to use in Accelerate in your apps. We haven't talked about how fast Accelerate really is. And LINPACK Benchmark is a perfect tool to do that. What is LINPACK Benchmark? Basically, it's trying to measure how fast you can solve a linear system on your machine.

There are actually three different LINPACK Benchmarks. The first one is solving a 100-by-100 linear system. The second one is solving a 1000-by-1000. The last one, which is the most interesting one, that's the one we're going to use today, is no holds barred. You can solve as large a system you want to fully utilize every last bit of the computational power on your machine.

Let's see the performance of iPhone X using Accelerate. The performance is measured in gigaflops, for double precisions catch up with iPhone 5S, iPhone 6, iPhone 6S, iPhone 7, iPhone X comes in around 28.7 gigaflops. That's double precision. Let's look at single precision. Again, we run out of space, we have to shrink to try to make it come closer.

IPhone X comes in at 68 gigaflops. Now you might be thinking, this is not that surprising. However, it improves over time, so the performance improves over time as well, but in fact, that's only half of the story. When there is a micro architecture change to have additional computational power into a machine, you need the matching [inaudible] to fully utilize this additional computational power. And that is where we come in. Remember, this is the same LINPACK Benchmark executable running on all five generations of iPhones. They all got the best performance using Accelerate without changing a bit.

The same is also true for your apps. If you use Accelerate in your apps, you'll get the best performance automatically on all the architectures we support. Moreover, Accelerate supports across platforms. Accelerate works on macOS, iOS, tvOS, watchOS. So let's say if tomorrow Apple comes out with a new architecture or new platform, you don't have to worry about it. All you need to do, the most you'll ever need to do is just to rebuild your apps, link against Accelerate. You will automatically get the best performance on the latest release platform or architecture.

Just as a summary, we talk about Accelerate supports a wide variety of functionalities. More than likely, you will find something you need in Accelerate. If you need something that's not available, please feel free to file feature requests. We constantly look at this feature request, evaluate them, and then put into builds. Actually, some of our best features come from feature requests.

Accelerate is easy to use. Most of the time it's just one function call and the job is done. It's fast and energy efficient, so your app is more responsive and the battery life is longer. Accelerate is portable across platforms and architectures. You get the best performance on all the platforms we support and all the architectures we support, and the best part is, you don't have to change your code.

For more information, you can go to our online documentation at developer.apple.com, and all our demo apps, simple code, and session material will be available online. We have a lab session tomorrow afternoon at 2. I look forward to seeing you guys there. If you have any questions or you want to learn more about Accelerate, I'd love to see you there. That's all for our presentation today. Thank you all for coming. Have a great day.