Good morning. Welcome to Leveraging Touch Input on iOS. My name is Dominik Wagner, and I'm an engineer on the UIKit Team, and I will tell you about how you can make the most out of our advances in Multi-Touch and of Apple Pencil. First, let's go over our new and recent hardware releases. Since last WWDC, we released a lot of stuff.

For example, 3D Touch with the iPhone 6s and 6s plus, which gives you access to the force for each touch. I won't talk much about this, but you can have all the great experience for Peek and Pop in the session at A Peek at 3D Touch later today.

We introduced faster touch scanning beginning with iPad Air 2 and iPad Pro, giving you a temporary resolution of twice the refresh [inaudible] display. We also introduced Apple Pencil for Apple iPad Pro, and this amazing device, thank you, gives you precise location at a [inaudible] accuracy for your drawings. It has an even higher temporary resolution of 240 hertz. It gives you access to its tilt and orientation, and, of course, all the force.

And our algorithms team did an amazing job of palm rejection. So you can rest your palm while you are drawing and don't have to think about it. We also released Apple TV with the Siri remote, and its track mainly drives the UIFocusEngine and your interactions with Apple TV. But you can also access the track pad using the Game Controller framework for both acting as a Game Controller or also getting the absolute position of the track pad. And, finally, you can also handle the track pad in indirect touches in our UIKit touch handling methods.

I won't talk much about this either, but there are the great Apple TV tech talks, and there's also Game Controller Input for Apple TV yesterday. What am I going to talk about? I'm going to talk about how you build a drawing app. We will build a drawing app from the ground up, and I will talk about all the new API for Apple Pencil so you can access all these great new data. We will talk you through it step by step, and show you all the different steps you can take depending on what you see, and the sample code of this app is completely available. So you can just relax on the code slides and play with it later.

So say hello to SpeedSketch. This is our sample. SpeedSketch is one sheet of paper you can draw on. It has full support for Apple Pencil on iPad Pro and 3D Touch on iPhone 6S. It also works on all previous iOS devices. So let's first talk about the model. So this is a stroke, and this is how UIKit sees the samples. Each of those points is one instance, the same instance of a mutable UI touch. You get handed down in the touch handing methods. And we will model our data as a series of strokes.

And because of the UI touch being a mutable representation of one touch sequence, we need to copy the data out of this UI touch in the touch handing methods into something more static. So we will build a stroke sample as being the innermost element of our data structure, which for now will just contain the location of this, of the UI touch, but later on, we will fill in all the additional data parts.

And we will put those samples into a stroke. Let's just an array of stroke samples, and we have a little bit of methods to add them. And because we want to use the stroke as our main data capture structure, it also has a stage. So it can be active while the user is drawing, is done, when the user is done, and it can be cancelled if a different user interaction causes it to be cancelled. So we can throw it away instead of keeping it around.

And, lastly, we put this into a stroke collection, which will just be an array of strokes, and we will add the down strokes, and to make it a complete data model for our app, we will have the active stroke as an optional in this. So we can use the stroke collection as our data model for a complete app.

So next up is where to capture those strokes. So, essentially, you have three places to do so. The first one, is a UIGestureRecognizer, a custom UIGestureRecognizer [inaudible] plus. The next place to look for is a UIView subclass where the touch goes down, and you can handle it all on the way up the responder chain, and I want to think, I want you to think about this in this order. If you can do a UIGestureRecognizer, or a custom subclass do that. Second, do a UIView as close as to the events as you can get, and only if you have to go up the responder chain.

So this is what we going to do. We are building a stroke gesture recognizer accustomed to a subclass. We will target our main view controller, and then in the action methods, we will trigger a redraw of our view with a stroke. So let's start building this gesture recognizer subclass. The first thing you have to do is to import the UIGestureRecognizer subclass. This exposes the internals of a UIGestureRecognizer to you so you can do subclassing.

Be aware, though, that these internals should not be used outside of a UIGestureRecognizer. For example, the state setter is exposed if you import this, and you shouldn't use it outside the subclass. Otherwise, the gesture system will act in not so nice ways, and you will have several issues to debug.

So let's add our stroke as our main data structure in our gesture recognizer to capture the stroke and implement the touch handing methods. And since we do various similar things in all the four touch handing methods, we will have a helper method actually looking into the set of UITouches.

Determining if we are interested in one of those touches. Adding that to our data model just by copying the location as a sample at the moment, and return to us if you were interested in those. We will use this helper method in the touchesBegan and set our state to begin.

For UIGestureRecognizer, that is unusual because the time between the state possible and the state began is the time that different gesture recognizers can compete about have, handing this touch sequence. However, for our stroke, we really want to begin immediately when the touch goes down. So this is exactly what we want to do. In touchesMoved, we do the same and switch the state to changed. Note that in a gesture recognizer, each state change triggers an action method, even if it's for the same. And we do the same for touchesEnded and Cancelled.

And, finally, we have to reset our gesture recognizer and replace the previous stroke with a new one so we can catch on the next one. And as a good citizen, we always call super.reset. Let's use this in our ViewController. In our viewDidLoad, we set up our stroke gesture recognizer, ourselves as the target with the action strokeUpdated, and then we add it to our main view.

And in the update callback, we just take the stroke from the gesture, and set it on our view. That's all for now. I told you about the stroke collection before, but just to bring up, I like to keep it simple. So let's see what this gives us. I always like this moment when the first time something will run. So what you see is this is the position of the pencil on the display. Let's have a look. Oh. That doesn't look good. What happened? Let's have a closer look in slow motion.

That's really not what we want to do. So let's, what did we see? We have the position of the pencil. We have a really, really big distance to the last line we drew, and the lines we drew were really long and choppy. That is not what I, what I was expecting. We have a really good temporary resolution.

Why does it look this way? So, obviously, we missed some events, and we missed some events because of multiple reasons. One of them is our drawing engine is implemented very naively. It is drawing a complete bit map every time, and event comes in, and that is not quick enough to keep up with the display rate.

And we will talk about this a little bit later. And the most, more important thing is we did not use the new iOS 9 API that we have to actually get to the missed events. And for that, I want to look a little bit closer at a stroke. So let's see that again. Here are all the rich samples we want to have and see. And in our touch handing methods, we always get the began and the ended, but in between, we get touchesMoved.

And we don't get all of the touchesMoved that came in as a sample, and that's for reason. If we would deliver to you all the touchesMoved even if you would block the main thread, you would see something akin to a replay of the touch interaction, and if you do live interaction, that's not at all what you want.

You want the most recent position delivered to you, and that's what we do generally, and that's what we did before OS 9. We actually dropped all the other events. If you weren't quick enough in the main run loop, then you just didn't see any of them, but beginning with iOS 9, we give you access to the previous ones. And the other reason is as our digitizer is now faster than the display, we don't want to give you each an event for each datapoint we have. So you do too much work. So we try to coalesce them together into one delivery per [inaudible] refresh.

So what you can do now is in the, using the life touch you get in our API, ask for the touches you missed, and that includes the touch you currently asking for. So you have a complete consistent picture. And those touches are called Coalesce Touches. And you do that for all your touch events.

And you do that also for begin and ended. Because you have to stay in the same area. Either you handle your life touches or you handle the coalesce touches. Because there are methods like previous location in view that reference the previous touch, and if you mix and match, then you into problems there. So now that we know how, how we can get them, let's do this in code.

The method is coalescedTouches for touch on UIEvent, and you get the event in the touch handing callbacks. And now that the optional of this result is not because we will give you nil. Anytime if you only have one touch, and we don't have Coalesced Touches, it's because you could ask for any UI test that is not part of the event, and then you would get nil. You're guaranteed to always get UI touches, at least the one you put in even if we didn't coalesce more. So you don't have to do an if statement around this.

So let's use this in our code. In our code where we look at the touch we're interested in, we did appendTouch, and all we have to do is to loop over the Coalesce Touches instead and append those. That gives us all the data. Let's have a look at how that looks. Nice. Now we really have all the data. Let's have a comparison. And, again, in slow motion with a stop.

So what are you seeing here? This is in [inaudible] the Pencil's location on the glass currently. These are the Coalesce Touches in gray, colored in gray. Now debugging drawing engine. The black one is also a Coalesce Touch, but the one corresponding to the live touch. And what you also see is that you see too many Coalesce Touches. If our digitizer is running at four times the speed of the display, you see, you should see an average three gray ones among black ones. These are way too many.

And we still have this excessive gap at the end, which leads to lag, visible lag for users. So what did we see? The speed is still lagging of our drawing engine. We didn't address this up to now, but UIkit helped by coalescing the touches, and the final drawing really has all the data it needs already.

So if you only take one thing of the sort, then use Coalesce Touches if you really want to have the rich data of your Pencil. So what is the problem with the drawing? You should not draw on every touch event because the display refresh rate is 60 hertz.

Although we try to actually deliver only one event per frame, you can draw, that sometimes is impossible because if you mix fingers and the Pencil or have other events coming in, we need to deliver them in order. And so you need to be prepared to receive more events than you can, than [inaudible] refresh rate. And do not try to draw faster because this is just costing performance and adding lag and doing work that doesn't even get displayed on the screen.

So when should you render? In our example, we are using a regular [inaudible] core graphics, and in that case, you should just use setsNeeds display on that view, marking that view is needing update and giving the work to CA to actually call the draw method you need to implement instead of your custom bit drawing.

If you're using a GLKView or a MetalView you can also opt into this behavior instead of a steady update by setting the enableSetsNeeds display property to true. That way those views behave in the same way if you want to. If you need to draw at a steady pace, then please draw at a steady pace and not based on the events coming in, and you can do so using the Metal and GLViews internal mechanisms, or you can do so using a CADisplayLink and calling display in the wake of your DisplayLink. What we did was we had in our strokes [inaudible] just a did set on the stroke to draw with drawImageAndUpdate. This created the [inaudible] this was bad. So let's just do setsNeedsDisplay, and move the drawing code into the draw method.

If you're using a regular UFU, you can do even better. You can only mark the changed areas a setDisplayInRect that also needs some kind of bookkeeping in the touch because your drawing might, might be a little bit bigger than the changed touches, and the touches change in our samples. You can look at the sample code to have one example of how you could do this kind of bookkeeping.

And even further, you can activate drawsAsynchronously on the layer. This puts all the drawing you do in the draw rect up to CG, up to CA, and CA draws it outside of the main thread which opens your main thread for quicker event handling, and you do so by simply setting drawsAsynchronously to true or your diffuse layer. Let's see how far this got us.

Again in slow motion. So now we have the steady amount of Coalesce Touches I was talking about, about three Coalesce Touches and one black one. But we still have some lag. It however, it's way smaller because now we really draw at the display speed, and we just have the remaining lag. And how can we improve that? So we have a facility called Predicted Touches since iOS 9, and Predicted Touches give you a glimpse into the future.

You use the same way as Coalesce Touches, and you ask your event for the Predicted Touches for touch, and you get an array of touches that are in the future. And what do you do with those touches? You add them to your data structure but temporarily. They change on every event callback. So you really have to do them just temporarily.

And you choose their appearance, depending on your app. I highly recommend that you make them appear like actual touches, and look at the result, and only if our prediction is off by too much, then tone it down. It makes them appears tentative to still get the closer look to the Pencil.

Let's look over that in code. So now in our touch setting methods, after you added the Coalesce Touches, you add the Predicted Touches temporarily, and you need to make sure that you remove the temporary touches before. So I will show you a video of how that looks with the opposite of what you should do. I will highlight the Predicted Touches in red so we can see if they are good enough for our example. And, again, in slow motion because this was so quick.

So these are the Predictive Touches, and they get you closer to the actual Pencil position, which is really, really helpful for perceived lag on the display. And as you can see in this example, this really worked out fine. So we will just use them and draw them the same way we draw regular touches.

So have we seen so far? We have seen how to collect the input using an custom UIGestureRecognizer, how to access the Coalesce Touches, how to make the rendering faster and efficient, and, finally, how to use the Predictive Touches. All of those techniques work on, across all iOS devices. We used them for the Pencil right now in our examples, but they work overall. Now let's go on to the actual new Apple Pencil API.

So let's begin with touch types. With Apple Pencil UITouch added a new method called type, and UITouch type can be one of three values. It can be direct, which would be all your previous touches you know about. There's indirect only for the Siri remote touches, and there's stylus for Apple Pencil.

And the first thing you can access with Apple Pencil is the higher precision, and you do so by using precise location in view, and you also have the precise previous location in view. You should use those whenever you want the precise location for something like a drawing. If you want to do hit testing, you should still use the previous ones called location in and previous location in. But for drawing, this really makes a difference. Without the precise locations, you will add some staircase patterns to your drawing, which you don't want to see. And you can ask all kind of touches for precise location. You will just get the regular one.

Next up, there's force. Force is exposed as a property called force and a maximum possible force. Those are CG floats, and they are in the range from zero to the maximum possible force where 1.0 represents an average touch. So these are not really physical values. So you shouldn't do anything that relates to the actual force, but you use this value to affect your drawing. And on all previous devices and for regular finger touches, it will always return zero.

A quick note on force. Since we added force to UITouch, there's one difference in touch handing, and that is touchesMoved gets called more often. Because you want to be able to discern if the force value changed. We will send you touchesMoved now all the time. Previously, we were trying very hard to only send you touchesMoved when the location actually changed, and that is through regular location, not even the precise location.

And that has implications for you. So, for example, we've seen a lot of this in the wild. If you did in touchesMoved, if you just cancel a tap willingly, that is bad. That doesn't work anymore. And what you see is that on a, on an iPhone 6s or with the Pencil, if you have to really, really just slightly touch your display, then this is what you're running into, and you should have a look at your touch handing code.

What should you do? You should actually use a UITouchGesturesRecognizer if you can because it encapsulates all of our knowledge there, or the least thing you have to do is to remember the location where the touches began and only cancel it if you moved enough distance away from the original location.

So let's add our force to the model, to our stroke sample, and we just do that by an optional force variable. We will add all the other things in there later, too, and I won't show this slide again. So let's see how the force looks in our drawing.

Nice. What did we do? We just changed the width space of the force. So next up is tilt. Apple Pencil gives you access to its tilt towards the device, and this is measured in an angle between the Pencil and the device which we call altitude. And the altitude angle is exposed as altitude angle. It's a CG float. It reported at angle at a radian from the range from about 10 degrees to 90 degrees.

And that second part is the orientation. The orientation is measured [inaudible] to your device, and it's measured between the positive x direction and the direction the Pencil is coming from. And this is called azimuth. Together, azimuth and tilt form the full location of the Pencil, and you can drive your UI or your datapoints for your drawing with it.

So azimuth is depending on your orientation of the device. So you have to call a method called azimuthAngleInIvew, and most of the time you probably want to use a vector anyways. So we exposed the azimuth unit vector in view to you which will be a vector pointing in the direction the Pencil is coming from with a magnitude of one.

Next up, force. So force, for the Pencil behaves a little bit differently than 3D Touch force, and that is the force is measured along the axis of the Pencil. With 3D Touch, the force is measured on the display and perpendicular to the device surface. That makes for some difference, and I really urge you to try out if you want to have the actual force, all the perpendicular force in your drawing tools because it really, it really feels different. Luckily, it's easy to calculate this component, and here's the code for that. So you can get the perpendicular force by dividing the force through the sign of the altitude angle and to make sure you stay in the same range, you should also clamp it to the maximumPossibleForce.

And there's finally another tidbit for Apple Pencil with the force. It's measured inside the Pencil, and then transmitted over the air to the iPad, and this is all the properties of over-the-air transmissions. That means they take a little bit of time, and data can be lost. So instead of making you wait for the over-the-air transmission of the force, we decided to give you estimated properties at first and update them later so you can have the best experience. So for that, we exposed estimated properties on UITouch, which is of the type UITouchProperties. So estimated properties can have a value of force, which for Apple Pencil will always be true for the first event you get.

But also the azimuth and altitude can be marked as estimated. That happens when you come in from the sides, and we not hundred percent sure what the values will be or if you draw very closely to your fingers because our senses can't really detect them super-accurately. And we tell you that they are estimated so you can do something with it.

For example, when coming in from the sides, you could back fill them with the first solid value, and in the sample I gave to you, that is what I did just as to illustrate that point. And there's also the location, which is an estimated property only for the Predicted Touches, which gives you an easy way to determine Predicted Touches from regular ones.

And for the updates, we also have the estimated properties expecting updates. This is also of type UITouch properties, and currently it's only force. It might be in the future also as an azimuth altitude, but, currently, it's only the force value, and if that is set, we will send you updates in the future, and we will do so in our new responder called touchesEstimated PropertiesUpdated, and we will do so after the fact. So we will send you touches when touches begin, and then send you updates later.

Let's walk through this. So what do you do? [Inaudible] touches began or moved, you check the estimatedProperties ExpectedUpdates on your touch, and if that is not [inaudible], you use the property estimation updated [inaudible] on the UITouch, which is [inaudible] and is only set for touches that either expect updates or represent an update, and use that to store your thing you want to update, your current touch sample in a dictionary so you can look it up later. In the touchesEstimated PropertiesUpdated, then you look up your sample, and using the estimation updates of the touch, you get an update just the values that were expecting updates.

Note, though, that some of the updates will arrive after touch has ended. That's a life cycle thing you have to be aware. If you don't keep your data structure around after touch has ended, you will see estimated force at the end of your stroke, and it will look weird.

So let's go with that in code. So we have touchesEstimated PropertiesUpdated, and we go through our touches. We look up the estimation index because we are in this method. We can just implicitly unwrap it. We find our sample in the sample index. We update the sample, and as I told you, we update only the values in that method that we're expecting updates before. And then we update our stroke, and to be future-proof, we check if this touch still expects updates, and only if it doesn't we remove it from our set. So let's see this in action.

So I tried to use the azimuth angle to do a calligraphy pen simulation. And now that we have all the data we want, let's look at this without the debugging mode. Doesn't that look nice? And that's literally just me connecting all the dots we got from the hardware. That is not interpolating anything or doing something fancy like your drawing engines [inaudible].

So with that, let's add some finishing touches to the final app. Up to now, we just drew a whole screen full. We don't want to be restricted to that. So let's support our arbitrary canvas sizes. For that, we include our stroke view into a container view, add a little bit of shadow, and put them that into a [inaudible] draw view, and we're done, right.

Not quite. So now we have to think about how to handle our gestures because the [inaudible] recognizes now conflicts with our stroke gesture recognizer. If we change nothing, then we will always draw a stroke, and we can never scroll. This is not what we want. So one way around this is to disable scrolling with Apple Pencil.

This would be useful for something like an annotation app where you would always like to just annotate with the Pencil and specifically disable that. You can do that because we added allow touch types to UIGestureRecognizer. UITouch types is an array of NSNumber of the touch types, and it defaults to all of the touch types.

So what we are going to do here in this example is we would get the pan GestureRecognizer from our scroll view, we will set the allow touch types to only allow direct touches so it only reacts to fingers, and we will change our stroke recognizer to only allow stylus.

So this obviously is not the full picture. In the sample code, you can see one implementation but switches dynamically depending on the usage and is a little bit more complicated, but it illustrates the point that you can restrict your touch handling to either the Pencil or regular touches.

And one final note on this. There's also a new property on UIGestureRecognizer that is called requiresExclusiveTouchType, and although our gesture recognizers are defaulting to all the touch types, if they see one touch and begin to recognize, they are stuck in that touch type. That is so you don't accidentally pinch with a Pencil and a finger.

This is the regularly what you want from a UIGestureRecognizer. If you don't want this, you set the requiresExclusiveTouchTest, type to force, and then you can do recognition between fingers and the Pencil. So to summarize, I showed you all the new properties of UITouch so you can make out of Apple Pencil.

I showed you how to use the Coalesce and Predicted Touches to both have the richest data available to your drawing and to have the least latency. I told you about property estimation, why we do it, and how you actually update your data to get to the full rich data the Pencil provides, and I showed you how to adjust gestures to either react to Pencil or finger only.

And this is a screenshot of the sample app that is available that does the nice calligraphy pen as a default, and more interesting to you, it also has the debug mode you have seen the videos made of them. So you can see how your Coalesce Touches behave, how they are estimated at the edges, and see the tilt and azimuth, and just play around with it to see how all of our touch handling works. So the full information of this session is available at this URL.

We have controlling game input for Apple TV yesterday for Siri remote handling. A peek at 3D Touch will show you all the higher level interaction with the force in 3D Touch to a Peek and Pop-like experiences, and to get, to know more about touch-to-display latency, you should have a look at advanced touch input on iOS from last year. And with that, that's it. Thank you very much.