Video hosted by Apple at devstreaming-cdn.apple.com

Configure player

Close

WWDC Index does not host video files

If you have access to video files, you can configure a URL pattern to be used in a video player.

URL pattern

preview

Use any of these variables in your URL pattern, the pattern is stored in your browsers' local storage.

$id
ID of session: meet-with-apple-293
$eventId
ID of event: meet-with-apple
$eventContentId
ID of session without event part: 293
$eventShortId
Shortened ID of event: meet-with-apple
$year
Year of session: 2026
$extension
Extension of original filename: mp4
$filenameAlmostEvery
Filename from "(Almost) Every..." gist: ...

Meet with Apple • Session 293

Touch on Mac Group Lab

2026-11-19 • macOS • 1:02:14

Connect with Apple engineers and designers online to ask questions, get advice, and follow the discussion about Touch on Mac.

Open in Apple Developer site

Transcript

Hello, friends. Welcome to our second touch on Mac group lab. My name is Tomas, and I’m a technology evangelist at Apple. Touch on Mac opens up new ways for people to interact with your apps. Tonight, my colleagues and I are dedicating the next hour to your questions. A quick reminder on how to participate. If you’re watching on the Apple Developer website, submit your questions and vote for your favorites in the panel on your right.

If you’re joining us on YouTube and want to ask or upvote questions, please follow the link in the description below the video. We’ll be prioritizing the top-voted questions and topics that help the widest group of developers. Please keep in mind that we can only talk about features that are available today, and will not be able to discuss future roadmaps.

And if you’re running into bugs or other issues, please head over to the Developer Forums or file a feedback report in Feedback Assistant so our engineers can take a closer look. Once again, we have a fantastic crew working behind the scenes helping triage all your inbound questions tonight. Thank you, team.

Before we pull up the first question, let’s meet our panel. Why don’t we go down the table for a quick round of introductions? Please say your name, your team, what you worked on. - I’ll start with you. - Okay. Excellent. Well, hello, everyone. My name is Elliot. I’m part of the Mac product marketing team.

I’m Raleigh Ledet. I’m an AppKit engineer, event magician, resident Cajun, and I’ve brought up touch on AppKit and all the common controls and things like that. I’m Elizabeth. I worked on touch support for Keynote, Pages, Numbers, and Freeform. I’m Wesley, and I worked on Mail. I’m Tim, and I helped bring up touch support in Safari and WebKit.

Thank you all so much for joining me tonight. Elliot, to get us up and rolling tonight, could you give us a quick summary of what’s new? Yeah, absolutely. So we recently introduced an all-new MacBook Pro, which features the incredible Pro XDR display, which you have right in front of you. So in addition to a stunning tandem OLED display that comes in an all-new lighter-weight design, we’re very excited to bring touch to the Mac on this new MacBook Pro for the first time.

How does touch enhance the experience of MacBook Pro? Definitely. So touch gives our pro users even a totally new dimension to interact with their content. It’s new, yet it’s totally familiar. We know a lot of our pro users are working across their iPhone, their iPad, and they can now use that same touch input now on their Mac.

So the touch experience is incredibly responsive and refined, thanks to some amazing innovations and work done by all of you in the hardware and the software side coming together to just deliver a fantastic touch experience. And we have some amazing examples of touch optimizations so far across apps, and we just can’t wait to see what else is to come. Thank you. We actually have a follow-up question from Western US. “What’s your favorite thing about building touch for Mac?”

So many things. When you’re trying to work on the Mac and you’ve been busy doing something and something pops up over here off to the side and you’re like, “Okay, where’s my cursor?” I need to go move around and find it and then finally go and interact with that thing that popped up on the side. Now I just reach out and I touch it, and it’s immediate. I can, you know, particularly notifications and stuff. I can dismiss them quickly and get back at hand without losing focus or context of what I’m working on.

It’s also just really satisfying to get to go build a new interface paradigm. You know, it’s familiar because of iOS, but just getting to go kind of take those lessons and bring it over and do it right and have it feel good is just really satisfying. So yeah, to me, it was really interesting to see how we could bring a macOS flavor of touch. We couldn’t exactly pull how iOS works. And so there was a couple of quirks there, but also some delightful moments where we kept some unique macOS-feeling elements to it.

I always just enjoy the delightful gestures that we have everywhere. But bringing them to the Mac, especially scrolling, pinch zoom, things like that, making them feel beautiful on the device we’re building the software on. That’s my favorite part. It’s been fun. How should a developer go about auditing their app’s UI to make sure it’s ready for touch? There’s a lot of things you can do. We can maybe split that one up a little bit. You can audit your app for things that cause mouse emulation. So these are things like mouseDown overrides, mouseDragged overrides, things of that nature. nextEventMatchingMask.

Exactly. Raleigh knows all the details on those, and his video will cover some of those details, but there’s a couple of other ones too. Maybe, Elizabeth, you have some? Yeah, I mean, if you’re using system control classes, collection views, list views, those sorts of things, you want to make sure that you’re using non-deprecated APIs, because oftentimes the deprecated APIs will force you into mouse emulation.

And the simplest and easiest way to get your app to just work more or less for free with touch for those controls is to make sure that you’ve properly modernized everything. That was something I had to do in a couple of places in our apps, and then everything just worked. It was beautiful.

Yeah. So one of the things I want to point out is, as you’re auditing and you find areas that are using mouse emulation, and as you start adding touch-native gesture recognizers to work in those areas, other things that seem to work correctly without you realizing it was using mouse emulation will start to fail as we back off from mouse emulation, as we see that you’re becoming touch native and we want to work that way.

  • So…
  • Thank you for bringing that up. Yeah. So as your app is coming up and becoming touch native, it might degrade a little bit during that process, because we’re giving you more control to be touch native and are excited to see what you’re going to be able to do with that, with your apps.

Yeah, we had a concrete example of that in Mail where we had a button in a table view, and we added touch support for it, and we accidentally broke the ability to double-click to open that cell in the table view. It’s because we just hadn’t configured the gesture recognizer, right?

So as you adopt it, make sure you also check with mouse and trackpad that things continue to work. And like Raleigh said, it may break a bit more in the interim before you fully adopt it. And did you all have to test your apps on the new MacBook Pro? Or did you also use Sidecar when you were testing? I did almost all of the development on Sidecar.

Truly, like, many months was pure Sidecar. It didn’t require the new machine for almost, you know, until the very end, basically. And yeah, it worked great. Thank you for sharing. We have kind of a follow-up question. “I use a lot of standard AppKit or SwiftUI controls. What works with touch automatically, and what do I need to check?”

Elizabeth said all of the standard controls should work automatically. So as long as you’re not subclassing a control and overriding mouseDown, for example, things like that, those should just work. The interesting thing there is we have gesture exclusivity, which allows you to only operate one control at a time. You know, we have an exception for scrolling, and we really want to move towards a situation where you can operate multiple sliders and buttons and controls all at the same time.

But we found that just turning that on globally would affect a lot of apps negatively, because they weren’t expecting two sliders to be putting out value changes at the same time. So we have gesture exclusivity, and we have API to be able to adopt into turning that off, so that you can then, as your app is ready for it, be able to have a much richer experience, being able to control multiple things at the same time with both touch and with mouse at the same time, kind of input.

And then, you know, another thing to watch out for is small UI, right? A lot of the standard controls, they do some beautiful expansions, sliders and steppers and so forth. But especially if you’re using SwiftUI to build your own things, smaller things are hard to hit. You know, they’re small, they’re hard to hit.

Your fingers are not as precise. And so those are things that I would also go think about in your app and look for. And try, see how it actually feels in practice with your hand, with other people’s hands. You know, the more different people you have trying things, the more likely you are to find things.

Everyone’s hands work a little bit differently. And so you want to make sure it works for all of your users and not just for yourself. But small things, I think, especially where you’re not getting the system control experience, is a thing you’d probably want to go spend some time.

You may not need to make the controls bigger. Yes. And I think the guidance in the videos is maybe adopt larger hit testing or use standard controls. But one of the beautiful things about macOS is your ability to have those really rich, dense UIs, because, you know, trackpad and mouse is still the primary input, right?

We want you to keep the dense UI nature of your applications. So you shouldn’t be changing your applications a lot to make them completely touch-sized and everything at the detriment of regular mouse and trackpad input. There’s still a lot of Macs out there that don’t have touch capability, right? And we’re always going to have the ability of attaching multiple monitors and things that aren’t necessarily going to have touch associated with them.

And the guidance here is touch targets should generally be 36 points. And if they’re standard controls that are close together, in most cases, AppKit will do a really good job of figuring out what you’re trying to do. If it’s your own canvas, that’s where you can look. You’re trying to grab small grab handles or something. You could expand them. There’s other techniques, like if you have a small button with a little image on it or something, and it’s really small, and even AppKit is not doing a good job.

Often with your layout, you can make the size of the button bigger, but the icon image itself is still the same size. So your UI hasn’t changed, but because it is technically a bigger button invisibly to the user, they’re able to hit it and interact with it much more easily with touch.

That’s a neat trick. I think we’ve got some great examples in Keynote and Pages, for example, of those expanded controls. Yeah, those we got, you know, we got for free. Other than going and hunting down mouseDown overrides here and there, whenever a control wasn’t expanding, I’d be like, “Who’s overriding it? Where is it?”

But yeah, no, those worked great. And then in other places, we had to go do kind of the expanded hit testing stuff that Raleigh was talking about. And we don’t always get to use, sadly, AppKit controls. And so sometimes we had to do it ourselves. But yeah, getting the controls to do the expansions was really cool to experience, honestly, the first few times. So speaking of mouseDown overrides on the standard controls, a quick plug for the NSControl events API that we’ve put out there. Yes. Yeah. That’s what I was going to ask you. How did you find that API?

That was great. We have a very complicated system for our, you know, our inspectors, and that we have relied for many years on the, you know, nextEvent, spinning-the-event-loop stuff. And so we basically went to Raleigh and said, “We can’t build this unless you give us these event APIs.” And he was like, “Oh yeah, here you go.”

And we were like, “Oh, amazing.” And so, although, since you guys spun the event loop yourselves when the touches went down, like, later, that actually fixed a lot of those concerns for us. But I still had to go hook into that in a couple places to deal with some corner cases with sliders in our case.

Well, you brought up a good technical tip that I can give developers. When there’s an active gesture recognizer in place, we spin the run loop with the tracking run loop mode. So if you install a timer or anything that needs to be delayed, make sure you add it to the tracking run loop mode so that it gets fired appropriately at the right time.

This tripped us up a few times. I was like, “Raleigh, why are my timers not firing? What’s going on?” He was like, “Oh, you have to do what he just said.” We’re great. That’s a good tip. Thank you for sharing. We have a question about context menus. “How do they work with touch, and where should I anchor them?”

You should use the menu API property on NSView or menu(for:). And if it’s a touch, for compatibility reasons, we’ll send you a fake mouse down event so that you can figure out the location. And if you return back a menu and we see that you override these methods on NSView, we will automatically install the correct gesture recognizers for you. We’ll handle both right mouse downs, Control-clicks for mouse, long press for touch, or even a Control-tap for a touch to bring up the context menus and display them on your behalf.

If you’re trying to do it manually and display your own pop-up menu, you have to install all those gesture recognizers yourself, and you have to manage the relationships. And so it’s more work on your part, which is where, again, like Elizabeth said, if you use the modern API and let us handle as much as we can for you, you get a whole bunch for free. And so where to anchor it?

We will notice that the initiating situation is from a touch, and we will try and shift the context menu over a little bit on your behalf. Yeah. In Mail, we had a couple of cases where we had to present our own context menus. I mean, the system does show context menus a little differently with touch. It makes the rows a little bigger, so they’re easier for you to select with your finger.

We also had to audit for locations or for code that was using the mouse location. And that’s sort of indirect. That’s what we were using as the anchor for presenting our menus. And we had a bug where the menu was appearing in an unexpected way, and we changed that by either using a gesture recognizer’s location for that input, or in some rare cases, we anchored it to the view that was, you know, the one that you’re presenting the menu in. In terms of menus in Mail, we also have a little flag menu when you go to flag an email with color.

And that was a custom palette that did not support touch very well. What it didn’t support was this experience that menus have, which is like a one-shot-type gesture where you can long press, drag your finger into the menu without releasing it, and then when you release your finger, it’ll activate that button.

Our custom palette didn’t support that, and the way I fixed it was I used standard AppKit menu items and a menu palette style. And then I also kind of made it work better for touch by moving the color label name above where the finger would be, because before, the name of the color would be below the selected flag, which looks great on mouse.

But then with your finger, you didn’t see the name, and so you couldn’t really tell which flag you were selecting, especially if you’re doing some that are similar colors. So I moved that up a little bit and made it look better. So it’s a minor adjustment just to try to prevent occluding things with your finger. But otherwise… Yeah.

Also, going back to what Raleigh was saying about, you know, if you implement menu or menu(for:), one thing to be careful with there is make sure those methods don’t have any side effects, because they will call them earlier to figure out if you are returning a menu. And then that’s just to set up the gestures. And then later, if it is appropriate to present the menu, then it gets called again, and then it gets presented. And so if you’re like, “Why are you calling menu? I didn’t want a menu yet.”

It’s because they’re trying to figure out if it may present a menu. And so make sure that you’re not doing any weird side effect stuff or anything like that in there. Yeah. Along those lines, another tip: you mentioned you were using the mouse location. You actually start looking for the global mouse location. Yeah. NSEvent.mouseLocation, the class property, or window’s mouse location in window. Right.

And if you’re using touch and things, you have a touch-native gesture recognizer, the mouse is wherever the mouse is, which isn’t going to be where the touch is. So many bugs from QA where they’re like, “It’s somewhere weird.” And I was like, “I bet it’s where your invisible mouse is.” Right. So that’s another good thing to look for, is to see where you’re asking for the mouse location on NSEvent directly and auditing that area of code where that’s going on. Thank you so much for covering that. It’s good to know.

Here’s a question from Parker. “How should an app tell whether the user is currently interacting with touch versus the mouse or trackpad? Is there an API for the last-used input modality? And can both be used at the same time?” If you turn off gesture exclusivity, which we would like you to do in your applications, then yes, you can operate with both touch and mouse at the same time, which is why finding out what the last input type is is not something we publish, because it could get confusing once you start doing multiple interactions at the same time. And that’s where it’s important to use gesture recognizers. Often it’s useful to just have a gesture recognizer that does work for both mouse and touch.

And sometimes if you really need to know the difference or you have slight behavior differences, you can install two pan gesture recognizers, set the buttonMask for one to left click and the allowedTouchTypes to none. And that is a mouse-only one. And now you know that that gesture recognizer’s action comes from a mouse. And the same thing with touch. You set the buttonMask to 0 and the allowedTouchTypes to direct, and any action from that gesture recognizer is going to be from a touch.

And also, like Raleigh was saying at the very beginning, you know, you’re doing something with your mouse, and an alert pops up, and you just dismiss it, and then you’re going back to what you were doing with your mouse. And so trying to change the modality of your app based on what you think the user, you know, based on what they did last, that may be bouncing around a lot, because we expect it to be kind of dynamic. And so it might not be the best. I’d only use that in extreme cases where you actually need it to disambiguate. Ideally, you do the same thing for all cases.

I don’t even mean, I mean that sometimes, you know, different testing and stuff like that. I just mean that, you know, when they were like, “I wonder what they did last?” And I’m like, “That might be a dangerous question to ask,” right? Not just because of the simultaneous usage, but even just from a, like, “Well, but now I’m going back to what I was doing.” And so yeah.

Well, I’m going to use this as a segue to, again, mention, like we mentioned last time, NSApplication.currentEvent is dangerous, especially in this new world of gesture recognizers and multiple inputs happening at the same time. When your gesture recognizer action fires, the event that started all of this is long gone, and you’re not going to get what you were expecting.

And so you really need to be passing along the location information or any other information from the gesture recognizers and getting that data there and passing it down to your call stack if needed, instead of just at some point going, “Well, I’ll just find out where the global mouse location is or find out what the global modifiers are.” Yeah.

We do that too much, and we have to remove a lot of those cases where if you hold the Option key, something different happens. And we were doing that like way down the tree, and that was no longer held true, usually because of timing. And so getting all that data up front in your gesture recognizer was useful for me, and not like when I’m going and writing to the pasteboard, because by then, the user’s already fully finished their gesture.

And you don’t have to pass through, like, “These are the modifier keys.” You can pass through, based on something like this, “Option key means that, you know, it should resize from the center or whatever,” and just pass that Bool down instead, which then also makes your code a lot easier to unit test, because you’re not relying on global state that’ll be changing on your system and all those sorts of things. Good point.

Elliot, I have a question for you from Lee Jason. “I want to ask why Apple developed a touchable computer. Is it because it’s convenient for users to use or something else?” Definitely. So I think as I mentioned earlier, I mean, I think we’re so, you know, excited about what touch can bring to the MacBook Pro experience. So, you know, pros are using their Macs in, you know, more places than ever.

Their workflows are more diverse than ever. And so touch, you know, just allows you, just gives you more, gives you more flexibility. It’s amazing. We’ve seen a lot of examples of how it opens up new capabilities in your workflow, being able to do things that maybe required inputs of other devices.

Testing an iOS app, for example. Now being able to do that directly from your MacBook Pro is an incredible value. So really, it’s giving pros that, again, that flexibility to work how they want. And again, being able, they’re used to this experience of being able to work, use touch across their iPhone, across their iPad.

Now they can use this new, you know, input now on Mac in tandem with the keyboard and trackpad, which just both just work fantastic together. I want to talk a little bit about that. Just another interesting place that I find I’m using touch almost naturally and unexpectedly, because as a programmer, I often keyboard warrior, move around the keyboard all the time. And so I’m not thinking about it.

But when I’m interacting with somebody else, it’s like, “Oh, come look at this.” And we’re looking at, or I’m over at you and I’m trying to look at your computer, and it’s like, “Oh yeah, well, you just need this thing over here out of the way.” I’m like, “Oh, it worked.” And it’s just one of those things that it just works, and you move on. But when you stop and think about it, it’s like, “Wait, I couldn’t do that before, and now I can.” It’s like, “Oh, that’s just so nice.” It’s so nice. It’s so fun.

I mean, I think in terms of, yes, totally, like, that collaboration that opens up, you know, the number of times. Yes, of course, a lot of times you’re just at your desk kind of sitting there, but so many situations, you are on your feet walking around, you’re on an airplane, you’re in your hotel. There’s so many situations that are just more dynamic than I think, you know, I’ve ever been in the past. I find myself personally in that situation almost all the time these days. Oh.

A couple of months back, my son was at his computer wanting to do something. And I’m like, “Oh, you just need to… Oh, that’s right. You don’t have access to the things I have access to.” That happens. All of us. I think it took us about 10 minutes to get used to it. And then we were all smudging our old laptops.

I use it on the bus. I find that the trackpad is actually slightly not ergonomic if it’s, like, that close to me on the bus. There’s not a lot of legroom. So using touch is really convenient, and the best thing ever is just like closing windows, flicking windows left and right, closing alerts.

Yeah, a lot of alerts and notifications that I just really swipe over. - Yeah. - It’s so natural. I’ve been working with developers who are adopting touch for their apps, and yes, I see that they start touching their screens minutes after they get used to. And I think you can tell, because at the Apple Store, I’ve seen a lot of smudged screens, so I don’t know if people were just pointing at them or did they actually try to touch it.

Not to mention that we’ve got an entire generation now who’s grown up in a world where touch is just status quo, right? You know, and I think that’s definitely true. A lot of people, you know, a phone or an iPad was their first device. They moved to Mac, and you kind of just expect that that same type of interaction works, right? - Yeah. - Totally.

I have a question from someone in Canada. “How easy is it to add custom gesture recognizers to an NSTableView? I want to do something special with touch. My first tests, like adding a tap gesture for single and double taps, seemed like they were ignored, and the touches went straight to the table view gesture recognizer instead of mine.”

I would have to look at your code in that case and see exactly where you’re putting them, but I want to caution a little bit on that, because if you’re just trying to get a tap or a double tap, there are delegate methods and action methods on NSTableView, and that’s the way you want to go. Let table view handle it, handle all those common cases.

So beyond that, if you really do need something completely custom, you can add it. But now you need to figure out, do I need to add it at the table view row or some child inside that, and that will give it a higher priority? Or do I really need it at the table view level? But now you have to manage the relationship with the other table view gesture recognizers.

In Mail, we have the thread expansion button that’s in the table view. And from what I can tell, it worked as expected with the gesture recognizer. You can single tap it, double tap it, it’ll just toggle closed, open. And I just have to make sure that double tap wasn’t opening the email in a new window.

And I believe it was just installing a gesture recognizer in our view that we were presenting in the table view. I’m guessing it’s because it’s such a child view, like Raleigh was saying, the lower-level views get higher priority for the gesture system. And it may be, again, purely… Maybe if they’re installing it on a higher-level view, the table view tap gestures might be winning. Maybe. It’s possible. Thank you.

I have a question for you, Tim. “My app uses WebView to display most of its content. What are some best practices that you keep in mind when adding touch support?” So as long as you’re using WKWebView or the WebView SwiftUI API, things will work great. Things will work best if you are using native web controls, sliders and things like that, buttons, and not trying to implement those yourself. Style them all you want, but use the native controls. And use pointer events, I guess. Yeah, don’t depend on mouse or touch events. The Pointer Events API abstracts across all kinds of input devices and works great on this device too.

Thank you. We are getting a few questions about testing without a Mac. We mentioned that earlier, but we want to… - Testing without that Mac or any Mac? - Without this Mac. Yeah, we talked about how Sidecar is like a very high-fidelity experience in terms of the behavior of your code. I don’t think I noticed a single case where there was a behavior that was different between Sidecar and with touch on Mac, so that was really useful.

I will say that the Mac itself was great for the performance aspect of it. Scrolling performance with touch was something that I improved in Mail when I had the real device, because I noticed with the 120 frame rate scrolling that it was a little jittery, and I was able to fix that with touch.

But you do need to be on the new betas for it to work. Latest beta, latest version of the OS on all the platforms, and you need to have the same iCloud account on the Mac and the iPad for Sidecar to work, and you can plug them in physically with a cable. Yeah, that could improve performance. The actual hardware was great.

That was where we really polished the experience. That’s sort of where we found all the ergonomic issues, where we were able to really squeeze out a lot of things that we had maybe ignored on Sidecar, because I usually had it just like on the table right here next to me, but with the Mac, touching it with a bigger canvas, I found some things, some ideas to improve things there.

And if you can, with your iPad, resist the urge to put it down on the table, and instead get something that keeps it a bit more vertical. That can help with it feeling more like your UI will actually feel in practice. And if you have the Mac and you really want to try touch and don’t want to use your trackpad, put some tape over your trackpad. And that’s what we did to resist the urge to use our trackpad and to force ourselves to use the screen. And we found a lot of small, small things doing that. That’s a good tip. Thank you for sharing.

We have a question similar to yesterday’s about third-party displays sending touch. “So for our enterprise kiosk on iPad and Mac serving 10,000 attendees, what APIs or gesture settings minimize touch input latency, prevent OS tap throttling, and maintain 120Hz ProMotion UI responsiveness during rapid check-ins?” Quick pop quiz. There was a little bit… Can you repeat that question? Because there was a… Yes. “For our enterprise kiosk on iPad and Mac serving 10,000 attendees, what APIs or gesture settings minimize touch input latency, prevent OS tap throttling, and maintain 120Hz ProMotion UI responsiveness during check-in?”

So as a kiosk, you’re probably going to want to turn off the system gestures, so you can go into System Settings for the trackpad for that. So because you don’t want somebody going up to a kiosk and swiping to another Space or swiping out to App Exposé or something.

And that will actually help a lot, because now we don’t have to have the latency of trying to figure out, is the user about to do a system gesture or multi-finger system gesture instead of a single gesture? So that will help some stuff out there. And then other than that, there’s the regular gesture recognizers. Use NSScrollView instead of trying to write your own scroll view. Write your own scroll view? Don’t do it, don’t do it.

Wesley, do you have to do anything specific for performance with Mail? I did notice that scrolling is a little different with touch than with a mouse, and I’m not sure exactly why. Maybe some smooth scrolling that happens with mouse and trackpad. So responsive scrolling. When we introduced that, that was actually processing the scroll wheel events on a background thread, which with gesture recognizers, we have to keep all of that main thread.

So if there’s a glitch in your scrolling performance with touch but not with trackpad, then that is likely a performance glitch that responsive scrolling was papering over, and you didn’t even realize it’s been there in your app for quite a while. And the way I fixed that was I took my phone and I did like a high-speed video just to see the frames as well.

That helped with seeing if it was tracking my finger, and then profiled it and tried to reduce all the code. We basically had some code for loading contact photos in an email. You can have that enabled where you can see, like, who sent you the email, their avatar.

We defer that to a different thread, essentially, and had that come in later. So we put in a placeholder contact photo until that photo was loaded. Before, we were actually loading that in main thread to avoid some flickering. But this actually allowed us to get that 120 frames per second.

So yeah, load images… Oh, all the typical scrolling performance best practices that we’ve had for a number of years now is preload what you can and move work away from the main thread. Yeah. Here’s a question for the whole panel. “Can you share an unexpected learning or story from building for touch for Mac? I would love to know about any delightful surprises or unexpected challenges.”

I’m going to let the team start off. Gosh, I mean, it was interesting kind of translating our iOS experience for where the UI is the same, like our content in our users’ documents and on our boards. You know, we already have a bunch of gestures there. And so getting those stood up on Mac and working, and then seeing what felt weird when it was, you know, vertical, or when, you know, you have a Bluetooth keyboard on iOS.

And we do support, you know, modifier keys and stuff like that. But just seeing what people were reaching for, that surprised… I was surprised by people really wanting to do touch with keyboard, where I was like, “But your mouse is right there. Why? Okay, fine.” You know, “Sure, we can do that too.” And so that was a surprise. And then they were delighted. So there’s your surprise and your delight. Yeah. Selection on macOS is different than on iOS. So you can just tap to select things, whereas on iOS, tapping will usually navigate.

So that’s one of those interesting differences. And that’s where people will be like holding the Command key or the Shift key to do a multi-select with the keyboard and with touch at the same time. We had to make sure that that worked correctly. It kind of did out of the box for us. But that’s something we tested.

One thing that delighted me, and I talked about one-shotting before, is like the traffic lights to close a window. You don’t have to, like, aim for the one that you want. You can aim for, like, the middle and then adjust while you’re touching it and then let go. And I find that my muscle memory is gone pretty fast to close windows without having to try to laser in or do the double tap. Sliders are very satisfying. The expanding sliders, very satisfying.

There was one thing that I found maybe not surprising, but certainly was troublesome, which is web content that notices or tries to notice that you’re using touch and then their mobile experience. And for us, that’s not what we want, right? We want the Mac experience on the Mac. So that caused us… Like request desktop, but you’re on desktop. Already, but you’re on… Yeah. Basically. Yeah. So if you’re a web developer and you have logic that’s like, “Oh, the user is using touch.

This must be a mobile device.” It’s just not true. It’s good to know. I mentioned this before, but on the last panel, but since I was involved very early on in working on the initial prototypes and stuff, and I would get into a habit of thinking, “Okay, we got it working this way. And I know that it works this way.” And then when somebody else would see it for the first time, it was really interesting to watch them try and do things.

And I’m like, “No, that’s not…” I was like, “Well, that’s how you’re trying to do it.” Now I got to go back and think about, you know, “This is the expectation. How do we get it to work?” Everyone’s hands are different. It’s so fascinating. Yeah, we had a lot of moments early on of drag and drop.

I worked a lot on drag and drop, and that was one where if you knew how to do it, you would watch people and be like, “No, no, no, you’re doing it wrong.” But then we had to take two steps back and be like, “Oh no, how do people intuitively do? How do they know that they’ve picked up the item?” And that’s why we added the lift animation.

“How do they know how to do a drag versus a context menu?” So we kind of do both at the same time. And you can kind of go into the context menu, or you can go through it and then continue your drag, or you go the other way, and you do your drag. Those were things that we had to figure out when we had our hands on it, and we were trying to use it as users.

So yeah, that was fun. And I got to influence the system a little bit, because Mail was kind of like, we had the highest bug count for a while, mostly because of drag and drop, because too many ways you can drag and drop an email in the app.

And it was kind of through that experience that we changed, like, the default AppKit drag and drop UI experience and the treatment that we do. That was because of that experience we had together. And let me take this as an opportunity to plug that for NSTableView and NSCollectionView. If you’re using the modern pasteboard writer API for drag and drop with those, you’ll automatically get the lift animation and interaction with the context menu that can be pulled up at the same time, and then you can go out of the context menu and continue the drag.

And if you’re not using that and you’re manually handling the drag or manually putting up a context menu on your own, if a context menu does come up and we see that you one-shot into the context menu, and you drag away or through the context menu, we will go ahead and restart the gesture recognition process and filter out the context menu, so that we will try and catch your drag operation instead. That blew my mind when you got that working. Yeah, it’s really neat.

It’s nicer when you can have the lift animation there at the same time. It gives more feedback to the user. And right now, if you are doing it yourself, you would have to install a simultaneous gesture recognizer early that fires off early just to show the lift animation.

And that’s if you’re not using collection view or table view’s built-in support. You have to manually do that yourself right now. And as previously discussed, you have to be a little careful installing gestures onto those views, or you might… It might not work. So yeah, there’s cases where you wouldn’t want the lift animation.

In Calendar, for example, the event’s size, its geometry sort of reflects how long the event is. So we didn’t want it to get bigger, because maybe a 45-minute event would look like a one-hour event as you’re dragging it. So there are ways to disable the lift animation if you have that.

But by default, table view now will take a snapshot of the entire row with whatever highlighting it has. So if it’s a selected row, it’ll be whatever the selection color is. It’ll look kind of like… The rule of thumb is whatever it looked like when you started the drag is what we want the drag image to look like in practice. And it’ll shrink as well. So when you leave your host view, that image will get smaller.

So you’re not dragging this giant image everywhere, which then obscures your drop target. And those are all the details that we worked on together. - Yeah. - Yeah. We spent a lot of time with HI and various app teams really fine-tuning what that drag interaction can look like when it’s with touch, and how to get the best experience out of that. You mentioned Calendar, where if you make the size of the event bigger, it can skew how long you think your meeting is.

And I think they solved that by using a drop shadow. They did. Yeah. And they also did, I think, when you tap it, the grab handle indicators show up. Even if you’re not resizing the event duration, they materialize. And that kind of gives you this, like, pseudo selection state indication, which helps with that. Yeah. Well, if I can segue in a little bit, I know a common question is hover, right? And this comes into hover.

We don’t have any support in AppKit that will magically turn on hover things. So it’s on you to add support for that. But the guidance there is tooltips are ephemeral kind of information that you would get from doing a mouseover and hover. That’s fine. You can leave that with the mouse.

But if it’s something that’s critical to use with the application, because you hover over something and it shows some controls or something like this, then that’s where you need to add either a tap to show the items for a few seconds and then make it disappear or something, or have a new menu item or another pop-up menu somewhere, some other kind of affordance in your application to be able to get to those controls and be able to interact with them with touch.

Thank you for bringing that up. And on that topic, on the web, of course, lots of websites expect hover too. And so we brought over from iPhone and iPad a long-standing behavior where we’ll try to determine if a tap will show something interesting on hover, and pause there and let the hover thing appear. And then you have to tap again to actually send the click. So we try to make hover work for you. - The same way as iOS. - The same way as iOS. Yes, exactly. Thank you.

At some point, we were discussing gesture recognizers and the possibility of a conflict. So I have a question about that, and “worry about taps firing when the user is meant to do something else, like a double tap or a long press. Is there a way to control recognizer ordering?” Yes. There’s… in all directions. Yes, there are a number of delegate methods that you can use to establish relationships.

By default, a… Well, here’s a common mistake I’ve seen. Let me put it this way. Some people will take a double tap and say, “Well, I have a tap and a double tap gesture recognizer. And so I want to make the tap wait for the double tap to fail before the tap fires.” Well, now you’ve introduced the delay. Every time you tap, you have to wait for the double tap to fire instead.

And we already have a built-in relationship for taps, which is actually in NSClickGestureRecognizer on macOS, that the single taps will fire, and if a double tap fires, then it’ll suppress the second tap. So you’ll get a tap. And then on the double tap, you’ll get the double tap gesture recognizer will fire.

And that way, just regular taps are fine. And that initial tap, you do a selection, and the double tap, you can do double tap action. And it is a much more responsive interaction, instead of having to wait for the double tap timeout. Yeah. We had a similar case where we oftentimes waited for mouse up to do something as a way to sort of eliminate, “Oh, maybe you’re dragging,” right?

And you didn’t do a mouse up. And removing some of those mouse ups actually improved like the feel of the system, because things were now happening as soon as you touched it. So we kind of had to rethink some of those and improve the experience for mouse as well while we’re at it.

One thing to think about with single versus double tap as well is you want to make sure that you design your app so that the experience of whatever happens on single tap makes sense to have happened by the time the double tap happens, right? Like, you don’t want completely unrelated things happening for single tap and double tap, because that will just feel weird, and there’s nothing you can really do about it.

Usually it’s like single tap is select. - Double tap is like open and… - Open or show menu or something like that. A button that does, like, two different things, one for single, one for double. Right. Because you just might miss the double tap window, right? - And now you’re like… - Oh, it just feels weird. - You’re just like, “What? What?” - Yeah.

We actually have both types of single taps. We have a single tap that fires immediately, and a single tap that waits for double tap. And the second one, this is mostly on iOS, honestly, although we have it for a couple of things on Mac as well. But it’s on iOS. It’s to show the edit menu, right? And you don’t want that to flash, and then on the double tap dismiss. That feels horrible.

And so you just have to suck it up and have a little delay before you decide if you’re going to show that menu or not. And so you have to be really deliberate about, like, “What are we willing to sacrifice time for?” Selection? Absolutely not. It gets selected immediately. But UI, those sorts of things, once you’re sure that the user is not going to do something else, then you can present those things. But that’s very… It takes a lot of careful management to get that to work properly.

So, Wesley, you mentioned something about drag and drop. And so here’s a question here about that. “So I have an app where when you drag a window near other windows, it will show where it can snap. With mouse and windowWillMove, I start a local event monitor watching for mouseMoved and mouseUp. With touch and user dragging a window, how can I actually know when they have stopped dragging it?”

So there is a touch version of that mouse monitor, but it’s only local, right? - Nope. - There’s no… - The only way to get touch events… - Is gestures. Gestures. Yeah. The event monitors don’t work for touches. Okay. The way around that is, I believe there’s delegate messages on NSWindow that says windowWillMove, windowDidMove. I’m not sure without being able to look it up offhand if there is something that denotes the exact end.

You should be able, theoretically, to install a gesture on your, like, root view on that window that can recognize simultaneously with other gestures, so that you can at least tell when the touch ends. So you might be able to finesse something there with the delegate callbacks combined with the gesture. Potentially, but you can’t monitor, like, other… I guess you can’t monitor touches outside of your window.

Correct. But even then, you can’t… Like a local event monitor, you can watch the events come in unless a tracking loop is involved, and you don’t see them during a tracking loop. There’s nothing like that for touch. The way to do that, like Elizabeth said, if you install a gesture recognizer and make it simultaneous on, like, your content view of your window, if it’s a click in the content view of the window with your finger, then your touch recognizer can fire simultaneously with whatever else is supposed to happen.

And then if you detected that the window started moving, “So now I’m going to actually look at that information from this gesture recognizer that was sitting there at the root level and running simultaneously, silently, with everything else.” That’s a pretty interesting technique, and it’s sort of an advanced one that we actually use in AppKit itself for a couple of situations.

Yeah, definitely delicate. Yeah. It’s also a similar technique we use for bridging for Mac Catalyst. Yeah. Elizabeth, this is more of a design question, but you may have had this experience. “What is the best way to communicate in UI a gesture availability, for example, like hold to slide in a text field or how to select a text field, things like that?”

So I mean, system standard controls will handle those, their own things there. But we did add a couple places where we wanted to show, you know, where you could scroll. And so we couldn’t just on touch down immediately do the thing, like you were talking about with mouse down or mouse up earlier.

And so we went and added some highlights to be like, “This is…” You know, we have small bits of UI where you’re not tapping on that UI necessarily, but near it, you know, to kind of, the label near a control or something like that, to interact with it. And so showing highlight on touch down, even with, again, simultaneously. Well, actually, you don’t have to recognize your gesture to do that. If you’re careful enough, you can kind of reset it on gesture reset so that it won’t preempt scroll if you start moving.

And then that way, you give that user some feedback that, like, “This is a thing that can happen if you let go here.” That can, you know, it should be subtle, obviously, because they may scroll or do something else. So with NSGestureRecognizer, there’s API that says your gesture recognizer is cancelable for scrolling.

  • Yeah.
  • And then we will allow that gesture recognizer to begin, but not exclude a scroll gesture recognizer from also recognizing later. So then you can get your highlight. And as soon as scrolling starts, we will then forcibly cancel your initial gesture recognizer, and you’ll get a canceled action. And then you can unhighlight, then you know to unhighlight.

Or it could be something other than scroll, right? You could have your drags, for example. - Or… - In Mail, we had, it’s sort of like something you can drag that is kind of like an Easter egg, I guess. You can drag a message header. You don’t need to. It’s sort of optional, but when you mouse over it, the mouse will turn into a hand.

And that’s sort of like, maybe, an old-fashioned way of telling people that you can pick up an item. So we don’t have that with touch, since you can’t, like, touch over it to then know that it’s draggable. But I did make it support touch. So if you knew about it because you’ve moused over it and seen the hand, it will work with touch. - Yeah. - It’s good to know. Speaking of selection and dragging, “How should custom text views handle text selection?”

You want to use the NSTextSelectionManager API. NSTextView and NSTextField use that. WebKit uses that. I think they’re using that in iWork. - Yeah. - And that will… automatically install all the appropriate gesture recognizers and… Install a view, and then that has the gestures on it. Right. Yeah. But we’ll help manage all of that.

And then you’ll have a consistent user interaction experience that you get in a standard AppKit text view, and the same experience that you’re getting in WebKit or, you know, iWork or anything like that. So, NSTextSelectionManager. And TextKit 1 does not… Well, it just uses mouse emulation if you’re still on TextKit 1, which you should not be. Mail maybe has a case of a TextKit 1 view.

But we determined it wasn’t… We didn’t have anything custom to drag or to select anyways. But you’d want to avoid that. Upgrade to modern APIs. That’s the message. Modern text ones. Here’s an interesting question. “Should I make my Mac app support both touch and mouse events? In other words, if I want to do something unique for touch, how should I think about making sure it also works with the mouse?”

This may be more useful in, like, when you’re, like, rotating 3D objects and you’re doing those advanced pro cases. I saw the demo of someone mixing music, and they were, like, editing some waveform or something. I’m not sure. I’m not a musician. Those seem super intuitive with touch, like that manipulation with pinch to zoom and stuff. I think with mouse, it was probably a question of, like, having to hold the Option key down or Command key or Shift key in order to, like, do a rotation instead of a pan.

And those paradigms probably apply. I mean, you know, we’ve been talking this whole time about, you know, you take your mouse behavior and you figure out how to make it work well for touch. This is just the same question in reverse, right? You almost want to be, like, putting touch aside, what is a good mouse experience for this feature, and build that.

And if it’s slightly different or meaningfully different than your touch experience, like, okay, as long as you can figure out how to make it intuitive for the user in either case, and, you know, give you hints and cursors and so on and so forth, you want the best experience for each. - And they don’t have to be the same. - Yeah, not exactly the same. But you want that functionality to be there on mouse, because someone will connect the mouse, they’ll use a display, they’ll go into clamshell mode.

Someone will connect their mouse on that laptop. Yeah. Let me give you an example of a case where they might be different, and that’s perfectly acceptable. We were working with a developer, and they had a sort of a canvas little area, and they had grab handles, and you could grab it and resize it, and grab handles were right along the edge.

And that was great for mouse, and it was fine, but it was also just called out for, since it was for resize, just pinch. And then you could use the handles to not resize but adjust the mask in that case. And so that was an example where you did have a slightly different touch experience than the mouse experience, but it was natural and intuitive.

Yeah. I mean, scrolling, right? Scrolling, zooming, all those things are different on mouse, and you just don’t even notice, because it’s so intuitive, right? And so make it feel good, and then make it feel good, and then figure out where those don’t really line up, and then figure out how to kind of get those to make more sense together.

Yeah. I would start off with, you know, sort of from a blank slate. I would try and see where can I use a gesture recognizer to do both the mouse and the touch interaction. And then, you know, test, like, “This doesn’t feel great in touch,” or, “It doesn’t feel great in mouse.”

And if I separate them, I can slightly change them and get a better experience. Or if it is like one of the cases I gave out where, well, the touch experience here is really crying out for something completely different, like using a pinch instead of grabbing some little grab handle, then go for that, you know?

Yeah, we had a developer we were working with earlier today, and they were exactly doing that. Once they added pinch and rotation, it changed how they were using the app. And they were super happy with how it looked and felt. Here’s another interesting question. “How can I tell if a menu bar item was tapped or clicked to show touch-optimized UI?” Menu bar items are now handled by the menu bar server, and it’s out of process.

Maybe you don’t need to know if it was. Well, they said they wanted to present, like, optimized, like, expand… Expand UI. Like… Okay. I don’t believe that is something we publicly publish. Yeah. And you should file feedback for that and ask for that. Yeah. Please file feedback using Feedback Assistant. But if you’re using a custom view, you might be able to see that you’re being asked to put it into a larger container space. Maybe something to try. Well, but I think they’re saying the menu action opens some UI that they want to open optimized for touch.

  • Okay.
  • So I think… I mean, so this poses an interesting design question. Yeah. I’m gonna put on my hat for a moment. The places where we have a different sort of UI, like we make menus a little bit bigger, or we pop up the small controls, we’re very particular about when and where we do that.

Very transient as well. Yeah. So a lot of transient nature stuff, right? Because we don’t want to go and have you change the design of your application to be all touch native, and now you have a huge… Yeah, yeah. You move the mouse, and then it goes back to something else. So I’m interpreting this question a little bit differently. If you choose a menu item, and then that opens up something else that’s touch native because you open that menu item with touch, that is a more permanent type of thing.

I would caution about doing that and really thinking about it. Now, if your menu item itself was “Open up the touch version of something, because that’s how I want to interact with it,” then I can open that with a mouse or a touch, and it should open up the same panel, right? We have seen some apps that do that, where they do have an option somewhere that says, “Oh, open…” You know, change the layout a little bit, because, “I’m going to primarily be using this with touch for right now, for some reason.”

And then that’s fine. And then you can go back to the standard layout for normal interactions. But these are specialized cases. And, you know, depending on how divergent your UI is for touch, you can also just make them tap on it once, and now, you know, now it’s expanded to a touch UI, and it’s a good experience, like our other system controls are. Right? So, you know, this is specifically, if I recall, asked about menus, right?

I’m saying you have the menu open, and then show the UI normally, right? And then a tap, and then expand. Right? Yeah. I mean, because that’s what you see with sliders and things. The small sliders, when they expand, they’re at the standard small size. They work great with mouse. And when we see that you’re interacting with a touch, at that point, we make them expanded and bigger.

And at that point, it’s a transient experience. And you can go back to interacting with your Mac with both a mouse or a touch, at any point in time, switch between them and have a seamless experience. And there isn’t a way to force system controls to show expanded. The only way to get them to expand is to tap on them. There’s no API to force it bigger, right? So, you know, it’d be weird if you had it for your custom UI and not system UI, right?

I have one more question. I’m thinking we may hit the top of the hour here. Wesley and Elizabeth, “Are there any scenarios in which adopting touch improves the usability of your app?” In Mail? The predominant use case that I think for touch that is very useful is being able to swipe your emails. If you have a lot of emails, if you have a lot of spam and you want to quickly archive them or mark them as read. In my case, it’s like all the code review emails that I have to go through every day.

That’s something that feels better as a user. Like, I don’t really use the trackpad that much personally. I’m more of a dedicated mouse user. I know with trackpad you can do it, but that’s something that, for me, I found was more delightful with touch. Resizing windows is another one where it just, it’s so obvious, because I’m doing something here, and I just want to resize the window.

That’s one that I think is a bit better. And you do have to check the performance of your system as you resize windows. A lot of apps are not super great about performance of resizing windows, so just check it out with touch and make sure it’s good there.

I mean, for us, it’s direct manipulation, right? You get to touch things. You get to move them around. I think you do get more control with a pinch zoom, right? You know, trackpad pinch zoom, you have the, like, percent granularity, but with touch, and you can, like, focus on your cursor, but just the ability to be like, “No, I want it here.” - Yeah. - You have canvases in Keynote.

  • Yeah, you get to…
  • You know, objects you can… Use while you’re kind of moving it around. “I want to see exactly this content.” You just have so much more control there. And yeah, we saw… I worked with the Maps team, and they actually, while they were implementing it for macOS, discovered a gesture that they didn’t support on iPad.

And it was because we were really caring about the different gestures, how to pinch and zoom and pan the map, that they actually added that gesture into iPad and improved that. So that was kind of a bonus. There have been fun situations here where I’m like, “I’m just going to also fix this on iOS while I’m in the area.”

That’s always a good thing. Let me give you a quick story. We’re already at the top of the hour, but while we were working on context menu support for a long touch, we really started running into the issue that, for those of you that know, Control-click and right-click to pull up a context menu work slightly differently, in that a Control-click only applies to the hit-tested view to pull up a context menu, but a right-click will walk up the responder chain and find something to display context menu from, which is another reason I’ve seen people override mouseDown, just to forward Control-clicks up the responder chain.

And so we now have it where Control-click will automatically forward itself up the responder chain and find a context menu to display if needed. So they are much more similar now to a right-click. And so we have that improvement. And we were able to… And we needed that for touch support, to be able to find the right level to put a gesture recognizer to put up a context menu for touch.

Fascinating. Thank you for sharing. So touch has improved mouse support in a number of little ways like that. But that’s a fun one. Yeah. I think we’re over. But I love the discussion. Thank you so much for sharing. That does bring us to the end of our questions for tonight.

Thank you, Elliot, Raleigh, Elizabeth, Wesley, Tim. Thank you so much for joining me tonight. You’re all awesome. Thank you. This was great. I want to give a big thanks as well to Ed, Allen, Paris, Travis, Doug, Brandon, Adam, Jimmy, Serenity, Casey, who are working behind the scenes to make this event possible. And of course, I want to thank all of you for tuning in from around the world. Thank you for bringing such great questions tonight.

If we didn’t get to your question, please bring it over to the Apple Developer Forums so we can keep the conversation going. Also, please check out the updated Human Interface Guidelines, the developer videos, and the documentation for touch on Mac. Thank you again for joining us, and have a good day. Thank you.