Hey, everyone. Welcome to the AR design session. My name is Grant Paul, from the human interface team at Apple. And, I hope you've been having a great WWDC this week. So, for this session, we're going to start off by, I'm going to talk about how to design great AR apps and games with the user interfaces and interactions. And then, Omar's going to come up, and he's going to tell you all about making 3D models that look great in AR, and feel great in AR.

So, before we get to that, I want to quickly talk about the basics. So, if you're new to it, you might be wondering exactly what it is that we mean when we talk about AR. But, even if you're an AR expert, I want to talk about what we mean for this session. So, AR, of course, stands for augmented reality. And, let's break down what that means a little bit.

And, we can start with reality, because it's a little bit easier to get into. So, reality means is that AR deals with things in the real world. And, that's a little bit different than other things we might have done on our devices, that take place on the device, or they take place on the internet. But, with AR, things happen in the real world. They happen in the room around you, in the environment you're in, or in your place on the map. Where you are. But, the important thing is, it's a little bit different.

And the other part of augmented reality, is augmented. And, that can mean a few different things. So, augmented can mean -- it can mean augmenting what you know about the world. It can mean getting information that the device can understand about the world, and giving that to you.

It can mean placing virtual things out into the world, giving the virtual this physical context. And, it can mean taking something that is real, like your face, when you put an animoji on as a mask. Taking something that is real, and enhancing it, augmenting it. So, that's what we mean when we talk about AR, when we talk about augmented reality.

And, with that, I want to get to the first half of this session. I want to start talking about how to design interfaces, and design interactions for your AR apps and your AR games. And, the first part of that is to talk about how to get people into AR. How to help guide people into AR.

So, I'll show you how iOS 12's built-in AR experiences help guide people into AR. And then, how you can take those principles, and apply them in your own apps. Next, we'll talk about the different ways you can present content in AR. The different possibilities that ARKit opens up, and also some tips and tricks that are great no matter what kind of AR app you're making.

And finally, we'll talk about interactions in the world. It's a little bit different when you're building an AR app than when you're building a 2D app, for what kind of interactions make sense, and what kind of interactions you're going to want to use. So, we'll figure out what still works great in AR, and then, where we might need some new kinds of interactions.

But first, I want to get started with talking about how to get into AR. And, what I'm talking about here is after somebody's downloaded your app, after they've found the AR experience in the app, they've opened it up, what I'm talking about here is when ARKit needs to understand the world.

Because for every AR app, ARKit needs some level of understanding of the world in order to start the AR experience. In order to get it going. Because every AR app needs to -- needs that understanding in order to place its objects into the world, or show the user that information.

And, the way ARKit builds that understanding of the world, is by getting you to move your device, by having you move around. And, that's a little bit different from other places that you might have looked through a device, seen a camera preview, in the past. Like, for example, when you're taking a photo, you just need to point the device where you want to frame the shot. But, in AR, you really do need to start moving, looking at the same place from different positions, and looking at it from some different angles.

So, the trick here is to let people know what they need to do. Let them know how to move, that they need to move their device. And, the way to do that is to give them a fixed reference. Give them something to base that understanding off of. And, let's talk about that. Let's take a look at an example.

So, this is the game Euclidean Lands. And, what it's showing here, is the device moving around inside of a room. And, that's really great, because you can see here, without any text exactly what it is you need to do, that you need to move the device within that room, and just turning the device to look at a different angle isn't going to be enough. So, without any text, it's really clear from that fixed reference of the room, exactly what you need to do.

And, usually, most of the time, that is all you're going to need. ARKit is even faster to build that understanding of the world in iOS 12. So, in even more cases that is all you're going to need. You just need to start moving, and you're ready to go.

But, of course, there are some situations that aren't quite as well suited to AR. Maybe you're in a dark room, or it's really reflective, and it's not as easy to -- for ARKit to start building that understanding. So, it can take ARKit a little bit of time to get ready. It might not happen immediately.

And, in those situations, if you're still being told to move your device, you're still looking at something that says to you, move your device around, people might start to wonder, why is the app not working? Does the app not understand the movement that they're making? And, it might get -- start to get confusing. So, what you need is some kind of feedback that lets people know that they're not doing anything wrong. They're doing exactly what they should be doing to get into AR. And, they should just keep doing it. So, let's take a look at an example.

So, this is how iOS 12's built-in AR apps help guide people into AR. You can see the device moving over the fixed reference of that surface, by showing you that you need to move it. You can't just keep it in one place, or rotate it. And then, once you've started moving the device, the surface transitions into a cube.

And, that cube spins with the movement of the device, giving you real, connected, direct feedback that you're on the right path, and you're doing the right thing. And then, once ARKit has built that understanding, it's all ready, the cube spins away, and you're ready to go in your AR experience.

So, that's how the built-in apps in iOS 12 help guide people into AR. And, your apps should follow those same principles. They should help people know that they need to move their device, and give them that feedback that they're doing the right thing. But, your apps don't need to follow that same style. They don't need to look like line drawings. AR should feel like an integrated part of your app.

It should feel native to your app's style. And, it shouldn't feel like something that's attached on, or this flow is something that's added on after the fact. So, the important part is to help people down the right path, but it should feel like part of your app while you're doing that.

And, the last thing I want to talk about for helping get people into AR, is that it's important to balance your instructions, your guidance, how you're helping people get into AR with being really efficient when people already do know what to do. So, if somebody already knows what to do, they're just going to start moving their device right away.

They don't need any kind of instructions. They don't need anything to tell them to do that. So, don't make them sit through any kind of guidance. Since ARKit is so much faster, in a lot of cases, they'll start moving, ARKit will understand the world, and they'll be ready to go right away.

Alright. So, that's getting into AR. That's how you help guide people into your AR experiences. Now, let's talk about how to present your content in AR. The different ways that you can show things in AR, and tips and tricks for all kinds of AR apps and AR games.

And, I want to start with some of the more radical possibilities of things you can build in ARKit. Because not every AR experience has to look like looking through your device, seeing a camera preview of the world, and then placing objects out into space. There are other options as well.

So, one of those options is to build an AR experience entirely in two dimensions. You don't need to show a camera preview. You don't need any kind of 3D graphics. You're using that information that ARKit gets about the world to present a great AR experiences entirely in 2D. So, let's take a look at an example of that. This is the game "Rainbrow." And, in Rainbrow, the way you control you character is by moving your eyebrows up and down.

So, there's no need of any kind of camera preview here. There's no need for any 3D graphics. It's a lot more fun entirely in 2D, but it's still a great AR experience. So, that was AR in 2D. But, another way, another thing you can do with ARKit is you can build things that I would consider to be full virtual reality experiences.

And, what I mean by that, is that the experiences take you to a different place. They really make you feel like you're somewhere else. You can walk around the environment. You can move around within it. You can look in all directions. And, to me that makes something really into a virtual reality experience, even if it is through a device.

And, there's some benefits to that. You don't need any extra equipment. It works everywhere. You don't need to wear a headset on your head. It works everywhere you are with the device you already have. You don't need any kind of trackers or anything else. And, there's some benefits to looking through a device rather than being fully immersed. Like, you're never going to accidentally walk into a wall because you always have your peripheral vision. So, ARKit can be a great way to make virtual reality experiences. We'll take a look at one.

So, this is the virtual reality experience called "Enter the Room." And, it was developed by the International Committee of the Red Cross. And, in this experience, once you walk into the room, you can look around in all directions. You can move closer to things. You can inspect them. You can move farther away. And, the sound comes in, and it feels like it's coming in from outside the room, from all around you. And, that makes it into a really powerful experience. And, that's what you can do with virtual reality using ARKit.

So, those are some of the more radical options of things that you can build with ARKit. We can also look at some tips and tricks that make sense no matter what kind of app it is that you're building. Whether it's a game, or a productivity app, these can work for everything.

So, the first thing I want to talk about here is showing text in AR. Because text is really important. All kinds of AR apps have different reasons to show text. If it's a game, maybe you want to show the title of a level, or some instructions. Or, maybe for other kinds of apps, you want to label something in the world, label a virtual object. Maybe show some annotations. But, the important part, no matter what reason it is that you're showing this text, is to keep that text readable. Make it really easy to read.

So, the simplest way to show text in AR, is to put it out into the world, to put it in perspective. And, you know, that can look really cool. But, it also has some drawbacks. And, one of those drawbacks is what happens when you look at it from an angle. The letters can, sort of, squish together. It can be a little hard to read.

And, another issue is what happens when you take a step back, when you're looking from further away. The text can get really small. It's like trying to read a piece of paper from all the way across the room. So, if you're showing titles, or other kinds of things that maybe people already know, or could get in another way, it can be really cool to show that in perspective. But, if you're showing something that people really need to read, need to get information from, you might want to try a different option.

So, a different option that you can use, is to show your text in screen space. And, what I mean by screen space, is that the text is always the same size. It's always facing you. It's always head-on. And, that makes it really easy to read. It doesn't have those problems of what angle you're looking at, or how far away from it you are.

But, the important part of showing text in screen space, is that it's still attached to a position in the world. It's still fixed to the place in the world that you attach it to an object, or attach it to some physical feature. And, that makes it really feel like part of the AR scene. So, screen space text is a great way to label things, put some annotations into AR, but still make them really readable.

So, here's an example of screen space text. And, this is from Measure, which is part of iOS 12. When Measure shows the measurements, it shows them in screen space. So, no matter what angle you're looking at those measurements from, or how far away you are, they're always incredibly readable.

So, screen space text is great for readability. But, you should still try and keep how much text it is that you're actually showing in AR to as little as possible. And, the reasons for that is when text is placed in the world, you always have to keep pointing your device at it in order to read it. If you turn your device to a more natural reading position, the text is going to go away.

So, if you have more detailed text, if you're showing details about some kind of object, or something in the world, you should show those details on the display. And then, people can use all that experience that they built up using iOS and reading things on their devices, to read those details directly on the display.

And, when you're showing those details, when you're coming out of AR, it's important to have a transition. Because those transitions can make it really clear what it is you're looking at. What text, what object it is that those details are referencing. What it is that you're getting details about. So, let's look at an example there. And, this is also from Measure.

In Measure, when you tap on a measurement, it comes out of AR to show the details about that measurement flat on the screen. And that's great because it's really easy to read. You don't have to point your device or your phone at the measurement in order to read it. But, it's also really clear, because of the transitions. The transitions show you what measurement it is that you're looking at details for. And, they comes out of the measurement, so you're never going to be confused.

So, transitions. They're really great when you're coming out of AR, and going back into AR, for showing details on the display. But, they're also really important when you're just showing objects in AR. And, that's because it makes it feel like there's one version of the object. It makes it feel like that object has its own identity.

And, that's important, because things in AR are so physical. They have that physical sense. When you see them in AR, they look real. And, things in reality, you can't just copy them. You can't just make multiple copies. So, it's important to keep that same principle when you're showing objects in AR.

So, this is what happens when you Quicklook at objects in AR. And, it shows a great example of keeping that identity. When you switch from the object tab to the AR tab, the object stays in place. It always stays visible. It doesn't disappear and then come from somewhere else. And, even when you're deciding where to place the object here, it always stays visible on the display. So, it's really easy to see that there's one version of this object.

And, even when you go back into the app that you were Quicklooking the object from, it still shows the object transitioning back to where it came from. So, it feels like there's one object moving between different parts of your app. And then, into the world, and back out of the world. It doesn't feel like there's multiple copies.

Alright. So, that was a bunch of information in a row about the different ways that you can present your content in AR. So, let's do a quick recap of that. So, first, we looked at the different ways you can make AR experiences, and we talked about using AR to create entirely 2D experiences, with no camera preview, and no 3D graphics. We talked about how VR can be a great way to build experiences with ARKit, and make you feel like you're somewhere else, that can be really powerful and really immersive.

We talked about using screen space to show text in AR, to make it really readable from any angle, and readable from any distance. We talked about showing your details on the screen, so that they're easy to read without pointing your device at some specific place in the world.

And that you can use all of that same knowledge that you've built up reading things on iOS. And finally, we talked about transitioning into AR, and out of AR, for showing details flat on the display. But, also to give objects that sense of identity, that sense of physicality that's so important in AR.

Alright. So, that's presenting content in AR. So, different ways that you can present it. And some tips for different types of content that you're going to want to show in your AR apps. Now, let's talk about how we can interact with that content. Let's talk about interactions that make sense in the world.

And, let's start with touch, because touch has been really important all the way back to the beginning of iOS. Multi-touch was there right from the start, and it's been the most important way to interact with our devices. And the reason touch is so important, the reason touch is so great, is that it enables direct manipulation. And, direct manipulation is when you interact directly with things on the screen, like they're physical objects. You're not using controls to scroll or to pinch to zoom. You're interacting directly with the content. It's like there's a physical thing that you're manipulating.

And, that's even more important in AR. Because in AR, as I said before, things are really physical. Objects feel like they're real. They feel like part of the real world. So, it's really important to use direct manipulation to make it feel like you're interacting directly with those objects. And, direct manipulation is also great because it uses gestures that you already know, that you have experience with from iOS. Because those gestures will be the same as any other content on iOS. They're things that you've been using for probably a long time.

So, the first one there is how you can move objects in AR with direct manipulation. If you want to move objects, you just put your finger down and drag them to a new place. And, it feels like you're picking up the object, because it stays under your finger. You get this physical connection to the thing that you're moving.

Another gesture you can do, is scaling objects. So, in AR, things start out at their physical size, at their natural size. But, if you want to change that, you can pinch the object to make it bigger. And, you can pinch out on the object if you want to make it smaller.

The important things to think about when you're scaling objects in AR, is to give some feedback when you're doing it. Because the change is really, really big when you take an object and scale it up to 4 times the size. So, it's important to give people feedback, so they absolutely know what happened.

And, the second thing is to make it really easy to go -- to take the object back to its natural size. Back to the size that it would be in the physical world. So, you can snap back to 100%, maybe with a haptic, to make that really easy.

Another thing you can do is rotate objects, by putting two fingers on the display, and twisting to rotate. And, that can be really great, but with all of these two finger gestures, another thing you should think about is your tap targets. Because things in AR are always moving as you're moving the device, and they can get really small if you get further away, or if you scale them down.

You should make sure to use really generous tap targets. So, it's easy to land two fingers on the object. And, be sure to use the center of those two finger touch positions to figure out which object to interact with. Because you might not be able to land two whole fingers, even on a generous tap target in AR.

Alright, so direct manipulation is really great in AR. It's really important because AR is so physical. But, it's also not quite enough for most AR apps. Because if you have a lot of objects, it can make it hard to touch the right one. As I said before, the objects are always moving on the display as you're looking at different places in the world. And, they're staying fixed to that place in the world. So, it can be a little hard to aim for those objects on the screen.

But, the fundamental reason, the number one reason, the real reason that touch is not enough for AR apps, is that it's fundamentally two-dimensional. The surface of the display is two-dimensional, and you're touching on that surface. That is what made it so great, multi-touch so great for flat, 2D apps in iOS. But, AR content, it's placed into the world. It's part of the world. So, that means we need some way to interact with that content also in three dimensions. Because the world is three-dimensional.

The answer there is moving your device. Because moving your device, it's natively three-dimensional. It's three-dimensional inherently. You can move up and down. You can move left and right. You can move forward and back. You can turn in all directions. You can stand up and move across the room if you want to go a little bit further. But, the important part is that moving your device is fully three-dimensional, and that really makes it the number one, the primary interaction in AR.

In fact, I would say that moving your device is more important than touch for AR apps. In fact, it's so natural, it's built in to every AR app by default. The way that you look at different things in AR is by moving your device to look at it from different angles, and different positions. So, it's really natural, and it's also really powerful.

And, moving your device, it accomplishes many of the things that you would have done in a 2D app by using multi-touch. So, in a 2D app, if you want to see more content, the way you do that is by scrolling on the display. You scroll down to see something new.

And, that's great in a 2D app, but in AR, you do that in 3D. You see different content, by moving your device to look at the content from different positions, and from different angles. So, it solves that same problem of wanting to see more, but it solves it fully in three dimensions.

Similarly, in a traditional 2D app, if you want to see something bigger, you pinch in on it to make it bigger. You pinch to zoom. If you want to see it smaller, you pinch out. You pinch to zoom out. But, in AR, if you want to see something bigger, you can just get closer to the thing you're looking at. You can just move closer in.

And, if you want to see more things at once, you want to see things from a wider angle, you can just take a step back, and look at all of the content at once. So, moving your device, it also replaces what you might have done by pinch to zoom to see more content in a 2D application.

So, movement is really great. It can replace some of those things you might have used multi-touch for. You can also use it to build totally custom interactions for your AR apps. And, those can be really natural. Just like using your device to look around at different things, or to get closer and further away. So, let's take a look at an example of that.

This is Swiftshot, the multiplayer AR sample game that you might have seen in the Keynote, or you might have actually tried it. In order to fire a slingshot in Swiftshot, the way you do it is you move close to that slingshot. You don't have to pick which slingshot you're going to fire from a list. You don't have to reach out and try and aim for it on the screen. You just move close to it, and when you want to fire the slingshot, you just pull back and release.

So, it's incredibly precise, because you can fire in three dimensions. You have three dimensions of precision while you're moving that slingshot, while you're pulling it back. And, that's more than you could ever do using touch. So, moving your device, it's not only really natural, but it's also more precise than you could ever do using touch in AR apps.

So, moving your device is great, but sometimes we need to combine the best of touch and the best of device movement to create the absolute best interactions. So, let's look at some examples of that. And, let's start with combining direct manipulation with moving your device. So, Quicklook in AR also offers a great example of combining device movement and direct manipulation in AR.

If you want to move an object, of course, you can just drag it to the new position, the same as you saw before. Touch on the screen and move it to a new place. But, you could also touch down to pick up the object and then turn the device and release in a new place. And, that's great, because it gives you the full three-dimensional control of moving your device to pick where you're going to place it. And, it lets you move objects to places that you can't see on the screen, by picking them up and turning.

But, it still keeps that sense of physical interaction. That sense of direct manipulation that you had from picking up the object directly. So, if you support any kind of moving objects in AR in your apps, you should definitely support picking up an object with direct manipulation, and moving the device to find a new place to put it.

Alright. So, moving your device and direct manipulation is one way to combine touch. But another way to combine touch with moving your device is through indirect controls. And, what I mean by indirect controls, are controls that are flat on the display. They're not placed in the world. They're not attached to anything. They're in a consistent position on the display, and you can learn that position.

And, that's really great, because that means they get out of the way. Once you've learned that position of that control on the display, it stays in one place. You can rest your finger above it while you're speeding your time focusing on the rest of the app. So, let's look at an example of that.

Here's Zombie Gunship AR. You're flying in a gunship above a horde of zombies, and you really need to aim at them. You want to focus on aiming. You don't want to focus on moving your finger, trying to figure out where you need to aim your finger on the display in order to fire. Instead, you can just rest your finger above the fire button, and spend all of your time using that full 3D precision of moving your device in order to aim where it is you're firing, to aim at those zombies.

So, indirect controls are really great combined with moving your device. But, they're also really great because they let apps work one-handed. In AR, no matter what, you always have to use one hand to control where it is that you're looking, at least one hand. So, if you want to build a one-handed AR experience, you're going to need to use really reachable controls that are really easy to access.

So, an example of that is Measure. Measure uses an indirect control, the plus button, the add button at the bottom of the screen to add points. And, that control is in a really reachable spot. So, even while you're using one hand to focus the reticle on the center of the screen, to precisely place your measurements, you can keep a finger placed above that plus button to easily add your measurements, to easily place those points. So, using indirect controls can be a great way to make it not only easy to use AR experiences, but one-handed ones as well.

Alright. So, that's how to pick interactions for AR. You can use direct manipulation to give that sense of physicality, and physical interaction. You can move the device, the primary interaction in AR. And, you can use really reachable indirect controls to focus on the content, rather than on controls or buttons interacting with it.

And, that's what I wanted to talk to you about today. Getting into AR, guiding people down the right path, and giving them that direct feedback that they're doing the right thing. The ways you can present your content in 2D and VR, and how you can show content to make it easy to read, and give objects that physical identity transitioning in and out of AR. And, the interactions that you can use in the world, especially and primarily moving your device. So, now I want to bring up Omar, who's going to talk about making your models, your 3D models look really great in AR. Thank you.

Thanks, Grant. Hey, everyone. I'm really excited to be up here to talk about some best practices that you need to keep in mind while you're developing your content for your AR experiences. So, we have a lot of different pieces of information to cover today. And, whether you're an engineer, designer, manager, or an artist, we wanted to provide you with a utility belt of tactics and definitions, so that you are best equipped to craft your own exceptional AR contents to delight people with.

So, to begin with, let's start with some essential concepts to keep in mind while you're developing your AR experience. AR is incredible. Having the ability to take anything you can imagine, and place it into the real world is absolutely magical. And, because of this, people tend to expect a lot out of their AR experiences.

They expect your 3D content to render at a smooth and consistent rate. It's incredibly distracting when you're really engaged with the content, and you start to move a bit closer to it, and then, you want to appreciate those fine details, and all of a sudden, wham! Poor optimization has caused performance to tank, and now it's as if you're watching a slideshow.

So, to ensure your -- the most smoothest performance at all times, and to keep people fully engaged with your AR scene, your app needs to render at a recommended target of 60 frames per second. Now, it's really important to maintain this target throughout the entire experience. Really stress test your content. View it from every possible angle. You know, move in close, move back from it, and just make sure that the performance does not ever degrade.

You know, maybe someday in the future, batteries will run for days. But, today, make sure that your experience is able to have as minimal of an impact to battery life as possible. Don't give people the chance to blame your app for draining their battery. The more power you save, the more you want people to come back to your experience and try it again.

I mean, I don't know about you, but whenever I see a battery indicator in this stake, I seriously feel like I'm going to have a panic attack. And, we definitely don't want to have our AR experiences cause widespread battery chaos throughout the lands. Remember, only you can prevent excessive battery drain.

I like to look at AR as having the power to take anything you can imagine and to transport into the real world. People want to explore your content, so you definitely want to bring your A game. Take the time to craft those nuanced details into your 3D content. Build a cohesive story and style. And, remember that every little detail, every little touch is an opportunity to surprise people.

So, let's say we wanted to make an AR experience to be about an aquarium. Even in its most abstract form, I think I would be really hard-pressed to find anybody who would believe that this marshmallow blob represents a fish. On a positive note, though, our app will pretty much rock the performance numbers if this little guy's flopping around. So, let's try that again.

Ah, now this is much better. That is one properly dead-looking fish. See how it exhibits some nice details, that once it's running in AR will really want to entice people to move closer to it to see all those nuanced details and explore its features. We should strive to maintain this level of quality with all the fishes swimming in our aquarium, or just floating in the top in this case.

Finally, it's important to remember that people really want to use your app in a wide variety of environments. You want to avoid having your content stand out in real world locations where potentially the lighting ambient conditions could conflict with the story that you're trying to tell. So, when working with your assets, try to avoid using colors that are either too bright or too dark.

And, make sure that you light your AR scene in a way that it casts even lighting on all the objects that you're planning to render, and at no matter what angle you view them from. You want your AR content to work whether it's day or night. Now, spoiler alert, we're going to go over some really nice features in ARKit that will enable you to really delight people when they see your AR content blend and react seamlessly to the real world environment.

So, as you're building your AR content, one great tool to help evaluate your progress is by using one of our recent announced iOS 12 features, AR Quicklook. Throw some of your assets up on iCloud drive, view them using the Files app on iOS, and quickly have the ability to project it into AR.

Heck, you can even show off your masterpiece by throwing it onto a website so that your friends can look it up, and you can view it anywhere, right from Safari. It's pretty awesome. Definitely go back and check out the session with Dave and David, who go over details of best practices of how you use AR Quicklook, and I'm guarantee you, it'll probably change your life with how you develop your assets.

So, now you've taken some time to consider what people expect from our AR experience, let's quickly plan out what type of app we're going to build today. You know, it's always a good idea to start thinking this through before you begin creating your 3D content. As knowing what you want to make will help narrow down how best to optimize your content and your assets for AR.

So, you're sitting at your desk, and suddenly you're struck by inspiration. You just came up with the most absolute brilliant AR experience ever. Alright? So, let's step back and ask ourselves a couple questions first. Does this experience really need to render hundreds of AR objects, or will we focus on a single hero asset? How much detail do we actually want? And, what graphical style best represents what we're trying to convey? Have we really paid attention to Grant earlier and are thinking about the level of interaction we want people to have with our experience? Having a clear answer to questions like these will help determine where best to put your rendering budget when you're developing your app.

For example, imagine you're building an AR experience similar to the IKEA Place app, where people can preview different pieces of furniture by being able to place them in their home, or in our case, outside on their patio. Now, the stars of the show are actually the different furniture pieces, so you need to present highly detailed objects that closely mirror the real world counterparts. In this case, it is a good idea to make sure that you spend a little extra time in your rendering budget on these single hero assets, because their level of quality can essentially make or break a sale.

On the other hand, you decide that you've had enough of accidentally stepping on those little, tiny, plastic bricks of agonizing pain that your kids start laying around the house. So, in order to preserve your sanity, you build an AR experience so they can play around with as many of these blocks as they can possibly imagine, similar to Playgrounds AR.

And, save yourself from ever having to experience brick pain again. In an app like this, where you potentially have a lot of objects being rendered and interacted with, you want to basically create very simple, low-poly models with a flat colorful material, so that you can have a ton of them onscreen, and still have really good performance.

So, now that we've asked ourselves these important questions, it's time to set up our AR canvas. Just like painters like to set up their canvas in space before they begin work, we would like to have a couple suggestions of how to set up your project up front, in order to position yourself for optimal success.

We are big fans of creating a focus square to determine where to start placing your AR content. And, if you're using SceneKit, right there on the bottom of the screen, you have the option to actually activate the statistics panel. This will allow you to see your current frames per second, as well as how many polygons there are visible on the screen at any given time.

Which should be very helpful as you start to build out your app and put all the different elements into it. So, now that we have a starter scene up and running, I was thinking, what will be a good example AR app to help get over these best practices? So, I'm not really an outdoorsy guy, but coming to California, I find that here a lot of people are. So, I've been trying to connect with nature.

You know, go camping, maybe make a campfire for once, and yet it really didn't happen. So, instead I thought, let's just throw it into an app, and see how it goes. And we're going to call it CampfiAR. I know, it's perfect, right? Now, we can work on building out a detailed, single object, and bring all the joys of being outdoors without the fear of any of those bugs or fresh air.

We decided to render with a stylized, semi-realistic, and playful graphical style. And, apply unique details to the use of careful application of some key, physically-based material properties. These choices mean that we could potentially use a lot of polygons to render the content on the screen, but why deprive people of the ability to spend multiple hours staring at our beautiful campfire? Let's avoid going down that route, and work towards optimizing our scene by using a few tricks of the trade.

So, we'll begin by focusing on the foundational structure of 3D objects, the mesh. And describe the typical development flow that will allow you to create highly detailed models, but still maintain a low poly count for all the models in your scene. And, for those of you who might not know, poly count is essentially the number of polygons, typically triangles, that a mesh is composed of.

So, one of the first things we like to do, is lay out the basic structure of an AR scene by using these simple meshes. We find using this type of white-boxing technique to be really helpful for testing out some basic interactions, as well as seeing how well the objects fit into the real world, what kind of scale are they at? You know, actually, I think this campfire looks really great.

I think we're going to call it a day here. Let's just call this done and ship it. Thanks everyone. I'm off to the afterparty, and -- wait. So, that didn't look like a campfire to you guys? Oh, alright. Sorry. Really? My bad. Let's jump back to actually building out this campfire mesh.

I want to give a quick high/low review of what a mesh is, and the basic data structures that comprise it. So, now you can think of a mesh as being a collection of triangles that are arranged in 3D space that will form a surface for you to apply materials on.

And, the corners of these triangles are actually made up of points called vertices, which hold different pieces of information, such as its position in space, UV coordinates for texture application, as well as a very important property that we'll go over later called the normals. Well, since I got busted for trying to ship early, I wanted to redeem myself by building out one of the most gorgeous campfires in the world. Take a look at the details of this camp. Look at that fish and those branches. You can see all the intricate details of the scales, as well as the etchings found on the bark.

But, man, my performance has tanked. And, that poly count has gone up to almost about a million polygons for this screen. So, I'm already in hot water, and I don't want to get in any more trouble, so we better go back and fix this. As I'm concerned about the impact that this will have on battery life, as well as how people are able to perceive and interact with this AR scene. So, let's see what we can do to, kind of, help reduce the number of polygons here.

Most 3D authoring tools have specific functions that make it easy for you to reduce the actual complexity of your models. Here, we've reduced the number of polygons associated with the high-density model of the fish. But, notice that, as we zoom in, many of the details have been lost.

But, don't fret. We can use certain material properties later on that will bring back a lot of those missing details. The key here is to build a solid foundational mesh that uses a minimal amount of polygons. So, let's put aside the high-density mesh for now. And, let's focus on building up this low-density mesh as we move forward.

Alright, so I'll admit that this isn't looking quite as good as before, but man, you can take a look at that performance. Not only have you saved a crazy amount of overhead by reducing the number of polygons found on the screen, but we're able to add a bunch of 3D objects to this scene to make it even more robust.

And, if you recall, in our previous high-density mesh, we were running at about 30 frames per second. But, now we're back to 60 frames. And, we were close to about, I think about a million polygons, and now it's down to 9,000. This is incredible. Because of this, we are well on our way to having those performance specification that we desire. A really solid frame rate with minimal impact to the battery life.

So, now that we have this optimized model in our campfire scene, let's see how we can bring back some of those details that we lost by working on what some of these different material properties and techniques that will ensure that our model looks as good as possible while still maintaining that great level of performance. So, you may have heard this time before, physically-based rendering thrown about regarding modern 3D rendering. It's a pretty complex topic that will take a lot more time than we actually have in this session to go over.

But, the basic concept describes the ability to take the application of your different material properties onto your mesh, in order to have it react realistically to the simulated lights found in your AR scene. And, moving forward, all the materials that we'll be discussing will conform to this shading technique.

If you want further details about this concept, there's a great WWDC 2016 talk that goes over details about physically-based rendering as it applies to SceneKit, called Advances in SceneKit Rendering. Now, with that said, let's start talking about our first material property, albedo, or what is sometimes lovingly referred to as the base or diffuse color property.

So, let's jump back into CampfiAR. Previously our base measure's looking a little bit boring, with the gray material associated with it, but after you apply albedo, you start to see that it looks a lot better. But, the campfire is still missing a lot of those small, finite details that were originally found in that high-density mesh. As you move closer to the campfire, you'll notice that all the surfaces are relatively flat. And, this is something that we'll definitely correct later, but first let's dive into the albedo property a little bit more.

Think of the albedo as basically being the base mesh of the various objects in your AR scene. This is the material property that you typically use to apply textures to the surface of your model. If you recall, your mesh contained different vertices, that held different pieces of information.

The ones you see here are actually called UV coordinates, which help map out how pixels from various texture maps are actually applied to the model. And, after we've worked with these textures, we've applied the abel material to this property on the fish. So, now we've essentially applied a texture to our fish, I want to remind you about the fact that you never know where people are never going to experience your app. You want to be able to have your content fit in as many different scenarios as possible. So, you want to take care in selecting the right albedo value that are neither too bright or dark. As, you want this to work in a wide variety of different situations.

So, our fish has got a skin, but we're still missing a lot of details from this, and the other objects found in the scene. So let's go back into how we can start bringing back a lot of those details through the use of the normal material property. So, as we jump back into CampfiAR, let's see how we can bring back some of those details that we moved from optimization. This can be done through the use of a special texture called a normal map, which you can see here as the blue tinted maps that are now applied to our AR scene.

These maps allow you to add those fine surface details back into your models, without the need to add any additional geometry. Now, after applying the normal maps, you can see how the fish exhibits some scales, as well as making the branches show just a tiny bit more detail.

Also if you take a look at the statistics panel, you'll notice that there was absolutely no change in the number of polygons related to this model. It's magical, isn't it? So, how do we make this normal map? Let's see a closer look at one of the branches, and see what we can do to make this work. In most modern 3D modeling applications, artists have the ability to generate these normal maps by projecting details from a high-density mesh over to a low-density one. So, here you can see what the normal map looks like on the branches after they've been generated from this high-density mesh.

And, after applying the normal map, you start to notice all those nice details that were originally lost come back into this model. But we still are able to maintain that high performance, of the low-poly mesh. So, you may be wondering why the normal map looks kind of strange, well the colors of the normal map actually represent visual representations of the vector data. And, determine how the normals on the surface model will be offset in order to change the way that light is being reflected, and our key to making this effect work.

That was a bit of a mouthful, so let's dive a little bit more into this property, because normals we feel are a really important topic. And, we want to spend a couple more minutes just going into how truly spectacular these can be. The art of manipulating normal vectors are one of the key tools that AR creators have in order to add a lot of the significant details back into their model. So, what the heck is a normal vector, and are there strange vectors as well? Well, no there's no strange vectors unless you forgot your high school trig, but normal vectors lie perpendicular to the surface of the mesh, and are associated with each mesh vertex.

So, why do we need these normals? Well, in order to see our object, you need to place simulated lights into your 3D engine. Normal vectors allow 3D engines to calculate how these lights are actually reflected off the surface of these materials. Similar to how light behaves in the real world, and are essential to making sure that your AR scenes mimic reality.

What's interesting is that by modifying these normals, you can trick the engine to thinking that your surface is actually more detailed than it really is, without need to add any additional geometry. If you take a look at this example, you can see a simple sphere, being rendered with flat shading.

What this means, that the normals associated with each face of the mesh are pointing in the exact same direction, as seen by the 2D diagram. Now, when you -- when light reacts to this surface, you'll be actually able to notice all the different polygons comprising this mesh due to being evenly lit across each face.

Here, though, we're using the exact same model, but leveraging a technique called smooth or phong shading. Notice that the normals are actually gradually changing as you move across the surface of the polygon. With the engine calculates the reflection off of this model, it'll give the impression of a smooth, curved surface due to the gradual interpolation of these normals.

What's really awesome is that these two models have the exact same number of polygons associated with them. But, through this normal manipulation, the object will seem to have a much more smoother and detailed surface, without the need to, again, add or change any of the geometry associated with this mesh. Whew. Alright, so we've gone through enough about normals. Let's, kind of, go over what it takes to add a little bit more shiny to your scene.

CampfiAR is definitely starting to look better. But, some of these normal maps [inaudible] but some of these parts seem a bit dull, especially the objects that you expect to be shiny or reflective, kind of like the kettle or the fish in this scene. What we see here is the results of applying a metal map to our AR scene. A metal map is used to determine which object surfaces should exhibit reflective properties.

Once the material property's activated, notice how shiny and reflective the area's we've designated as being metallic are. And, on the kettle as well as the scales of the fish. So, let's focus specifically on the kettle. We'll begin by taking the original albedo map, and then apply a metal map to the metalness property of this material.

After the application of the metal map, the 3D render will actually designate the surface to be reflective in the areas on the map that were actually white. And, despite begin called metalness, it doesn't necessarily mean that your object needs to contain metal. Rather, we're just letting the 3D engine know that this object should exhibit, kind of, a reflective surface.

Now, it's best to use the metalness map on your model, like our kettle here, when it has both a mixture of metallic and non-metallic surfaces. It's a simple grayscale map, where the level of metalness ranges from being black, for non-metal surfaces, to white, for metallic surfaces. And, it allows for a single material to be used to represent both reflective and non-reflective surfaces on your single object.

However, this kettle's a bit crazy reflective. And, doesn't really quite provide the look that we're hoping for. So, in this case, we want to potentially vary the amount of reflectivity, as well as simulate the fact that all surfaces are not perfectly smooth. And, they actually might exhibit slight small, microabrasions to their surface.

And, this is where the roughness material property comes into play. So, returning back to CampfiAR, you can see how the reflective surfaces are a bit too smooth. As we layer on the roughness maps, you can see that we are going to modify both the kettle and the fish to adjust the way that they're reflecting. And then, after we apply the roughness material property to both of these objects, you can clearly see the reduction of the reflectivity. This combination of roughness and metalness properties are another important concept to focus on. So, let's dive a little bit deeper into the roughness material property.

Use roughness to simulate micro surface details, which in turn will affect the way light is being bounced off that surface. If the roughness property is set to completely smooth, then light will bounce off of your surface like a mirror. As you increase the roughness of the material, light reflects over a wider range of angles.

Here, we're slowly scaling a constant roughness value between no roughness to max roughness on the kettle itself. And, this is a good way of being able to simulate the concept of having microsurfaces, and blurring reflection to the point where you might not see any reflectivity depending on which range you put your value to.

So, for the kettle, we've taken the original metal surface, and instead of just applying a constant roughness value to it, we actually applied a roughness map. And, this will help designate the surfaces where we will be scattering the light more often, and more than others. Once we've applied the roughness map, we get to see the real final look of the reflectiveness of this kettle, which is a lot less shiny as before.

Now, a combination of this metalness property and this roughness property really make your reflective AR models look phenomenal. Roughness can be used to tweak how much of your objects will reflect the environment. And, can really be used to add a lot more realism to your metallic surfaces. You can use this roughness map to add additional minor details as we've done here, to make our kettle look a little more scuffed up. Now, to close out materials, we have two more material properties that I want to discuss to further refine your models, and have a good balance between performance and aesthetics.

Ambient occlusion is a material property that is used to provide your model with self-shadowing, which in turn can lead to adding additional depth and details to your AR models. Now, while normal maps are great for applying significant amount of details back into your AR model, you can use ambient occlusion to really hammer in those details.

Here, we're visualizing the ambient occlusion maps for CampfiAR, and it's a bit of a difficult effect to demonstrate, as it's meant to be relatively subtle. But, see if you can notice the additional shadows on the logs, as well as certain areas near the bottom of the kettle. In our case here, it's kind of like playing Where's Waldo? with shadows, so let's focus in on the logs in the scene.

So, here we're showing you the normal maped logs before. Now, there's some great details found in the ridges, but we can definitely improve some of these areas. Now, as we look at the ambient occlusion map, you can see how we've added some regions of self-shadowing. So, lower portions of the log, as well as around the small embedded stumps. And, after we apply the map to our ambient occlusion property, I hope you can see the benefits of adding those baked detail shadows without the need of using expensive dynamic lights in your scene.

When working in AR, we recommend that you actually bake your ambient occlusion into a map. Which is what we've done for CampfiAR. Rather than use the alternative, such as screen space ambient occlusions, which is a camera-based post-process effect, and can potentially lead to poor rendering performance in your scene.

And, last but certainly not least, be frugal with the use of your transparency in your materials materials. If you must use transparencies, we recommend that you use separate materials for objects where you see a combination of transparent and non-transparent surfaces. In general, when working with AR content, the use of a lot of transparent surfaces can potentially have a huge impact on performance, especially if you are having transparent surfaces that are, kind of, stacked in front of each other when you're viewing them. This is known as overdraw, and it's something that you definitely want to avoid when you're working in AR. Whew. Alright.

I hope everybody's still with me, as that was quite a lot to go through. So far, we've focused mostly on how the AR content will react to the simulated spotlights found in our 3D engines. But now, it's time to focus on some ways to make our content seem like it's actually part of the real world.

A fantastic option to use to compensate for varied lighting conditions is to leverage one of ARKit's well known features, and light estimation. Let's begin by activating this functionality and see how it affects our kettle. Notice that how when the ambient light changes in intensity in the real world, a similar adjustment is made to the ambient light in our AR scene.

The way this works is that ARKit analyzes each frame of the video, and uses them to estimate the lighting condition of the real world. It is an absolutely magical feature that helps assure that the amount to light applied to your AR content matches what you see in the real world. Now, that we have magical light wizards living in our AR scene, let's discuss shadows.

Shadows in AR are really, really hard to get right. Your shadow needs to be able to work in a wide variety of situations. Remember that people can be potentially using your app anywhere. And, if your AR shadows differ from the ones that are seen in the real world environment, it can be relatively jarring for your experience.

Here, we have tried to incorrectly cast a dynamic shadow by using a directional light as a sharp angle in our 3D engine. Shadows are a great way to make objects actually feel grounded in the real world, but in this case, it doesn't really match the shadows being seen in the surrounding environment. It's like we're purposely trying to defy the laws of physics here.

Instead, we suggest that you place your directional light directly overhead, and play a little bit around with the intensity of that effect to make sure that it actually feels a little bit more subtle. This will allow your shadows to work in a lot more different situations, and a lot more different scenarios.

An alternative to this is actually using a method where you can create your own drop shadow, rather than using dynamic lights in your scene, which can get expensive, and can severely affect performance if you're rendering a lot of 3D content. Take the time to craft those really good shadows. And ensure that they will fit in as many real world situations as possible.

Ah, environment maps. If you really want to astound people, you definitely want to use these maps, especially on those AR objects that exhibit reflectivity. It'll make it seem like your AR content exists right here, in the real world. And, to make it super easy to leverage their powers we want to show you how one of the new features in iOS 12 and ARKit 2.0, automatic environmental mapping, can help you achieve this effect.

If you take a close look at the kettle, originally we're using a baked environment map that has a kind of blush tinge to it. Once we've activated the automatic environmental mapping, notice how the kettle now reflects a lot of the ground, and the surrounding color of the current environment that we're in. You can even see a little bit of the green from the grass that this kettle's sitting in. This is a fantastic feature that'll really help ground your objects into the real world, and with careful use of roughness, can really enhance the believability of your scene.

So, now why is automatic environmental mapping so actually incredible? Typically, these maps are used to simulate the ability of metallic surfaces to mirror the environment around them. You can see example of a cube environment map here. Now, before ARKit added automatic environmental mapping, you had to provide your own image, and hope that it was generic enough to work in a large variety of situations that your app may be used in.

But now, with ARKit 2.0, you can kiss those sleepless night, fretting about whether your environment map is actually ruining your AR experience. For more information about environmental mapping, and other new features found in ARKit 2.0, please check out the talk by Arsalan and Reinhard called What's New in ARKit 2.

And now, to close out CampfiAR, let's put all this together, and add the final touches to the campfire. With a little bit of animations and [inaudible] effects, along with the applications of all the techniques discussed today, CampfiAR is ready for primetime. Now, if I ever get the crazy urge to go outside, I can suppress those urges, and stay safely at my desk, enjoying the joys of simulated outdoors with CAmpfiAR. Who needs EpiPens? Not this guy.

We went through lot of different topics today. So, I quickly want to reiterate some of the important things to keep in mind while you're developing your app. Remember that your app can be used in a wide variety of real world conditions, so always make sure aesthetic choices allow your content to blend almost anywhere. Once you've decided what kind of AR experience you wish to build, adhere to what you think your rendering budget is, and work as many optimizations as you can to get that smooth performance and efficiently use your power.

And, finally leverage the use of various material properties, as well as the built-in features of ARKit to get your AR content looking great, in order to delight people who use your app. And, for your reference, here's a table of all the material properties we worked with today to build out CampfiAR. You can get additional information at the link provided. Thank you.