Hello, everyone. And welcome to designing for future hardware. I'm Jeffrey Traer Bernstein. With me are Matthaeus Krenn and Bill Lindmeier. Who is this session for? Perhaps you're creating a device that's connected to an application. That device could be, say, a dishwasher that you would like to control from your application.

Or maybe it's a drone you would like to control from an app. Or maybe a golf club that would send information about your swing back to an application. Perhaps you have an application and you would like to create a device for it. Maybe it's a game you would like to create a custom controller for or a messaging application. Or maybe a device to go with a dating app. Who knows what that would be.

You might also find this useful if you are designing for a device you just don't have yet like perhaps an Apple Watch or some fancy new thing we've released. Who are we? We are a team of inventors, designers, prototypers. We work across all of Apple's products and explore what new interfaces mean to our devices, operating systems and applications. We brought you exciting features such as Multitouch gestures, Force Touch and a Taptic Engine. This time last year we did Prototyping at WWDC 2014. Was anyone there for it?

Thank you - excellent. Thank you for joining us. Did anyone watch it online? Less people, all right. Well, it is not required for this session, but I suggest you check it out afterwards. I'll give you a brief synopsis right now. We unveiled a disruptive, revolutionary new application called Toast Modern that allowed you to browse for artisanal, overpriced toast on the streets of San Francisco, take photos of the toast you're eating, and share it with your friends and loved ones.

We used this application to explain why to Prototype. There are two important reasons. One, test ideas to save time and money building the right things for your products. Number two is to get new ideas to make the experience of your product even better. We also talked about how to Prototype.

Make fake apps, show people, and learn from their feedback, and do it over and over again until the experience of your product is incredible. As you might have imagined, the reception on the Internet was incredible. People said things like, "The workflow shown is something I may try." (Laughter)

The Prototyping video for WWDC is great. "Grab a plate of toast." " Takeaways: Number one, make fake apps; number two, um... missed that part." "I'm so proud of you." My mom. So today, you're all very fortunate to be here. We are going to unveil a brand new, revolutionary Toast Modern product that builds upon the success of Toast Modern, the application.

We are going to give you a peek behind the curtain, a little bit of how Apple and partners prototype the watch. We're going to create a device and an app that connects to it. We're going to show you along the way a few prototyping strategies for hardware and software. I would like to share our Toast Modern product today. Would you all like to see it?

Excellent. It's a revolutionary, disruptive, connected, social, big data-y Internet of Things thing and this is it.

Thank you. Thank you very much.

So we call this Toastal Service. It is the world's first social toaster. It allows you to send Toast messages to your friends and loved ones. Of course, it's available in three great colors: Silver, Space Grey and 18-Karat gold.

So we have an iPhone and we have a Toaster. Both have excellent qualities to recommend them. The iPhone has a great interface. It is connected to the Internet. And it's always with you. Our toaster can turn bread into toast. So we want to combine the capabilities of each. We want to blur the lines between hardware and software, create a perfect meld of the two. I can think of another product that has done the same. That is the Apple Watch.

Now, you might think that at Apple we would begin here with the fully formed Apple Watch springs from the brain of its designers and is carved out of aluminum, and sent to all of you. In reality, we actually begin by drawing the Watch on another device so we can experiment with the interface. This is a pretty well formed watch as it exists today.

We start with this, a rectangle, and we use this platform to experiment with the different things we might need to add to the Watch to make the experience great. We might add a Digital Crown that we draw on the screen and scroll over to understand how that works and experiment with different buttons. This is great, teaches us a ton about the interface of the Watch and the experience of using it.

But you can't strap an iPhone to your wrist and go outside and try it in context. So some of our app partners when they were developing for the Watch before Watches were available to them did some clever things. For example, this French Transit app. They created their own Apple Watch. It is more rustic but useful. They drew some pictures of their different screens.

And then slid them inside of the watch. This was important. This is a Transit app. I need to be able to see what it's like to run for the train I'm late for and see what train I'm going to catch. They have to try it in context. The German newspaper Bild printed out a picture of an Apple Watch and slid underneath various screens. It was important for them to test the legibility of their articles in the places where they expect people to read them.

We learned a few things about Prototyping hardware and software. We learned we can fake hardware on screens as with the Apple Watch. We can fake software with pictures as we explained last year with Toast Modern, and it is important to try it in context at the right size, in the right place as you saw with the paper Watch Prototypes.

Here is our prototyping cycle from last year: make fake things, show them to people and learn from their feedback. This year we are going to make fake hardware and software. Before we begin we are going to ask ourselves three important questions for each of those. What needs to be more real? What can be faked? That's nearly everything except for the one thing we are going to choose to be real, and where will it be used? And then we are going to go and show it to people.

Who do we show it to? The people you app is for. If it's not meant for your office mate, go find the people it's for. Where? In the place where people will use it. You might have to go outside. And remember, don't argue, defend, or dismiss. You're there to learn from people's impression of the Prototype you put together. After we've shown it to people, we learn from their feedback.

We ask ourselves three questions: Number one, what is working? Number two, what is not working? Number 3, what other ideas does this give us? Now we are going to go through the cycle three times today and show you a few different techniques. We are going to begin with pictures and animation. Then we are going to use a technique we call behind the curtain. After that we are going to create a lightweight interactive and connected prototype.

I would like to quickly recap Toastal Service for you guys. It's the world's first social toaster. Allows you to receive toast messages, we call them Toasties, and send toast messages, and we want that experience to be magical. To kick it off, I'd like to ask Matthaeus up to the stage.

Thanks, Jeff. Cool. So how are we going to get started with this? So first of all, let's look at our hardware and our app and ask ourselves the question that Jeff just mentioned. First of all, what do we make more real now? Well, I want us actually to start with the receiving and the Toasting part of the experience. So on the hardware, let's figure out a way how we can indicate that a new Toastie has been received. On the app we can look at new Toastie notifications and see what kind of information the app can display with it.

The next question is, what can be faked? And really here just go for as much as possible really. We are not going to build a Toaster. We are not going to look at how to turn bread into Toast. All we are going to do is put pictures on the device and simulate the hardware like that.

For the app again, everything we are going to take pictures for everything. We are not going to make anything really interactive and definitely not writing any code. And third, where will the product be used? For the Toaster, that's the kitchen. For the app it's anywhere or at least anywhere around the house for sure.

So let's start with looking at our Toaster. We said that we wanted to simulate hardware by putting pictures on a screen. That sounds awfully a lot like what Jeff just described with the Apple Watch on an iPhone. Luckily Apple also makes a great Toaster shaped product called the iPad. That is just the right thing for us here. To sketch out our Toaster, I'm going to use keynote today. For you really, whatever gets you started easily and simply, I'm going to use keynote today. Feel free to use whatever you can.

I'm going to try to keep it as simple as possible too. A Toaster is really just a box, some feet, a lever, maybe we will actually call this a digital lever. And a slot where you can put in a piece of bread. We also said that we wanted to indicate when a new Toastie has been received. We'll put an LED up there that can light up. And while we are in keynote, we are also going to add some bread to the Toaster. This is the Toaster toasting.

And eventually the Toastie pops back up and we can see what's on it. We can take this. We can open it in keynote on our iPad. And we have a Toaster. For our app we are going to do something similar. We are going to get started with a home screen, a screen shot of a home screen on the iPhone and put our app icon on it. We are going to take a screen shot and use it as an underlay for an application that we want to customize.

That's simple. For our app, we're going to take a screen shot of Toast Modern, the product that we unveiled last year. You can see in the top left corner there's a convenient open spot that is the perfect place for the link to our Toast inbox. I'll put a Toaster icon up there and a little notification badge.

On the Toaster inbox I'm going to use a slice of bread right now as a place holder because I don't want to take away what is actually on the Toastie. We don't want to take away the anticipation, that surprise moment when you are sitting waiting for your Toast to be toasted.

We definitely want to show who is this Toastie from? We will put a picture and name of the sender up there. And we are also going to add the date and time when it was sent. At this point I think it would be great if we can immediately also choose how dark we want our bread to be Toasted. We can do that easily with a slider here.

Now, again let's open that on our iPhone and see what it looks like on an actual device. This immediately gives us a really life like but completely fake app. Now we can look at how these two things work together in action. You can see that I'm just tapping on a screen to make the notification appear to open the app and to go into the inbox.

And on the Toaster, I can also -- you can first of all you see that the LED is already blinking. Then I can tap to insert some bread. I tap again to lower our digital lever. Then in a couple seconds later the Toastie will pop back out, and we're going to see what's on it.

That is not too bad for now. We can immediately take this and show it to our friends and ask them how they feel about this so we can learn from what they have to say. So what is working right now? Well, the LED seems to be really good indicator that there is a new Toastie, and the notification on the app helps us know that there is a new Toastie even when we are not sitting in the kitchen in full anticipation looking at our Toaster.

What is not working so well is, a lot of people mentioned to us what happens when you have more than one Toastie? That's true, our design doesn't account for that very well. So what can we do about this? Well, maybe we'll actually upgrade the LED on our Toaster and maybe our app can show a list of Toasties instead of just the last one.

So let's take it from the top here again. Let's look again at our hardware and at our app. And at this stage again let's ask ourselves what needs to be more real now. Well, we want our hardware to handle multiple Toasties and toast the right one. On the app we want to be able to browse Toasties and select the right one for us. What can we fake? Still we are going to fake as much as possible. There is not going to be a real connection between the two devices. Anything related to actual toasting we are not even going to touch that.

For the app sending data back and forth, again we are going to fake that and our UI is mostly still pictures and animations. Where will the products be used? Still the same as before. The Toaster in the kitchen and the app really anywhere. Let's look at our Toaster again.

We said we wanted to change something about the LED. So maybe we can use a dot matrix display here instead. We could indicate how many Toasties have been received. And instead of using hi res touch screen that we put on the Toaster here which would make our product complicated to produce and expensive to buy, we are not going to recreate capabilities that we have on the phone anyway. We'll keep the hardware simple and focus on the complicated stuff on the phone.

So we'll add additional Toasties to our inbox. Easy to swipe through so we can select which one we want to Toast. We will create a state where a different Toastie has been selected. We will also add a screen that shows up on the app when the Toaster is toasting. We can add a count down to the top so we know how long it will be until we enjoy our delicious piece of toast. Finally the Toastie is toasted and it's back on the list and we can see what's on our app as well.

Now we have an iPhone, now we have an iPhone running a keynote of a fake app and we have an iPad running a keynote of a fake Toaster. With those two together we can start playing more realistic scenario of our product experience. A notification comes in on both devices at the same time. We open the app. We go to our inbox and select a different Toastie. We add some bread to the Toaster, push down the digital lever and eventually the Toastie pops back out on both devices.

Now, that's pretty cool. But there is a fundamental problem with this unfortunately. When we test our product, we can't just put the devices next to each other in front of us and happily tap on both of them in a really orchestrated way to play this perfect scenario. What is more realistic is that eventually the Toaster is going to be in the kitchen and the app is going to be all over the place with you.

That's a fundamental part of the experience, right? You get a notification. You walk up to the Toaster. You do a couple things. It starts Toasting and you walk away to brush your teeth, something like that. Eventually the Toastie is done and you come back to the Toaster. If you don't capture that part of the experience, you're testing for something that is entirely unrealistic.

How can we make our two Prototypes talk to each other over a distance. Ideally how can we do that without putting any additional effort into it. I want to keep using the keynote prototypes we just built. It turns out our team has a really convenient technique for circumstances like that. I want to show that to you right now.

You can see this is still our keynote prototype on the phone. But now the notification actually comes in on its own. We can still tap to open the app and then go and select the different Toastie that we want to Toast. And on the Toaster, you can see that we can still tap to insert some bread. Now look at the app, what happens when we lower the lever. The app responds to it in realtime. When the Toastie is back finished Toasting, it appears on both devices at the same time.

That's pretty awesome, I think. But how do we do that? Especially considering that this is really just images on devices, and we didn't write a single line of code. Well, it turns out we have something that is a little bit cooler and a little bit faster and a little bit smarter than code.

We call this ...

We call this technique behind the curtain. It is essentially about a person pretending to be a computer. In our case, it's me behind the sofa, using two keynote remotes connected to the iPhone and iPad and I can supplement tapping on the Prototypes by remotely controlling the state of each device. When you do something on the app I can press on the remote and the Toaster reacts to it. We create a perfect simulation of this realistic experience and we didn't really have to do anything to achieve it.

You wouldn't believe actually how many people at Apple we tricked into thinking we have implemented some magical technology where really, as we were demoing our Prototype, some of our team is just kind of sitting in the corner controlling one of the Prototypes. And so again with this more realistic setup we can go to our friends and ask them for feedback.

And we ask them what's working. Well, selecting from many Toasties and then Toasting the right one, it's all done, all good. What is not working? Well, despite our incredible Prototyping efforts, the devices still don't feel really connected and responsive. What can we do about that? Maybe we can have the Toaster react in realtime to the input in the app and vice versa. To do that we are going to have to step up our Prototyping game a little bit. I'm going to ask Bill on the stage to show you guys how to do that.

Thank you, Matthaeus, that's awesome. Hello, everybody, I'm Bill. I get to spend my days experimenting with top-secret Apple hardware which is pretty awesome. So we would like to share a little bit about our process that might give you some ideas for designing for your own top-secret hardware or possibly a device that you just don't have yet.

So after a couple of iterations, we are feeling pretty good about the designs that Matthaeus created for us. However, with the keynote prototypes, it can only take us so far. We can tap through from state to state or fake that, but there is no true interactivity yet. For example, the digital lever. We can't drag that up and down, which is pretty critical.

So let's return to our three questions and decide how can we enhance the fidelity of these prototypes with a little bit of interactivity and a little bit of connectivity to make them that much better. What needs to be more real? On the hardware side we want to wire up the digital lever. We also want to connect the state of the Toaster to the app itself. On the app side let's allow our friends to browse the Toasties in the inbox. Also pick a Toastie to Toast.

So we like faking things. What can we keep fake? Let's not worry about servers, edge cases, turning bread into Toast. And on the app side, we are going to keep that message queue completely canned and also the Toasties themselves, we'll have them be images on both devices. We won't actually send them back and forth between the devices over the air. Okay. So where are we going to use these things? Toasters live in the kitchen, so we'll use the Toaster in the kitchen. The app can be used anywhere that maybe you're hungry for a snack. Maybe it's the first app you launch when you roll out of bed.

We have to ask ourselves, what is the best way of moving forward? Matthaeus said he wanted to connect the Toaster and the app. Maybe we start with a real Toaster, hack it together with a microcontroller. Maybe put a wireless radio on it, some LEDs to make that a nice little display. And this would probably work.

But we are not committed enough to this design to move forward in this direction because this is starting to feel very real. Maybe it's time to 3D print a model of our Toaster and start customizing Toasties with a laser cutter. Let's be honest, we are always looking for excuses to use the 3D printer and the laser cutter.

But it is actually really fun making this slide, but that is probably putting the cart in front of the horse a bit. Okay. So how about a virtual reality Toaster?

This will be extremely awesome and fun to use, but Toaster haptics are still about five years out, so we have to put it on hold. Seriously, all three examples have in common that they are distracting us from the task at hand which is designing the experience.

So to the extent that we do any engineering at this point, let's keep it to the engineering which enhances our design and lets us explore that a little bit further. That way when we actually do go to make this thing we know that we are making the right thing.

Okay. So with that in mind, for the purposes of our Prototype, what is a Toaster? It is not a bread heating device, but you knew that. It is a connected state machine. When I say that I just mean that it's a physical object with attributes like where the digital lever is, whether or not there's Toast in it, what message is toasting, and we want to know how do these attributes respond to user input. We can easily model this with software.

So we are going to take our existing keynote Prototypes, the app of the Toaster. We are going to port them to Swift. And when I say we are going to port them to Swift, I don't mean that we are going to make a fully functional, feature-rich Swift app.

It is a lot like the Prototypes in that there are screen shots and they animate based on user touches, but then we are going to sprinkle in just a little bit of extra code for features we want to explore further. Of course this doesn't have to be Swift. It can be another programming language you like. It can be another tool that allows you more interactivity. But we at Apple happen to like Swift so we are going to move forward with that.

Okay. So specifically let's define what we are going to implement. The Toast-a-Toastie flow. So for the first step we want to pick a Toastie from the inbox. Next we want to send the Toastie to the Toaster. Third, we want to Toast the selected Toastie and finally, we want to indicate that the Toastie is Toasted.

So here is our new interactive app prototype. What we've done is turned this carousel into a UI scroll view that allows our friends to easily paginate between the different messages. We have a UI slider at the bottom which allows us to pick the color of the Toast. We've also added a gesture recognizer to each slice of Toast so we can pull it down into the virtual Toaster and simulate that it's being sent. Let's see how that looks in action.

We launch our inbox. You can see as we swipe through the Toastie selected increases. And we can adjust the Toast darkness variable. And then we are going to drag the Toast down into the virtual Toaster which sets Toasting to true, kicks off an animation and our Toastie is presented. All right.

Moving on to the Toaster. We are going to add a gesture recognizer to the digital lever. When you pull it all the way down, it begins Toasting, set Toasting to true, begin a timer. We have also added a segmented control at the bottom that lets us easily pick the message we want to Toast without connecting it to the app, and finally the animation will present the Toast.

Let's take a look at that. We are selecting our Toast which loads the bread. You can see we have this nice animation now in the LED matrix. When we bring the lever all the way down to 1, it is Toasting set to true, kicks off the timer. And our Toast is presented.

So this is a critical stage. This technique of modeling hardware with software is super useful and we use it in our studio all the time. You can imagine that you are making a connected Fireplace and you want to integrate that with HomeKit. You can use a projector or that large screen TV that you've got. Put it where you expect the Fireplace to be in the den and you can simulate the controls right where they are going to be used.

Maybe you are creating the next great Transit app for the Watch but you don't have a Watch yet. Throw the interface on an iPod, bring the iPod down into the subway station and use it in context. Maybe you're making the next great drone. You can simulate the hardware controls on an iPhone, strap it to the paper prototype, and when you're moving it around, you can stream gyroscope and accelerometer data back as if it's the real hardware.

So we've got the interactive app, the interactive Toaster. Now we need to send data across. Pick the Toastie from the app and send physical states across from the Toaster back to the app. What is the best way to do this? The best way is whatever is easiest and fastest for you. We are not designing a network API.

We can let somebody else do that later. We are not worried about efficiency. If you're Prototyping in the browser, maybe you use Web sockets. If you've got that drone and the phone connected to it, you can stream the gyroscope data with O.S.C. Maybe you have a back end server and you can use HTTP requests.

I know that some of you have probably never programmed a networking layer before. So think creatively. You can use a service like if this then that to observe emails that you're sending back and forth between the devices and take action upon that. Maybe you're noticing whenever a new photo is added to a shared photo stream.

So we are going to use a Multipeer connectivity peer to peer connection. This is what our network is going to look like. We have two iPhones connected to an iPad and call our network Toast Talk. The reason we're using Multipeer connectivity, it is incredibly easy to set up.

It doesn't require a server and it allows an arbitrary number of devices to connect. So here is a bit of code. I'm going to fly right through because we are running short on time. We are going to create a session. We are going to name the network Toast Talk and take any data that comes in and convert it to a string.

We can choose devices with this handy class called the MC Browser View Controller. Super easy to set up. You pass the name into the constructtor and also pass in that session. You can present it on a button click or when the app launches. Then you just dismiss it when they've connected. What messages should we send back and forth? The app Prototype wants to tell the Toaster the Toasty number and the Toast darkness. The Toaster needs to send the lever position when Toasting began and when Toasting ended.

A little bit of code that does that. We are just going to convert our string to data and send that to the session. And this is an example of how we might send the letter position. We are going to say we've got our message which is lever position. Throw a carat in there and pass some data along with it. This is a totally arbitrary format but fast and simple. We will split the incoming messages on that carat and switch overall the message names and pass the data off to whatever function is appropriate.

This allows us this really cool dynamic relationship between our Toaster and our app. You can see as I drag the digital lever up and down, that's reflected in the app. That's pretty great. We've closed the loop. We have interactive apps and Toasters. The world is good. Let's give this to our friends, let them try it in context and see what they think.

So Toastal Service is shaping up and starting to feel real and we haven't done that much engineering. Let's learn from the feedback. What is working? Clearly we can pick messages from the inbox and send them to the Toaster. That's a fundamental part of this experience.

What is not working? Our friends told us it's a little odd that there's no visual feedback from the Toaster when you select the color from the color slider. Maybe in the next round of Prototyping iteration we adjust the color of the LEDs on that matrix to reflect that the data is being sent over and the color is being remembered. So with that, I would like to invite Jeff back up to the stage and he can wrap it up for us. Thank you.

Thank you, Bill. Thanks for taking so much time on that last sentence, all right. Quick recap for everybody. We Prototyped using pictures and animation so we could quickly see different inputs and outputs for the Toaster. We showed you the behind the curtain technique to see how the app and the device work together. Then we created a lightweight interactive and connective Prototype so we could try the entire end-to-end experience.

Along the way we learned a few things about our Toaster. We don't need a darkness setting on the Toaster where most Toasters have that. We need a number display on the Toaster. We need a sensor on the digital lever to communicate back to the app. We figured all of this out really, really fast.

Look at all the time and money we saved. Imagine that we had to build a real Toaster to learn all of these things and we got really cool ideas along the way. So all of that reminds me: There are two important reasons to Prototype. Number one is to test ideas to save time and money building the right things for your app. Number two is to get new ideas to make the experience of your product even better. How do we do it? We make fake things. We show people and learn from their feedback.

And we do it over and over again until our product is incredible. So here is what we would like you to go and do after this. If you are creating a thing connected to an app, for the thing, fake it on a screen. In context. For your app, fake it with pictures in context and for the connection between the two, just tap through both. Or use the behind the curtain technique or behind the sofa.

And maybe create a lightweight network prototype. If you just don't have the device, especially perhaps the next thing that we unveil, don't wait. Fake it on a screen, in context. Here is some stuff about more information. There are related sessions. Most importantly, tomorrow Designing with Animation. Some of our teammates at 3:30 in this very room. I think you'll find it very useful. Thank you very much, everybody!