Hi, everyone and welcome. I'm Michael, here to talk today about Automated User Interface Testing with Instruments, or as I like to call it, "How to find bugs while you sleep." Now, I know a lot of you are developers, and you're building these great applications, and maybe you don't have a dedicated QA person.

You're trying to split your time between testing your application, actually developing your application, and testing the new features that you're implementing. So what I'm going to show you today is how you can turn a little bit of your time automating some of your tests and then get right back to developing those great new features.

Now, maybe you do have a dedicated QA person, or maybe you are that QA person. Well, I'm going to show you how you can automate your tests, kind of automate those tests that you do maybe every day, on every build, that kind of mundane stuff that you have to go through all the time, and then you can spend your time hunting down those edge cases and those more interesting scenarios. So let's get started. What am I going to talk about today? Well, first of all, why you should create automated tests.

And we'll talk about what is UI Automation? What is it that we have for you today? We're going to talk about how to automate an application; we are going through step by step by step how to build up a test case, how to actually create your first automated script; and, lastly, I'm going to cover some of the more advanced topics. These are the more interesting things that you can do with UI Automation. So why should you create automated tests? Well, you can find bugs while you sleep. One of the great things about automation is the system can be testing your application while you're doing something else.

You can be getting back to doing developing, writing those great new features for your application. You can be, maybe you're doing some other manual testing; digging in deeper into your application while the automation is running for you. You can create repeatable regression tests. So let's say you find a bug in your application, you write a script that covers that scenario, and then from that point on, you never need to worry about that bug coming back. You never need to be concerned that you've got to keep testing that because you've got a script now. That script is going to find that bug should it ever come back; something you never need to worry about again. You can quickly turn around updates to your application.

Now, users love getting updates to their application. Personally, I check the App Store every day just to see what new updates I have available. The trouble with pushing out an update for your application is that you have to test everything again because you don't want to introduce a new regression or a new bug that your user's going to run into.

So you want to make sure that you've maintained or exceeded that level of quality that you had previously. So the great thing with automation is that if you have a suite of automated tests, you can run through those with your new update, and you instantly know if you've maintained that level of quality.

So what is UI Automation? Well, User Interface Automation for the iPhone. Internally at Apple, we've been using UI Automation for a long time. This is how we do a lot of our testing. We're working through the User Interface in our applications, we're making sure that what the user sees is what we want them to see.

So we automate UIKit based applications. Now many of you are familiar with UIKit, you're using it to build your applications, and so really you don't need to do any more work for automation. It's just going to better work through your application. Fully touch based. Anything a user can do with a finger on a device, maybe Tap, Scroll, that kind of thing, you can automate through UI Automation. Support the iPhone, iPod touch, and iPhone Simulator -- one script runs in all three exactly the same. So this is any device that runs iOS 4.0 and supports multitasking, will run with UI Automation; one script exactly the same each time.

It's integrated into Instruments. So Instruments is an amazing tool. Maybe you've already been to some of the sessions on Instruments this week. So with Instruments you can monitor kind of your performance of your application, memory usage. And so we've added in the ability to automate your application. And what this means is that you can use the Automation Instrument alongside other instruments and see exactly what is happening in your application.

So maybe you've been trying to track down a memory leak, maybe you've had other performance issues with your application. You can now write an automated script that runs through your application with those other instruments, and you can pinpoint exactly what was going on when you saw, maybe, a memory spike or some other issue.

Fully accessibility based. Now, there's one key thing with accessibility that's going to make automating your application easier, and that's up here. The Accessibility Label. You set this in Interface Builder, you put a name in for your controls that you're building up, that is going to make your life when it comes to automation so much easier. It's not required, but it's going to make life easier because it will enable you to know exactly which control you're using when you're automating your application. And we'll show an example of this a little later.

So we use JavaScript to automate. And you're probably thinking, "Michael, why are you using JavaScript for automation?" Well, there's a couple of reasons. Automation has always traditionally been used with a scripting language. And so the reason behind this is that scripting language is very easy for anyone to pick up.

There's a lot of very talented QA engineers out there that maybe don't have any programming experience that you developers have, and so something like JavaScript is very easy for them to pick up and learn, yet it also has the flexibility and complexity for you developers to do more complex things, and we'll cover some of those a little later.

And lastly, with the JavaScript, it runs outside of your application. So the application that you're testing through your automation is exactly the same as what you'll submit to the store. You're not adding anything to your application in order to do the testing, so it's exactly the same code each time. So with that, I'm going to hand it over to Matt who's going to show us a quick demo of what UI Automation looks like.

So let's see how this looks like on a device. What we've got here is a piece of sample code it's available on the Developer's Site UI Catalog. We've gone ahead and written the script. You can see that we can identify buttons, navigate the navigation bar, click on things.

We can interact with text fields. So we're going to go ahead and set -- cap the field to make it First Responder and set the text. We can simulate various gestures such as Inch Open and Drags. Notice I'm not interacting with the device. Easy to do double taps. You can set the orientation, in case your app happens to run in landscape, you can do some testing there.

Show you a few more examples of it interacting with some controls, and then I'll show you exactly how you set one of these up in Instruments to run your script. Interacting with Switches here, Sliders. You can also simulate a Lock, and you can also do Backgrounding and Foregrounding of your app, so you can test the Multitask interface.

Okay. So how did I run that script? Let's go look at the desktop machine here, and I'm going to bring up a new Instrument session. You notice when you first launch Instruments, you'll be prevented -- presented with this template chooser. Under the iPhone templates here, you'll want to select the Automation template. Then you need to make sure you're targeting your iPhone.

So up here in the target chooser, select iPhone, and select your -- the application that you want to target. And then down here on the left there's a Choose Script. You just need to navigate to your script, and then start the Tracing session which will automatically start up the script. So you'll see as you run the automation, you can log the beginning of tests, and they show up in this Outline view. You can expand that and see more detailed logging inside.

We also have the ability to capture screenshots, so if there's something you're interested in onscreen while you're running your test, you see it over here on the right in the Extended Detail, it will display the screenshot that you took. And up at the top here in the timeline, we highlight green when you passed this test. If you were to fail a test, we'll highlight that red. So I'm going to hand this back to Michael, and he's going to tell you a little bit about how you write this script.

So now you've seen kind of an example script and how you can use that. So let's look at how we can actually automate an application. First thing I need to talk about is Elements. So every control in your application gets represented to UI Automation as an element, Now the base element is UIAElement. This is from what all other elements are inherited from. And there's a couple of properties on that that are really important to understand. There's a lot more than this, but these are the four that I really want to focus on today. First up, Name.

Now the name is what was set -- if you remember, we talked about accessibility, the Accessibility Label gets translated into the name of your element. And this is what makes it very easy for you to find individual controls in your application. The Value. As you'd expect, this is the value of a control. Say you had a text field, for example, any text that appeared in that field gets represented as the value. So you can very easily read maybe some text out of your application for verification purposes.

Elements. Well, you think of your application as having a hierarchy of controls. This gets represented as individual elements, so every element can contain zero or more other elements. Think, for example, a table here may have multiple cells as its child elements, and equally parent within one level back up. So let's look at the control hierarchy a little bit more. Here we have the rest of his sample from Developer website, and we're going to dig through this and see how we build up a line of code to access a particular element in your application.

So let's start off with the target application. This is the application that -- your application that you're testing. And we access that through UIATarget, localTarget, and the frontMost application. From there we move down to the mainWindow, which looks identical, it is, after all, the main window of the application. Move down from the mainWindow, we get to an individual view. In this case we have a tableView. Moving down from the tableView, we get down to an individual cell.

And you'll see how I've accessed this, we have an array of table views, we just chose the first one, so the Zero Table view. And again I've chosen the first cell which is highlighted here. Beyond the individual element, you get a child element, in this case, we're directly accessing "chocolate cake." So how do you go from that to actually building up a test case? Well, let's start with something really simple.

I'm going to start by tapping a button. Nearly every application uses a button at some point, you're going to want to tap one at some point during your testing. So we got a little code snippet up here that shows you how it's done. But what I want you to pay attention to is when I've worked through the hierarchy of the navigation bar, individual buttons, into the add button.

And so that's actually the plus button that you see at the top of the navigation bar, and the reason we can access it using the word "add" is because that's the Accessibility Label that we've set on that individual button. So we've made it easy for ourselves to go and be able to find that rather than having to see, oh, the zero button or the first button, you know.

Accessing it by name makes not only a more readable script, but it makes it much easier for you to write your script. So let's go ahead and tap that and see what happens. Well, we've moved to another screen. It looks like that worked. So what are we going to do next? Well, we've got a text input there, you probably want to type something into it.

We start off by declaring a local variable, in this case a bit of text with "Turtle Pie." It's a favorite recipe of Matt's, and he's promised to bring some into the office next week. So how do I get that text into that text field? Well, again, I work my way through the control hierarchy to the first text field.

There's only one there, so I don't really necessary access it by name. I can just go for the first one in our array and course setValue on it, and that's going to insert the text directly into -- into the display. So let's take a look at that. Sure enough we've managed to insert Turtle Pie.

Tabs are great in your application. It lets you display a lot more information users can navigate back and forth between. But how do you access individual tabs? How do you move between them in your application? Well, it turns out it's a lot like tapping a button. Let's have a look at this little code snippet here and see how it's done.

First of all, I'm declaring a local variable to access the tabBar directly so I don't have to write that huge long line of text every single time, I can just narrow it down to one element. Once I've got my tabBar, I want to know which tab is selected. So I'm accessing the tabBar, asking for the selected button, and getting the name of it.

Then I'm checking to see what tab is selected. Now, is it the Unit Conversion tab, and if it's not, we just tap that button, and let's see what happens. Well, we moved to the next tab as we expected. So let's move on in building up our test. Scrolling. You got a table in your application, you're already used to scrolling up and down to find individual elements, so let's see how we to that.

First of all, again, accessing localTarget, frontMost application, my mainWindow, and the individual table, and now I want to scroll to that Turtle Pie recipe we've added. So I'm using the function called scrollToElementWithPredicate, and what this enables you to do is scroll to an element that maybe you don't know the complete name of. This will come in handy if maybe you have some unexpected text, maybe some text that's come from the Internet, maybe a date or a time.

Something that you can't predict at the time you're writing your script. In this case, we could predict it, but let's have a look at how it works. So I create a predicate, namebeginswith "Turtle Pie," and that enables me to access that cell directly and scroll down to it.

So once you've worked through your application, it's great that you can tap individual elements, you can find stuff, you can scroll, but that doesn't make a great test case. For a test to actually work, you need some sort of verification. You need to actually verify that you've done something and that a test has passed or failed. So let's look at how we'd verify that we've added Turtle Pie to our recipe. This big chunk of code here.

Again, what I want to do is access the cell that contains Turtle Pie and determine if it's -- that -- if it's there or not. So first of all, I create a local reference to that cell, and I'm using this case, firstWithPredicate, and namebeginswith "Turtle Pie" as well. That will give me a reference to that cell. I then check if the cell is valid or not valid. And if it is valid, I go ahead and log a pass for my test; and if it isn't, then I go and log a fail.

And what cell.isValid is actually doing is checking to see if that cell exists or not. How do you go about logging information? Now, when you're writing your tests, you need to log as much information as you can because this is what's going to make it easy for you to diagnose any failures that occur. One of the most important things you need to log is when you start your test and when you end your test. And we've made this really simple.

All you need to do is call UIALogger, logStart and the name of your test, write your test, and then ultimately at the end call logPass, assuming your test is passing; otherwise logFail for a test that's failing. So it's very easy just to mark off in your code exactly when a test is starting and when it ends.

Of course, that's not everything. You really need to log as much as you can about what's going in your application. What we recommend is every time you interact with a control, log what you've done. That way when you go back to look at the log of your test, maybe when it fails, you can read through your log and see exactly the steps that have occurred. And that'll make it much easier to diagnose any failures that have happened.

But logging is as simple as just calling logMessage and a text string. But they say a picture is worth a thousand words, and sometimes text is just not going to be enough. So you can go ahead and log a screenshot. And, in fact, that's how we've captured all of the screenshots in this presentation.

All of the screenshots we've used were captured through UI Automation using code just like this. So you go ahead and call captureScreenshot, give it a name of the screenshot, and this gets saved back in Instruments as the file name. So, you know, a little more useful name than Screenshot 1 is probably a good idea; otherwise, you're never really going to know what your screenshots are until you look at them. This will make it easier for you to find them, do any post processing that you need to do, after the fact. So with that, we're going to hand it back over to Matt, who's going to show you how the script we've just written works in practice.

So we've gone ahead and written this recipe script. You can see, just as Michael showed you, clicks the add button in the navigation bar and sets the text. We've added some other tests that enter the description information and the prep time of this recipe, adding some ingredients, just give you more examples of the type of testing you can do.

So this test is coming along well, but you'll notice that we're -- left with the recipe of the night and it might be nice to implement a Delete test. So I'm going to go ahead and walk through implementing this Delete test. In order to delete this item, the first thing you're going to need to do is select the Edit button from the navigation bar. So we're going to have to isolate that, click that button.

Then we're going to choose the switch to enter the Delete mode and then click the Confirmation Delete button. I want to exit Edit Mode again by tapping on that done button in the navigation bar, and then, once again, this time we'll want to verify that the recipe's been removed. So here's the last run we had on this test.

You'll notice there's this Edit button next to the script. If you click that, that will go ahead and bring up that JavaScript in the editor that you have selected for JavaScript. In this case, we've -- we have Xcode set. And I'll just run you through this script quickly here.

We've gone ahead and saved a reference to the local target here in this variable target, and the frontMost app and this Variable app, and then we've implemented a few convenience functions here that help us get back to the main list view because we know we're going to be doing that a lot. Let's see, just finds different tab bar and navigation bars and it goes ahead and selects those to make sure we're on the right mode.

And you'll see at the bottom here, we're calling the various tests we've implemented -- this is fairly simple script, you may want to add more complexity to yours, but I wanted to give you a simple example. And here's the Add Recipe test that Michael's been working on. So let's go ahead and add that Delete Recipe test. We're going to start out by just sort of framing this function here, function Delete Recipe. It's going to take one argument, the name of the recipe, and we're going to log the start of the test.

And we're going to call this Return to Recipes continue function to get back to the list. And then, if you recall, the first thing we want to do is isolate that navigation bar and the -- the Edit button and the navigation bar. We're going to log what we're doing, and on the app Object, find the navigation bar, get the list of button elements, and select the Edit button, and click Tap.

Now one of the things you'll notice with the automation is that we will patiently wait up to a certain timeout for elements to come on screen. One of the things that happens often in automation is that the thing that you're trying to target won't be -- there'll be animations that won't be on screen right away, so in this case, we're actually getting the right button, and we're making sure its name has changed to Done before moving on. So now that we know we're in Edit mode, we're going to go ahead and scroll to the element using that scrollToElementWithPredicate. In this case, when we're in Edit mode, the name of that cell will be prefixWithDelete, so we're constructing this string Delete and the recipe name.

We've saved a reference to that cell, and now we're going to look inside that cell for all of its switches. There'll only be one switch. We're going to get the first switch and set its value to one. Added some delays in here to make this run slower so you can see what's going on. Next, we look in that same cell, and we look for the Confirm Deletion button. So we're using Predicate again, nameBeginsWith "Confirm Deletion," and we're tapping that element.

Then we want to exit the Edit mode by selecting the done button in the navigation bar. And we'll search the table to make sure that that element has been removed. And we're going to take a screenshot there to make sure that it's gone so we can confirm later if we need to. And then, finally, all we need to do is call our new delete function here at the end.

Okay. So let's return to Instruments and let's see that run. Notice I don't need to refresh anything in Instruments, just hit Record again, it will pick up the changes. This time I'll let you watch it from the desktop side.

See how easy it is, once you have these tests to just iterate them over and over again. If you write them flexible enough, you can add several recipes to your, you know, app, and stress test your app. You can run this in conjunction with other instruments to maybe do some leak checking, as Michael mentioned. Here's our new test.

Okay. As you can see, we've passed the test, and we have the screenshot, we can see that it's been deleted. One of the things, though, it's all fine and dandy to know we have a test that passes. But what's really valuable is to know if we can catch a failure.

So it's important when you're writing your tests to test that failure case. So let's go ahead and switch over to Xcode, and we're going to select the Delegate function here that's responsible for deleting this recipe from the list. I'm just going to go ahead and add a Return, simulate a bug.

There we go.

I'm missing...

Okay. I'm not sure if I did what I wanted there.

Okay. So you'll see we failed to delete and --

We got a test failure. So I'll go ahead and hand this back to Michael, and he can talk to you about some additional things you can do with the automation.

So now you've seen how we can not only, you know, show that your test's a pass, but we can also catch a failure as well, so you know when something has failed in your application. So now I'm going to cover some of the more advanced topics, some of the more interesting things that you can do with UI Automation. So we'll start off with handling unexpected alerts.

One of the traditional problems that User Interface Automation has had is what to do when something unexpected happens. When something like this pops up on screen, I mean, you are testing on a phone after all, what happens if you get a text message in the middle of your automation run? Well, traditionally, everything would now fail at this point.

Your world, as far as the application is concerned, has changed; something has appeared on screen; your script wasn't expecting this to happen; it can't find the controls it's looking for; and everything just falls apart. And you end up looking at a whole load of failures, and you find out that Johnny sent you a text message and destroyed your automation run. So let's fix that. We can automatically dismiss these alerts. Let's have a look at how we do that. Well, create a function called onAlert. This is our Alert Handler function.

And all I've done in this function is just log a message, just logging the fact that this alert happened. And then I return false. And what false is telling the system is that I, as a script writer, have not handled this alert, and I want you to get rid of it for me.

And so what UI Automation is going to do is click that Close button, get rid of the alert, and your script continues running exactly where you left off. So whatever action you were doing previously, you go right back to where you were, and your script carries on as if nothing had ever happened.

Now occasionally, of course, you're going to get expected alerts. You're going to get stuff pop up that is that is part of your application. Now, maybe, as in this case, you've got a confirmation dialog. Now obviously you don't want to dismiss this alert automatically because this is part of your application, and you want to be able to deal with this appropriately when the time comes.

So let's have a look at our Alert Handler again. If we just run it as it is, we'd go ahead and probably click that Rename button, that is the default button, and that's not what you want to happen. So how do I deal with that? Well, I have this little code snippet here, that looks for this precise dialog box that's going to pop up, and when we find it, we're going to tap the Continue button instead of the Rename button and return true.

And that way we're by-passing the Default Handler, which means that UI Automation is not going to deal with this alert, you've told the system, you know, "I've dealt with this alert. Don't handle it. It's all dealt with. It's all fine." And at this point, your script goes back exactly where it was and continues running. So you don't have to sort of try and write extra code all over the place to deal with alerts as they happen, you just have one function that deals with them no matter when they occur.

And Multitasking. Huge new feature in iOS 4.0. Something we're very excited about, and everyone else is excited about it. You've probably already started implementing Multitasking in your application, and you want to be able to test that. So we try to make it as easy as possible to test Multitasking, and there's only one line of code you need to do to test Multitasking in your application.

As you know, when a user exits your application by tapping the Home button or causing some other application to come to the foreground, your application gets suspended. So we've simulated this entire work flow just in this one line of code. We code deactivateAppForDuration, give it a time, in this case ten seconds, and what that's going to do is simulate a user tapping the Home button on your device, your application will get suspended, ten seconds later your application will resume, it will come back to the foreground exactly the same way that a user would do it, as if they'd brought up the app switch, or tapped your app's icon, brought it back to the foreground.

And your script just continues running. You don't have to do anything else. That one line will simulate suspending your application and resuming it. It's that easy to implement and test Multitasking. So how do you deal with Orientation? All right, you've got your devices hooked up to your desk, it's running with your Mac, and you can tap on various things, but how do you simulate Orientation? We've not gone and built a robot that will rotate a device for you. That would have been awesome, but we kind of had to do it in software instead.

So how do we go about doing that? Well, again, really, really simple. Just simulating that Device Orientation. And to do that we setDeviceOrientation with one of the constants that are defined, in this case Device_Orientation_Landscapeleft. This is what's going to rotate the device. That just simulates the rotation that's occurred.

Now if you deal with raw accelerometer data in your application, this is not actually going to do what you want. This is just purely simulating that a rotation has occurred. So if you're using a UI device just to get notified when a rotation has occurred, this will do that work for you. Of course, once you've rotated, you do need to rotate back again.

And what I recommend is if you're doing a test that involves changing the orientation of your device, that you set your rotation at the beginning of your test, and then set it back at the end of your test to Portrait. That way, no matter what happens, your next test that runs, it's always back in a known, good state, and you don't risk getting confused about not being in the right orientation for whatever tests are carrying on. So let's talk about some of the advanced touch interactions that you can do.

What you've seen so far is how we can tap into individual controls and elements in a very simple fashion. But that might not be enough for your application. So there's a few things that I want to cover. We'll cover how you can do more advanced Taps, some Pinches, and Drags and Flicks. So let's have a look at, first of all, Taps.

So you see from this first line of code, I'm just tapping at an arbitrary location on the screen. So I've set x and y coordinates, and that simulates a tap happening at that location. Doesn't matter what's at that location, it's just going to tap right there. Double Tap is exactly the same. And also a two finger tap. So think how, like in Maps, do a two finger tap to zoom out, and this is how you do that in a simulated environment.

Pinches. So you can Pinch Open and Pinch Close, maybe to zoom in, zoom out, that kind of thing. And this is little bit more complex. Code pinchOpenFromToForDuration. And so again this is coordinates that's starting and ending the pinch and the duration, this is the time in seconds for that action to occur. So in this case, we're doing two seconds. Well, what that means is, for your application you may want to simulate a very slow pinch or a very fast. You have some flexibility in the timing of that action happening. Equally, Pinch Close does the exact opposite, zooming in.

So Drag and Flick. Maybe you just need to scroll through a table really quickly, maybe you've got other elements that you just need to move around on screen. Call dragFromToForDuration, and this is our starting location, our ending location, and a time. So in this case, this is going to drag from 160 by 200, to 160 by 400, over a period of one second. Again, you have the flexibility of specifying a time.

Whether you want it to happen very quickly or very slowly, completely up to you and what your application needs to do. flickFromTo is a little bit different. A Flick is, by definition, a fast action. So there is no time on this because it has to happen quickly. But otherwise, code in much the same fashion.

Timeout. As you see when Matt built up his script, occasionally you need to wait for an element to appear on screen. Maybe you've got some great animation or transition that is occurring in your application after you click the button, and you want to wait for that to happen before you start interacting with another control.

So you want to wait for an element to appear on screen before you interact with it. How do we do that? Well, just try and access the element directly. And what this will do is keep trying to access that for a given amount of time. Once it finds that element, that line of code returns, and you can carry on doing whatever you need to do with it. By default will wait up to five seconds. If that element doesn't appear within that five second period, it will get an exception because clearly something has happened in your application that wasn't expected, and you need to deal with that in some other way.

But maybe five seconds is too long or too short. Maybe you've got this fantastic animation that takes ten seconds to occur, but it's awesome, but you need to wait that amount of time. So much like a stack, just go ahead and push a timeout, in this case I'm pushing a two second timeout to the top of my stack. I write whatever code that I need that's going to access these elements whenever I want to wait two seconds to occur, and ultimately I pop the timeout.

And the whole reason we have this timeout is so you don't have to pepper your script with delays. You might have seen that during Matt's run, that we had a few delays in there -- that was purely to slow down the script to make it easier for you to see what was going on; otherwise it tends to run through stuff really quickly and makes it quite difficult to see. But using this, you end up with a robust script, waiting exactly the right amount of time for something to happen.

So in summary, what you've seen today is how you can take UI Automation, how you can write scripts for your application very easily. It's going to cover as much testing as you can so that you don't need to spend a lot of time manually testing your application. You can have something do it for you while you focus on other things.

There's another session a little later this afternoon on Accessibility, so if you want to know a little bit more about how you can name your elements and interact with them, a little later this afternoon go to this session, they'll tell you more about it. I've got a couple of resources up here. We've got some documentation that's great, it will cover all the elements we have and how to write scripts.