Good afternoon and welcome to what's new in testing. My name is Wil and I work on Xcode and on XCTest. So, what is new in testing? The answer is a lot. In fact, so much that we don't have time to do all of the justice here today, so let's get started.

First off, we have a bunch of enhancements in both Xcode 8.3 and Xcode 9. I'm going to just call these out and then I'm going to let you read up on them in the reference documentation for XCTest which has just gotten a major overhaul. Then we'll take a look at new APIs for asynchronous testing followed by multi-app testing and some great performance improvements in UI testing. Finally, we'll wrap things up with a group of technologies called activities, attachments, and screenshots.

Let's look at those enhancements. So, in Xcode 8.3, we added UI testing for Siri intents with the new XCUISiriService. We also introduced UI testing support for the Touch Bar on macOS. In XCTest for Xcode 9 we have refined the framework Swift interface as part of the Swift 4 effort and we've also added a new block-based teardown API that lets you structure context specific teardown in your test methods without additional state or properties in your test classes. In UI testing we've introduced a new element type for the macOS menu bar items. In addition, there's new API on XCUI element that waits on the existence of an element reducing the need for sleep or other artificial delays in your tests.

Xcodebuild now launches tests directly via core simulator so you'll no longer see the simulator app launch when running tests from the command line. We're also very excited to announce Xcodebuild now supports parallel device testing.

This means if you pass multiple destination specifiers to Xcodebuild it will build once for all specifiers sharing the same platform and then run the test for those destinations simultaneously. This should give a huge speedup to many continuous integration scenarios. On the localization side, in Xcode 9 you can set the language and region for your test to run in. This scheme option allows you to easily test many different localizations for your projects.

I also want to take a moment to call your attention to some improvements in Xcode Server. So first of all, as of Xcode 9 you no longer need macOS server, you can turn on Xcode Server for your team with just the flick of a switch by going to the new preferences pane directly in Xcode. Xcode Server also has an improved provisioning workflow and adopts the Xcodebuild improvements for CoreSimulator and parallel testing. And those per scheme localization support automatically extends to all of your bots.

So that is our whirlwind tour of the most exciting new enhancements. Now let's slow down a bit and take a closer look at asynchronous testing. Async testing allows you to validate code which doesn't finish immediately, but instead calls back later with closures, delegate methods or other delayed completions.

This includes tasks like opening documents, work done on background threads, communicating with other processes, network activity, animations, and a range of UI testing scenarios. We introduced APIs for async testing in Xcode 6 several years ago now. These methods on XCTestCase let you define conditions or expectations as we call them and then have your test wait for them to complete or be fulfilled.

So, here's a simple example. This fragment of test code opens a document, waits for it to finish, and asserts that it was successful. The expectation object is created before the document is opened. The test waits for it below the call to open and fulfills it inside the closure which allows the test to then continue on and execute the test code after the call to wait. But this is the original API that we had in Xcode 6. It works well enough, but it has some limitations.

First, timeout string waiting are always treated as test failures. Second, waiting requires the test object itself which makes it hard to factor out into any kind of test library or support code. It's also not possible to have what we call nested waiting. Nested waiting involves additional groups of expectations that get waited on inside an outer weight context.

To solve these problems, we've introduced a new class XCTWaiter. This class extracts the logic of waiting that used to be contained in XCTestCase and allows you to explicitly declare the expectations you're waiting on. Timeouts and other events are handled through a delegate API and are also returned from the wait API as a result value. This provides considerably more flexibility in how you can structure your asynchronous tests. If we return to the previous example, let's take a look at what the options are now for waiting.

In the original code, note that no expectations are mentioned. This is because the implementation is implicit, it waits on all active expectations the test object has created. With just a small change now the test is waiting explicitly on the document expectation. Now another equivalent way of constructing this is to create an instance of XCTWaiter with a test case as delegate. And yet another option is to wait using a class method on XCTWaiter and then handle the result value. This flexibility makes it really easy to use XCTWaiter in helper method, nested contexts or in test library code that's completely decoupled from your test cases.

Now in addition, to creating XCTWaiter we've also expanded the API for XCTTestExpectation. The initializer is now public API which also decouples the creation of expectations from XCTestCase. We've also introduced an expected fulfillment count property, this is where expectations representing conditions which occur multiple times. Now for conditions which should not occur at all there's an inverted API for expectations which will cause the waiter to raise a failure only if the expectation is fulfilled before the timeout collapses.

Finally, XCTWaiter allows you to enforce the order in which expectations are fulfilled using an optional flag with the wait APIs. So, those are the updates to asynchronous testing, XCTWaiter, our new API for managing expectations and a bunch of improvements to XCTest expectation. Both of these classes are fully decoupled from XCTestCase giving you a much easier and more powerful system to use.

Now I'd like to talk about UI testing and in particular, UI testing with multiple applications. The starting point of almost every UI test is an XCUIApplication instance. This class lets you launch and terminate the app you're testing, as well as create queries for finding the user interface elements which you then automate by sending synthetic events.

UI testing has a target application concept which is the application your tests are primarily testing. In your project settings for your tests you designate an app in the same project as being the target application. This enables you to call the default initializer for XCUIApplication and the instance will be created with the information for installing, launching and interacting with that application.

This mechanism is very convenient, but it doesn't give you a way to test other applications. Some examples of what we would call multi-app scenarios include app groups where you have more than one application in your project and they somehow interact together and pass data back and forth. Another example would be settings if you need to automate the settings for your app and change preferences for different test scenarios. And yet another example, our app extensions. All of these require more than a single target application.

To solve this, we've added some new APIs to XCUIApplication. First, there are initializers that allow you to specify an app by its bundle ID or alternatively on macOS you can also use a file URL to designate the location of the app on disk. Second, we've added an activate method that will bring the app from the background to the foreground if it's already running and launch a new instance if it's not.

But launch API would terminate any previous running instance first, so you always have a cleaner slate when you start your test. Activate is useful for those scenarios when you're not interested in wiping out previous state, but you actually want to resume from some earlier point in the test. Finally, there's a new state property that you can use to monitor changes in the applications you're testing.

So, here's some example of using these APIs. This code creates two XCUIApplication instances with bundle IDs. This is a pair of apps perhaps in an app group. After launching and interacting the readerApp the test then launches the writerApp and finally, after some more interaction it uses the activate API to bring the readerApp back to the foreground without terminating it first. So now let's see that in action, here's Warren Ma with a demo of multi-app UI testing.

Thanks Wil. So as Wil just described, now in Xcode 9 you can write UI tests that involve multiple applications and I'm excited to show you guys how you might incorporate the new APIs into your own test suites. So, let me go ahead and start by showing you these two related apps that I have here. I'm running them on a development device and I'm using QuickTime to show you what's on the screen. The first one is a message posting app, the writerApp.

Here we have a username, we have a text view that we can tap into, we can type out a test message, tap return on the keyboard and post it to the server by tapping on the Send button. Great, so it's been posted. Now how do we view these messages? Well that's the purpose of our second app the readerApp.

Here is a list of all messages that have been posted to the server and at the very top is the most recent message posted the one I just sent and we can tap into that to view it in detail. And then we can return to that list of messages using the button at the top left All Messages.

So, let's say you want to test this process of typing out a test message in the writerApp, posing it to the server and then verifying that it appears at the top of the list of messages from the second application. Well before Xcode 9 you would have to write separate UI tests for each application and even then, you wouldn't really have a way of verifying the behavior between both.

But now with multi-app testing in Xcode 9, multi-UI testing in Xcode 9 we can easily test real-world scenarios between multiple applications. To show you how that looks let's write a quick UI test to automate the workflow I just demonstrated. All right, the first step in our test is to launch our Reader application. In order to do that we're going to initialize it using its bundle ID. Afterwards we can go ahead and launch it from a clean state.

Once our readerApp has been launched we're just going to go ahead and verify the first message in that list of messages. To do that we'll tap into the first message in the list, we'll verify that its contents are what we expect and finally, we'll return to the list of messages using that button at the top left.

Once we verified the first message in the list we want to type out our test message and post it to the server into the writerApp. To do that the first step once again is to initialize the writerApp using its bundle ID, in this case it's com.mycompany.Writer. Afterwards, it's as easy as just calling Activate. Now Activate also waits for the application state to become running foreground before returning, so we don't need to do any sort of manual waiting on the application state. Once Activate has returned we know that the test is good to continue.

Once the writerApp is running foreground we can go ahead and compose our test message and post it to the server. To do that we'll tap into the text field, we'll type out our test message, we'll tap on the return button on the keyboard, and then we'll tap on the send button to post it to the server.

Now we want to make sure that it appears on the top of the list of the messages in the readerApp. Earlier we used Activate to switch between the two applications, but in this case, let's make use of that back to app button that's built into iOS which appears at the top left corner of the screen.

So, to do that we're going to tap on the back to app button. However, in this case because we're not using Activate we do need to manually wait on the application state to become running foreground before continuing with the test. So, to do that let's make use of the new predicate based expectation API.

So, we want to make sure that the Reader app state has become running foreground before returning. After we've defined our expectation we'll go ahead and wait for that expectation to become true with a timeout of 10 seconds which should be more than sufficient. Once the Reader app has become running foreground we can go ahead and continue with the final step of our test which is to verify that the message appears at the top of the list.

Similar to what we did before, we'll tape into the first message in the list, we'll verify that its contents are the message that we just posted, and then we'll return to the list of messages. So, let's go ahead and run the test to see it in action. To do that I'll click on the test diamond in the left-hand gutter.

So, when you run UI tests in general it has to install what is called the UI test runner and that is the test process in which UI tests run in addition to installing the app. So, it just takes a bit of time for that to launch. Once it's launched the test can go ahead and continue to run.

So once again, the first step of our test was to launch the Reader app and verify the first message in the list of messages. Then we use Activate to open the Writer app, type out our test message and post it to the server. And we use the back to app button to switch back to the Reader app and verify that the message appeared at the top the list. So [inaudible] how easy it is to make use of the awesome new multi-UI testing APIs available in Xcode 9 to test multiple applications. And with that I'd like to hand it back to Wil.

Thanks Warren, that's really great stuff. With just a few additions the realm of UI testing possibilities has gotten much broader. The new initializers let you test any application and the Activate API means you can switch between multiple applications without restarting them, multi-app UI testing. So, let's switch gears and talk about performance in UI testing.

At the heart of UI testing are the user interface elements that your test will want to interact with. These are the buttons, labels, table views, etcetera. The UI tests create queries to describe how elements are found. For example, here a button element has the query app.navigationBars.buttons with the subscript of done.

That means that the element has the type of a button, the label or title of done and can be found in a navigation bar. So how do these queries work? Queries use accessibility data, the same semantic information used by VoiceOver technologies as a kind of searchable structure for the application.

To evaluate a query the test process requests what we call a snapshot of the current data. This request is sent from the test process to the application so these are separate processes, we're using interprocess communication here. The request is sent from the test process to the application, now the application captures the snapshot data, serializes it and transmits it back to the test process. Once the test process has unpacked the data it can evaluate the query by searching through the snapshot.

Snapshotting works and because of how it works it takes a single atomic representation of the state of the UI at that moment, but it introduces some performance challenges. If we consider two axes of performance, time on one hand and memory on the other, snapshots have potential problems with both.

Snapshots that take too long for apps that have many UI elements, for example tables with thousands of rows, large collection views these will cause timeouts that in turn trigger test failures. If the snapshot data is too large memory pressure on the system may also result in processes being terminated. So, these challenges led us to the question of how can we improve snapshot performance. And we came up with several approaches.

First of all, we wanted to cut down on the overhead of transporting all this data between processes. To do that we implemented what we are calling remote queries. With remote queries, instead of the test process requesting the snapshot it actually transmits the query itself, which is a very small amount of data. The app will still create a snapshot, but instead of transporting it anywhere it simply evaluates the query right there in process. And at the end, it sends back just the results, again a tiny amount of data, to test process.

So how does all this perform? Well remote query performance turned out to speed things up by as much as 20% and reduce memory spiking by about 30%. Now this was a good start, but it was nowhere near what we had in mind to achieve. So that brought us to our second optimization which is query analysis. Our goal here was to reduce the size of the snapshot itself, to simply collect less data.

Snapshot uses a fixed set of accessibility attributes, but we determined by analyzing your queries we could identify a minimal set of attributes, roughly half the full set for many common queries, greatly reducing the amount of data collected. Other properties that you might need for assertions afterwards could still be fetched on demand for specific elements, but the snapshot itself will now be much slimmer.

So, query analysis turned out to be an even better performance optimization, roughly 50% faster in the common cases and having a memory high watermark around 35% lower. So things are looking pretty good, but we wanted to step back and see if by really getting outside the box we could reimagine the system in a way that would transform things.

That brought us to the idea of eliminating snapshots entirely. So how could we do that, snapshots gave us this guarantee of an atomic representation of the state of the UI? Well it turns out because of the remote query infrastructure we now have that just because we were already running our query in the process of the application.

So, where traditional queries work by examining the snapshot data exhaustively we have introduced an API called first match that tells the query stop as soon as you find the first thing that matches because many times it's excessive to search all through the data to find every possible match because you can specify elements with precision that really makes them unique. So, first match causes the query to return early and you can add it to any query that you already have.

Here's an example where first match has been added to a query we looked at earlier. And if you imagine that the app we were testing was one with a navigation bar and a table view and the table had thousands of rows in it, traditional evaluation would've examined every single row in the table even though we're looking for a button in the navigation bar.

But first match allows us to stop as soon as we find that button and we never look at a single row in that table. So how does first match perform? Well it's safe to say first match is a game changer. For many queries, it's as much as an order of magnitude faster and eliminates memory spiking entirely.

Now the performance improvements in first match are fantastic, but it is important to notice the difference between traditional queries and not simply litter first match throughout your code without some consideration. Traditional query evaluation finds all matches, this helps detect ambiguous queries because will raise a failure if you attempt to interact with an element that has multiple matches.

First match is removing that protection, your test will get the first match. And if something in your app's UI has changed so that query is actually not particularly precise and results would be a non-unique result first match won't protect you and the results could be surprising. So, let's take a look at some example queries and consider whether they're good candidates for first match.

So app.buttons.firstMatch is not a good idea, this is like going into the grocery store and just saying I want food. You might get a frozen chicken or a piece of bubblegum or banana there's no telling right. So, this kind of query is simply not precise enough for first match.

Now this is a little better because we've added an identifying string to it as well. So many apps this might be sufficient right there. But taking it a step further and adding more precision, more level of detail to the query makes it a better candidate for first match and more likely to just be robust everywhere while still giving you the performance improvements that come with first match.

So while talking about all these optimizations we need to tell you a little bit about block-based NSPredicates and using them in your queries. Unfortunately, they are at odds with these optimizations, they effectively inhibit them. That is because first of all, blocks can't be serialized, you can't pass them around between processes so that means no remote query and no first match.

The other problem is that we can't introspect, we can't look into a block at runtime and no which attributes your query actually needs. So that means the reduced snapshot performance improvements are also off the table when you're using block-based NSPredicates. Fortunately, the use of block-based NSPredicates in UI testing queries is relatively rare and can almost always be replaced with a format string or an NSExpression based predicate instead.

Now if you find yourself with a case that does require a block-based predicate, I mean they're still supported, they still work or even something where it's much more convenient to use one. We'd like to know about it, we'd like you to file a bug so that we can give you an API that does the same thing, but in a way that works well with the query optimization. So, we'd like to hear from you if you feel you have cases where you need a block-based predicate.

That wraps up our discussion of UI testing performance improvements. Queries should be all around faster in Xcode 9. Some of the improvements, the remote query and first match do also require the newest OS as the newest macOS, iOS and tvOS. But the query analysis benefits should work even on older OS's. So, UI testing performance we've done a lot of work here and we look forward to seeing your tests running faster.

So finally, I'd like to share with you a new group of technologists called activities, attachments and screenshots. So, let's start with activities which are a new way to create additional structure and longer running UI and integration tests. There's a single API that lets you group together sections of code by wrapping them in closures passed to a new class XCTContext.

Now here's part of the test report from Warren's demo earlier, which we didn't get a chance to look at, but here it is now. And that test wasn't particularly long or complex, but you can see there's quite a lot going on here. So, consider these four actions. Let's look at the code that caused them.

We have a query to find the view, we tap on it, type some text and interact with a few buttons. This code makes a good logical grouping, it's the code that composes the any good coffee places message. So, creating an XCTActivity for this code is very simple. We simply wrap it in this run activity call, we give it a nice label because that's what we'll show in the test report. Now let's look at how that changes the test report.

We have the original four activities now enclosed in this new activity compose coffee message and you can expand it still and see the original, more granular details underneath. But for the high level first pass when you're looking at your test report things will now be more concise and semantically meaningful.

Using this API throughout your longer running tests and in helper methods will just make the test reports much easier to explore. Now along with activities we've introduced something called attachments. For a long time now we've wanted the ability to attach richer data to test reports. The primary motivation is to make it easier to triage failures with additional logs or other data that give insight into the conditions at the time of failure. In addition, this could be used to support various postprocessing workflows. For example, sitting down with your designers and looking at screenshots together.

Now attachment support any kind of binary data with convenience APIs for strings, property lists, codable objects, files and images. Any of these types can easily be attached to your test so when you look at the test report there that data is. And that brings us to the third edition in our group of technologies which are screenshots. Many of you have been asking us over the years for an API to explicitly capture screenshots on demand. Well we're happy to say that here it is, we hope you enjoy it.

This new XCUI screenshot providing protocol is implemented by both XCUIElement and a new class XCUIScreen. So, when you use an element and you capture a screenshot it'll be clipped to just the frame of that element. So, a button, you'll just see the button, a window you'll see the full window and so forth. And if you use the screen API you'll get the full screen regardless of what application or applications are present on it.

Now with attachments and screenshots, some of you may be wondering how quickly will this fill up my hard drive. Well the default policies for attachments and also for screenshots that are captured automatically during UI testing is that if your test passes we delete these for you. The assumption is in the common case you don't need them, but if your test fails they're present for you. Now you can override this policy in your scheme. In the same part of the UI where that localization control exists you can tell us whether you want screenshots captured at all automatically and whether to delete these and attachments when tests succeed or not.

There is also API on the attachment class that lets you on a per instance basis say keep this, don't keep that, and that sort of thing. So, you'll see that in the next demo with attachments and activities and screenshots in action. Please welcome Honza Dvorsky to the stage.

Thank you Wil. Good afternoon, my name is Honza and today I'd like to show you how we can organize your tests using activities, then how to screenshot your UI with the new screenshot API, and finally how to attach arbitrary data with your tests. So, we'll start by looking at the test report of Warren's test. We can do that by Control-clicking on the test diamond and selecting [inaudible] the report.

When we disclose the test, we reveal the test to transcript. The test transcript contains all the details about our test, but when it gets longer it can be difficult to navigate. In addition, what we see here are these leveled steps like these tabs and swipes, but it would help us to organize our test around these higher-level tasks like launch the app or compose in send a new message. So, we'll use activities to do just that.

We'll jump back to the source and look for good candidates to wrap in our first activity. For example, this piece of code, compose and send a new message takes care of that. So, we wrap it in an activity by calling XCTContext.runActivity. It takes two parameters, the first is the name of the activity, compose and send a new message in our case, and the second is the block, the block represents the scope of the activity. But we'll close it down here and that's it, that's all you need to do to wrap a piece of code in an activity. Now I'll sprinkle more activities around this test. And we'll rerun the test to see how the test report changed.

You can see that I wrapped the launching of the Reader app, the verification of the first message, the activation of the Writer app, the composing and sending a new message and so on. Now this is the same exact test that Warren wrote here, but this time the activities and the names we give them are getting included in the output as first-class citizens. So, we send our message, verify that it's the right one, and we're done. So, let's go back to the test report now.

When we reveal the transcript now we can see it's much shorter and in addition, it better describes what our test does. It launches the Reader app, verifies the first message, activates the Writer app, composes and sends a new message and so on. But if you still need to know all the details about that activity all the sub activities are hidden one level deeper. So, activities are a great way to organize your tests.

Now let's switch gears a little bit. Our designers that helped us with our app wanted to make sure that our message cells here follow their beautiful specification exactly. So, what we'll do is write a UI test which captures the visual state of the app and then we'll run this test every night on our bots. This way, our designers can come in and see what the app looks like anytime they want.

So, this is a new class visual validation tests. I already have the code to launch the app in the state I want it in and I also have this empty activity gather screenshots and this is where we'll place our screenshots now. We'll capture two screenshots, the first of the full-screen of the app and one of just the first message cell.

So, to capture the full screen we use the new XCUIScreen API to get a handle to the main screen. Now as Wil mentioned, XCUIScreen conforms to XCUI screenshot providing so we can just simply ask it for a screenshot of itself. Now we have the screenshot in memory and we somehow want to persist it with our test and this is where attachments come in. Attachments can hold any data and XCTAttachment provides convenience initializers for types like strings, files, images and screenshots.

So, we'll create a new attachment to hold our screenshots. Now as Wil mentioned, the attachments get deleted whenever the test passes, it's a default behavior. But in our case here we want to persist these attachments regardless of the test result. We can do a separate attachment by customizing its lifetime to keep always. And finally, we add this attachment to an activity, we'll add it to this activity that represents our gather screenshots.

Now we captured the full screen, we also wanted to capture just the first message cell. So, we first used the existing UI testing API to get an XCUIElement for the cell. An XCUIElement also conforms to XCUI screenshot providing so we can just follow the same steps here. We ask the cell for a screenshot of itself, we create an attachment for it, customize its lifetime and add the attachment to the activity. So, let's run the test now and see what we get.

This is a simple test that just launches our app and captures the two screenshots, adds those two screenshots as attachments and finishes. That's it. So, we go to the test report. Now when we reveal the test we see our gather screenshots activity and it contains our two attachments and [inaudible] activity. So, this is the attachment that represents the first main screen screenshot and this represents the screenshot of just the message cell.

We could use this QuickLook icon here to get a preview window for it, but in fact I want to use this the Assistant Editor to do that. So, we just select the right screenshot and this is our full-screen screenshot and this is the screenshot of just the first message cell. So, now you can see how super easy it is to capture, oh thanks.

Now you can see how easy it is to really capture any screen or element you have in your app with the new screenshot API and attach it to the test with the new attachments API. I really hope you give it a try. Okay and with that I would like to invite Wil back to wrap things up for us. Wil.

Thanks Honza, that is pretty awesome stuff, you can see just how easy activities make it to improve the test structure and capturing rich data with attachments will make fixing your test failures simpler than ever and screenshots, I mean who doesn't love screenshots. Activities, attachments and screenshots it's a great new set of technologies for use in your tests.

So, we started off with what's new in testing and as you can see, quite a lot. We have many new APIs for you to use and we didn't even touch on all of them here today. So, we have lots of new APIs and we have the new workflow and continuous integration features in Xcode, Xcodebuild and Xcode Server and we have those great performance improvements in UI testing. So, thanks for coming today.

The session, page session of that link above has all the related resources and documentation, everything you're looking for. And of course, we have lots of sessions in the past and one tomorrow that I encourage you to check out. Also, the previous years' sessions have some really great information about other parts of testing in Xcode that we didn't really take a close look at today. Thanks.