Hello and welcome to WWDC. Hi there, and welcome to my session, "Get Your Test Results Faster." My name is Sean, and I'm an engineer in the Developer Technologies group at Apple, working on the XCTest framework and its Xcode integrations. This session is grounded in a concept called "The Testing Feedback Loop," and in case this concept is new to you, I wanted to briefly take you through it. After this, we'll be going over some techniques and features within Xcode that you can use to speed up getting results from your tests.

If you've ever written an automated test before, there's a good chance that you have an intuition of what the testing feedback loop is, and this is because writing a test is the beginning of the loop. The loop then continues into you running those tests, usually alongside some other tests. And then ends up with you interpreting the results, and making some decisions based on what's in your test report.

Depending on your report, you may decide to write more tests because there are cases or features you're interested in covering. Or you may decide that you've written enough tests and the results give you confidence your code behaves as you expect. This confidence lets you move on to other tasks.

Having short feedback loops is important because that means you get results from your tests faster. If your tests are faster, then you can get confidence in your code faster, which then means you can ship features to your users faster. Now, having gone over the test feedback loop, I wanted to let you know what we'll be going over today. We're first going to discuss some features in Xcode 12 that will ensure your tests always complete. We'll also talk about how to use some diagnostics these features surface to figure out what may be breaking your feedback loops.

Then we're going to talk about how you can get even faster results from your tests by using the test parallelization options available to you in Xcode 12. Now, let's ground this discussion in a real-world example. Imagine you're at work, looking at the results from a CI job that you kicked off on Friday before leaving the office. With a hot beverage in hand, you realize that your long-running test suite never finished. You'd probably feel like how this person looks-- frustrated and a little upset that you need to start the week understanding why your tests aren't working.

If we don't investigate this problem, our tests will continue to take longer and will intermittently not finish, which is gonna ruin our confidence in our tests and application code, and that's gonna hurt our ability to quickly deliver new features to our users. This is a very unfortunate situation to be in, but lucky for us, there are some features in Xcode 12 to help us out. So, let's look into it.

We're gonna start with fixing the hang in our test suite, so we can always get feedback from our tests. Right now, the feedback loop is broken. Because our tests hung and never finished running, we never got our results, so we can't interpret them. We're left to cancel the tests and forfeit getting a complete understanding of our code base's quality.

This image is from a result bundle from that CI job which never finished. It has an error message that says testing was canceled because we had to cancel the CI job, which isn't exactly actionable. I'm left wondering what exactly went wrong in the first place. Without much in terms of diagnostics to understand why our tests are hung, we can try thinking of a few causes offhand. A classic example is a deadlock, where two sections of code are waiting for the other to make forward progress, and therefore neither does.

Even if our tests aren't stalled, their rate of progress may be so low that they're effectively stuck. Alternatively, this could just be due to poorly chosen time-out values in some application code, or it could be due to large amounts of CPU work that we're doing on the main thread of our app or framework we're testing.

Available in Xcode 12, though, is a solution to our problem of hung tests. It's a new test-plan option called "executionTimeAllowance." executionTimeAllowance is a customizable feature that you can opt into when running your tests. When enabled, Xcode enforces a limit on the amount of time each individual test can take. When a test exceeds this limit, Xcode will first capture a spindump, then kill the tests that hung... then restart the test runner so that the rest of the suite can execute.

We know it's not a great experience to try to guess at what could be causing our tests to hang. After all, our code base could be large and complex, and that doesn't lend itself to being easily reasoned about. Instead, we'd benefit from having some better diagnostics given to us so we can understand the cause a bit better.

A spindump can help us out here a lot, and this is why executionTimeAllowance attaches them to your test report. A spindump shows you which functions each thread is spending the most time in. If our tests are stalled, a spindump would help us see what functions the issue may lie in.

It's also possible to manually capture a spindump from Terminal using the spindump command, or from within Activity Monitor if you prefer a GUI. By default, each and every test will get ten minutes. If a test successfully finishes before that ten minutes elapses, the timer will get reset for the next test. If you need more time for all tests, you can customize the default allowance in your test plan's configuration. And if you need more time for a specific test or test class, you can use the executionTimeAllowance API to special-case a particular test or subclass.

executionTimeAllowance is represented as a time-interval property on XCTestCase. It's important to note that time-allowance values will be rounded to the nearest minute. For values under 60 seconds, they'll be rounded up to 60 seconds, and for a value like 100 seconds, it would be rounded up to 120 seconds, since that's the nearest whole minute. Having gone over the new executionTimeAllowance feature, let's go through a quick demo of turning it on and using the spindump it attaches to fix our hung test.

I have here in Xcode 12 the test that was hanging in CI. It's a test called testUpdatingSmoothiesFromServer. And it's a test of a method called fetchSynchronouslyFromServer. I'm gonna try to reproduce the issue at my desk here and I'm gonna do that by pressing the play button in the source editor gutter.

Now when I do this, I see the activity indicator in the Test Navigator spinning. If this test was working and not hung, it would execute immediately. I'm gonna stop the test because there's no use in waiting for it to stall. Now, I wanna turn on the executionTimeAllowance feature to get a spindump, so let's do that. I can do that by opening the test plan... menu, clicking the "Edit Test Plan" item, selecting configurations, turning test timeouts "on"... and then rerunning my test by pressing the play button again in the Test Navigator.

This will generate a new report for me that I can use, and I can view that report in the navigator. If I go ahead and I open up the report that I just generated, I'm gonna see the same test as failing, but its failure reasoning is different. It's gonna say that it exceeded the test executionTimeAllowance of ten minutes, which is the default. And it's also going to attach a spindump. You can open that spindump by double-clicking it and it will open inside of an editor tab.

Spindumps are generally broken up into two sections-- a preamble, which contains metadata, and then a series of stack traces for each thread within the process that was sampled. Since we're sampling our test-runner process, I know that my test names should be somewhere in that spindump. If I do a quick find, I can rapidly find my test within the stack trace, and see that it's calling the method under test as well as a private helper method. After that, I can see that it's acquiring a lock, and then it's waiting. This suggests to me that the issue is in that helper method that we have. I'm gonna look at the code, and I'm gonna navigate to the code by opening up the Smoothie.swift file.

Upon looking at this code, I can see that in this second method it's acquiring the same lock as our method under test, fetchSynchronouslyFromServer. It seems questionable to me that this helper method, which is just for performing a Get request, is acquiring a lock. And so I'm gonna try deleting this lock acquisition code from here to see if that fixes our deadlock. I'm going to reopen the Test Navigator by selecting the Test Navigator icon, and clicking the play button for our tests. We see that the test immediately executes, indicating that we fixed the hang.

Having demoed turning on time allowances for our project, let's talk about some ways you can customize them. There are two ways you can customize the default time allowance. The first, is using the test-plan setting, which is available in Xcode 12. And the second, is using the xcodebuild option.

Once you've enabled time allowances, there's a precedence order the configurations follow. This is so that you can set coarse-grained defaults and finer-grained values for special cases, such as CI jobs or long-running test suites. The TimeAllowance API has the highest precedence, while xcodebuild's TimeAllowance option has the second highest precedence. A Test plan setting has the third highest precedence, and the system default of ten minutes has the lowest precedence, and will be overridden by any of the other three options.

With all of these ways to set a time allowance, a question emerges, which is, what happens if a test requests unlimited time? There's a way to prevent this from happening, and that's by enforcing a maximum allowance. Your test is guaranteed not to exceed this value, regardless of the configuration you set in test plans or through API calls. You can enforce a maximum allowance either via a setting in the test plan or through an xcodebuild option.

Having gone over how to use the new time-allowance features, we wanted to offer a couple of recommendations for how to get the most out of them. For starters, use time allowances specifically to guard against test hangs and ensure you get diagnostics when they do. If you're concerned about keeping your tests fast, we recommend using XCTest's performance APIs to automate testing for regressions in the performance of your code.

And if you need to identify what parts of your code are slow, we recommend using Instruments to profile and understand your app's performance. Instruments provides a rich set of tools that will give you a lot of info that can help you figure out where to begin adding perf tests to your app code.

If you're interested in learning how to use Instruments, check out this talk from WWDC 2019 entitled "Getting Started with Instruments." Now, having adopted time allowances, our feedback loop has gone from being broken during the "running tests" phase, to being complete. And what's more, is that we will now always get results if our tests unexpectedly hang or stall.

We now have the ability to turn our attention to the fact that our tests take a lot of time, so let's dig into how we can speed up Fruta's test suite. Xcode 12 can help us shorten the loop even more by letting us run tests on multiple devices.

This is a test report from Fruta. We see the results of about a dozen tests that took between a few hundred milliseconds to several minutes to run. Overall, our tests took just about 13 minutes to run, with many of these tests taking time on the order of minutes to complete running. This is a clue that we would benefit from parallel testing.

Right now, Fruta is using non-distributed testing. That means each and every test case defined is executed serially on a run destination, and that will always take the most amount of time. You've likely experienced this if you have ever pressed Cmd+U in Xcode with parallelization disabled. A solution to speed this up is to use a feature we call parallel distributed testing. In the case of parallel distributed testing, xcodebuild will distribute tests to each run destination by class. Each device will then run a single test class at a time.

Once a run destination has finished running a class, xcodebuild continues to give it a new one until there aren't any left. It's very important to note that the allocation of test classes to run destinations is nondeterministic. If you're testing logic that is device- or OS-specific, this can lead to unexpected failures or skipped tests.

When we first added support for parallel distributed testing to Xcode 10, the supported configuration matrix looked like unit tests on macOS, and unit and UI tests on iOS and tvOS simulators. Starting with Xcode 12, the matrix now looks like this. You have the ability to run tests in parallel on physical iOS and tvOS devices via xcodebuild. To enable parallel distributed testing, set the parallel-testing-enabled flag to "YES," then set the parallelize-tests- among-destinations flag. This makes xcodebuild divide your tests over the destinations you specify.

With just two devices, XCTest's own tests suites achieved a speed-up of 30%. Just imagine what this would do for the Fruta app, or your app with more devices. By adopting distributed testing, we've been able to take our long feedback loops on XCTest and shorten them, enabling us to write, run and analyze our tests faster.

Now, we wanted to offer you a few recommendations for how you can leverage distributed testing in your own tests. Since test allocation is nondeterministic, it's ideal to use a device pool of identical devices and OS versions. This is so you can avoid difficult-to-reproduce test failures that may have been driven out due to the particular destination allocation xcodebuild made.

If you are using a device pool of different devices and OS versions, then we recommend you prefer distributing tests that are agnostic to the devices and OS's they would be running on. For example, tests for a framework of pure business logic are less likely to encounter issues since they wouldn't be running code that depended on destination-specific details.

Lastly, if you're interested in testing your code against more OS's and devices, for example, to prove that your app works with both iOS 13 and 14, then we recommend you use Parallel Destination testing. Destination testing runs the entirety of a test suite on a given destination, and does not distribute the individual tests across destinations.

If you're interested in learning more about destination testing, and how tests are allocated, check out "What's New in Testing" from WWDC 2018. Now, as a result of focusing on our test report and using Xcode 12, our tests will no longer hang or stall, they'll give us more diagnostics should a test take an unexpected amount of time, and they're faster! In conclusion, we recommend you use Execution Time Allowances to ensure your tests always complete in the event they hang.

Use spindumps for diagnosing application stalls and hangs, both for when your tests and app stall. Use parallel distributed testing to speed up your tests by running portions of your suite on different run destinations. And use parallel destination testing to simultaneously run your tests on more OS versions and devices.

Do this to get faster feedback on whether your code is behaving as expected between different OS's and devices. Thank you all so very much for joining me during this session. We hope your test suites are fast, that their feedback is actionable, and that you enjoy the rest of WWDC 2020.