Speeding Up Your Test Execution

For the online version of Agile Testing Days 2020, I had the opportunity to do a talk. I opted to talk about speeding up test execution; specifically, the five tips I have compiled that will help you do just that. Overall, these five tips tie together quite well, but can also be applied separately. In the course of researching and preparing this talk, I was surprised to discover a new finding on a widespread recommendation related to using XPath as a locator strategy for automated tests.

Before we start talking about how to speed up test execution, we need to understand first why and what the driving motion behind this initiative is. For that, let's take a look at the DevOps cycle.

Continuous Testing means that there are multiple different tests done throughout the whole cycle. If we can pass the testing stages faster, QA and developers benefit from faster feedback. Overall, this will also reduce time to market.

Various testing frameworks exist in the market. Some frameworks might take care of cleaning up resources on their own, but some others won’t. If a driver is not stopped after the test script has finished, the server will wait continuously for further commands until it hits the idle timeout. The default is 90 seconds in the Sauce Labs Continuous Testing Cloud. As a result, not having a simple one-liner like in this image shown can cause each test script to run 90 seconds longer than it needs to.

This means as a writer of test scripts, it is important to make sure that the driver quits under all circumstances. Senior Solutions Architect Titus Fortner explains it in more detail in this video here.

Using XPath as the locator strategy of choice when writing test scripts is a common occurrence. It is easy to understand and allows you to target all elements of a mobile app easily. There are two main reasons why XPath should be avoided, however, and one of these reasons only applies when used with older Appium versions on older OS versions.

XPath is usually slower than other available locator strategies like Accessibility ID. The usual timing difference between XPath and others is around 200-500ms, although this depends heavily on the structure of the app. In more complex apps with multiple level deep layers, the timing to find elements will increase exponentially. Appium Desktop in version >1.18 can show the different time needed to locate an element based on the used locator. As shown in this example the time needed to locate the Login button is 744ms, which is more than twice as long as using Accessibility ID (350ms).

XPath contributes to flakiness and instability of the whole test suite. XPath expressions are used to traverse XML trees. This XML tree is based on the current state of the mobile app, which can change after an interaction and therefore requires recalculation and a new traversing through it (slow!). Using XPath also causes a certain mismatch between the expectation of the test creator and the actual state.

For example, the XPath for the selected element has the index of [1], which makes sense because it is the first element in this hierarchy.

The second element in this list has an index of [2], so as a test creator I would note these expressions down and add them to my test script, expecting to select the backpack and bike light.

But the reality is that when we have selected the backpack, the XPath expression changes to something completely different as shown in the following image.

This makes sense so far, but when we take a look at the bike light we will see that it has changed.

The bike light now has the index of [1] and as such, the test case referring to the item with index [2] would select the T-Shirt instead of the bike light—making the test script wrong.

The generated XML tree for an Android app is different from the one generated for an iOS app, therefore making it necessary to maintain two different sets of locators. Accessibility IDs are cross-compatible, making it possible to have a single locator file for both Android and iOS. In addition, using Accessibility IDs is a best practice that will improve the accessibility of the app.

As I mentioned in the beginning as part of my preparation for my talk at Agile Testing Days 2020, I wanted to provide code examples showcasing how much slower XPath is compared to Accessibility ID. To my surprise, this wasn’t the case (anymore). Thinking that the difference in timing would not be great enough to affect a quick test, I created some longer-running end-to-end tests mimicking a user journey through Sauce Labs Swag Labs app. There was no noticeable difference. This prompted me to look for guidance from Sauce Labs Senior Solution Architect Wim Selles. After some investigating, he suspected a change in Appium and iOS (I only focused on iOS for this). Sure enough, after running both test scripts, one using only XPath and the other one IDs, against various combinations of Appium and iOS versions it was clear there was a performance improvement.

Starting with Appium 1.16 and iOS 13, both Appium and Apple have improved the speed significantly when using XPath, making it a viable locator strategy if the only KPI is speed.

This means it is wrong to say XPath is always slower, because under the right circumstances it is not. But it still should be avoided wherever possible.

As mentioned, the drawbacks of having to maintain two locator files because of incompatibility between Android and iOS XMLs, the inherent flakiness introduced and the positive side-effect of making one’s app more accessible when using Accessibility IDs makes it a no-brainer to choose IDs over XPath.

Although it should be noted that developing an app for Android and iOS in two different technology stacks and following the design + coding guidelines will result in accessibility IDs being less compatible between both apps. The real value of accessibility IDs is not only in testing but rather than they give real added value to visually handicapped end users.

Parallelization allows us to run multiple tests at the same time on multiple devices instead of running everything sequentially. For the purpose of running tests early in the development pipeline/lifecycle, I recommend using virtual devices. This means emulators (Android) and simulators (Apple) should be used instead of real devices. They are easy to set up and use, cheap, and will deliver quick results. On the other hand, real devices should be used mainly later in the pipeline/lifecycle.

The table shows the data for running the comparison XPath vs. ID. Overall there were ten configurations (Appium/iOS version), two tests (XPath/ID) resulting in 20 test executions in total. The average test execution comes down to 2m 18s. This results in a total test duration of 46m when run sequentially.

If full parallelization is used the total test duration drops down to 2m and 18s—an improvement of more than 20x.

When testing an application in the cloud, every Appium command becomes two web requests: one to get the element and the second one to act on it. Reducing the amount of web requests will naturally decrease the test duration. Depending on the used framework, the framework itself might cause a lot of unnecessary requests. This bloats tests up, causing simple tests to have hundreds or thousands of requests and responses.

The total amount of requests and responses for the created end-to-end test amounts to 36 events only, as there are no unnecessary querying of elements. Most frameworks utilize implicit waits, which means that the framework will wait for elements to become available. Appium is also using it as documented here. The use of an explicit wait as shown in the following figure is not needed. 

Instead the use of an implicit wait as part of the driver object can be used to simplify the wait strategy:

In cases where it is expected that an element might take longer than the timeout to load an explicit wait makes sense.

As a company progresses through its automation journey, it is important to do due diligence on tools and frameworks. This includes researching and comparing them with each other to try and find the best ones based on each organization's unique requirements.

The tips in this post should be easily implemented into existing frameworks. But as with all optimizations, it is better to have a solid foundation before spending a significant amount of time on optimizing the last few percents.

This post has hopefully helped reduce the amount of time you need to research test speed improvements and helped you understand why tests can be slow and how to improve them.