A Celebration of 5 Billion Tests With a Data Deep Dive

When there is almost one mobile device per person on Earth, it becomes clear that this is an enormous market with seemingly limitless potential. Sauce Labs recently completed our 5 billionth test, a huge milestone and a number we are proud of for sure. Five billion, after all, is a big number. It was the world population just 35 years ago.

Now 5 billion is a number dwarfed by the current number of mobile devices at play. The sheer volume out there speaks to the need for every company to be investing in mobile testing now so they don’t start to fall behind their competitors. If you aren’t using mobile testing yet, you’re starting from the back of the pack, but it’s not too late. Using a trusted partner to accomplish this is one way to stay above water in the everlasting fight for mobile supremacy. Even if you’re already a Sauce customer you may not be aware that we recently introduced network capture, device vitals, audio capture, and private device management to provide a more comprehensive insight into mobile app quality and help keep you competitive in this growing space. The point is, that we have the tools to set you apart from your competition and the experience.

Speaking of that experience, we’ve learned a lot over the past 14 years and several billion tests. We thought a great way to celebrate would be to dive into some data and trends we’ve uncovered that set our customers up for success. At the very least, after reading this article you will be aware of the massive growth in the mobile testing space and realize that there is no reason not to be getting in on the action.

Mobile testing increased exponentially from 2020 to 2021 and the number is still on the rise. If we look at year-over-year data, Sauce completed 105 million mobile tests in the first half of 2021 and 122 million in the first half of this year, an increase of around 17 million. If we break down testing into real device vs virtual device testing, we can see some weekly trends that are also pointing up and to the right. Using January 2021 as a starting point, we’ve scaled our RDC testing from 257k a week to 605k, an increase of 135%. In the same time frame, virtual tests have increased from 17 million to 22 million weekly.

If we take a step back and look at testing as a whole, the news is just as positive as the mobile front. You are familiar with the phrase "a rising tide lifts all ships." In this metaphor test is the tide and each framework is a ship. 

Sauce Labs has embraced new frameworks like Espresso and XCUITest, which are seeing substantial growth. We’re also seeing large jumps with Selenium/Webdriver up 25% and Appium up 16%. These increases, along with positive trends across nearly every framework, resulted in 700 million total tests thus far in 2022, an increase from H1 2021 of 23%. Not only are the number of tests up from last year, but customers are also testing more each week in a magnitude of millions (21% increase). For those scoring at home, the three test automation frameworks showing the most growth have been Cypress, Espresso, and Appium. 

We’ve covered so far that both testing and mobile development are rapidly on the rise. What this means in a vacuum is that all your competitors are testing their mobile apps and surely putting out a high-quality product with an excellent user experience. Why? Here are a few numbers that will paint the picture in a macro sense. In the last five years, mobile app downloads have increased worldwide by 63% and mobile internet access is now sitting at over 90% meaning that 9 in 10 users of the internet are doing it via a mobile device. All these market forces and trends lead to an expected mobile app revenue of $613 billion by 2025.

Understandably, many brands are trying to secure their slice of this very large pie. To assure quality and timely releases for your product, Sauce Labs designed the mobile DevOps Test Toolchain. What is this magic toolchain you ask? A set of pluggable tools that accelerate the delivery of user value by providing capabilities to manage quality risk throughout the software delivery lifecycle. Pairing this with a data-enabled collaboration platform is the best way to answer the growing demands of your consumers. Essentially, the test toolchain is a one-stop shop for quality management on your mobile apps. 

We know that the user experience is everything to a digital brand and in this world, every brand is a digital brand. How do your customers interface with your company, place orders, and access support? Likely on the web or a mobile app. The best way to delight your customers and keep them coming back is to deliver a product that works the way it is supposed to every time. Even one bug on your mobile app can lead to up to 20% user churn. Bearing this in mind, it is imperative that you bake mobile testing into your overall corporate strategy before it’s too late. The Sauce Labs DevOps Test Toolchain can help you accomplish this whether you are an enterprise or a DIY developer. 

The first 5 billion tests taught us a lot, the next 5 billion will teach us even more. With over a decade of experience supporting a broad range of digital leaders across almost every industry, we are ready to take your mobile testing strategy to the next level.