2020 Continuous Testing Benchmark Report reveals 27 percent of all test failures are traceable to a single common failure pattern
New Failure Analysis offering leverages AI and ML to rapidly analyze test data and use pattern recognition to surface most common causes of test failure
Enables developers to quickly address the most pervasive issues impacting test quality
SAN FRANCISCO – May 12, 2019 – Sauce Labs Inc., the leading provider of continuous testing solutions that deliver digital confidence, today announced the launch of Failure Analysis, a new analytics solution that applies sophisticated machine learning to customers’ pass/fail data in order to surface the most common reasons a given set of tests fail. Failure Analysis gives developers, testers, and QA managers insight into how often the same type of failure repeats itself across a given test suite, enabling them to move quickly to address the most pervasive issues and drive rapid improvement in test quality.
The newly released Sauce Labs Continuous Testing Benchmark Report for 2020 reveals that a significant portion of test failures can be traced to a small number of underlying issues impacting a test suite. Based on anonymized data from millions of tests run on the Sauce Labs platform, the benchmark report found the single most common failure pattern was responsible for 27 percent of all failures within a test suite. The top three most common failure patterns accounted for 48.2 percent of all failures, and the top five most common failure patterns accounted for 59.3 percent of all failures. By finding and aggregating the most common failure patterns, developers can remedy the majority of test failures with just a few fixes, eliminating much of the costly and time-consuming manual follow up traditionally required to determine the source of failed tests.
“Testing is critical to delivering flawless applications and creating great user experiences, yet the task of maintaining a high-quality test suite has only become more difficult as organizations accelerate the pace of development and automate testing at scale,” said Matt Wyman, chief product officer, Sauce Labs. “With Failure Analysis, developers finally have a machine learning-driven tool to help balance the demand for faster deployments with the heightened need to maintain a reliable test suite. We’re excited to deliver this new resource to aid in the testing community’s collective efforts to improve test quality, increase developer productivity, and build digital confidence.”
Improving test quality is a critical need
The release of Failure Analysis comes as many development teams reach an inflection point in their digital journeys. Though more organizations than ever have shifted to agile development methodologies and implemented continuous testing throughout the software development cycle, poor test quality continues to undermine many efforts to deliver quality applications at speed. Poor test quality also increases the occurrence of false failures, frustrating testers and developers responsible for manually following up on those failures, and eroding confidence in the overall automated testing process.
The findings of the 2020 Continuous Testing Benchmark Report underscore just how prevalent poor test quality is among most enterprise organizations. The report found that only 24.37 percent of organizations pass at least 90 percent of their desktop tests, while only 25.45 percent of organizations pass at least 90 percent of their mobile tests.
Failure Analysis from Sauce Labs addresses this increasingly urgent need by helping testers and developers better understand why tests are failing and enabling them to prioritize remediation accordingly. Leveraging powerful machine learning algorithms to analyze an organization’s pass/fail data and its Selenium and Appium command logs, Failure Analysis gives testers and developers visibility into precisely where and how often failure patterns recur within a test suite, all from a single dashboard view within the Sauce Labs UI. Using the learnings provided by Failure Analysis to improve test quality, organizations can also dramatically lower the occurrence of false failures, ensuring that failed tests carry a high signal-to-noise ratio and developers feel confident that a failed test has truly detected a breaking change as opposed to the flaky noise of an unreliable test suite.
Failure Analysis is available now as a standard capability for enterprise customers as part of the Insights package within the Sauce Labs Continuous Testing Cloud. Sauce Labs provides a host of analytics capabilities that provide intelligent views into test data, including a new test overview and trends dashboard, interactive visualizations that enable customers to drill down on granular data sets, and a powerful REST API that allows users to aggregate data into their home-grown quality dashboard or test analytics tools. For more information, visit the Insights landing page.
About Sauce Labs
Sauce Labs is the leading provider of continuous testing solutions that deliver digital confidence. The Sauce Labs Continuous Testing Cloud delivers a 360-degree view of a customer’s application experience, ensuring that web and mobile applications look, function, and perform exactly as they should on every browser, OS, and device, every single time. Sauce Labs is a privately held company funded by Toba Capital, Salesforce Ventures, Centerview Capital Technology, IVP, Adams Street Partners, and Riverwood Capital.
