Did you hear the news? Sauce Labs has now surpassed four billion tests run on our cloud-based continuous testing platform! Users are now running more than three million tests per day on Sauce Labs. As we celebrate this milestone, we’re reflecting back on some lessons we’ve learned along the way. Here’s a roundup of our thoughts and advice after working with customers across the globe to help them develop with confidence and ensure their web and mobile apps look, function and perform exactly as they should.
Ensure stability with a consistent testing environment
After four billion tests, we know that environmental instability is a big source of uncertainty in testing. Tests that fail due to factors such as misdeployed APIs and incorrectly configured network/web infrastructure are often due to a couple of key factors:
A test environment that's different from production, both in level of resourcing and complexity of configuration.
Teams being dependent on other team environments in order for tests to run correctly.
--Marcus Merrell, Head of Technology Strategy
Smaller is better
Browser tests have a tendency to be too long (more than 30 seconds locally). The data clearly show that the longer a test executes, the more likely it is to fail from a false positive. Atomic and autonomous browser-based tests are recommended to alleviate false positives.
--Nikolay Advolodkin, Principal Solutions Architect
Innovation in quality can truly affect people’s lives
Companies that provide a remarkable digital experience understand that’s where promises are felt most acutely by customers. When users have an enjoyable experience with a company, they are more likely to continue doing business with them. I’ve seen this in my own life, too—when I needed movie tickets, I downloaded the movie theater’s app and the buying process was so easy and seamless that I decided to buy an annual membership! Leaders in organizations that empower their development teams to innovate and ensure quality have the power to create meaningful moments in people’s lives, and I love partnering with them.
--Jasmine Williams, Director, Customer Success
Don’t get comfortable with failure
Teams have a high tolerance for failing/error-prone tests. We frequently see teams execute the same failing tests over and over, indicating that they are not fixing the problems, and that the severity of these failures isn't significant enough to trigger a change in the process.
--Marcus Merrell, Head of Technology Strategy
Get better coverage with more types of testing
There continues to be way too much focus on browser-based functional testing. Many modern web applications should also leverage unit tests, component tests, visual tests, and API tests for reliable and efficient test coverage.
--Nikolay Advolodkin, Principal Solutions Architect
Sauce Labs will be here for the next four billion
We've seen the browser ecosystem change significantly, and our customers change with it. Internet Explorer was once the king, only then to be subsumed by Firefox, which then gave way to the upstart Chrome. Edge replaced Internet Explorer, and then Google's rendering engine replaced Edge's rendering engine. Meanwhile, Safari has continued to dominate a large part of the Apple ecosystem. Through it all, Sauce has been at the forefront, not only providing these platforms to our customers, but also guiding customers as they choose what kinds of coverage they need. We’ll be here for our customers with what’s next, too.
--Marcus Merrell, Head of Technology Strategy
