Maximize Test Automation ROI with Hosted Test Orchestration

Despite years of investment around DevOps, test automation and observability, testing still remains the bottleneck to getting your product to market, and there can still be bugs in the code after you ship. 

Every day that you don’t ship your new features is a day of lost revenue, a day of disappointed customers, a day of letting your competition get ahead of you. The pressure to ship faster and more reliably is mounting, and you only get so much by throwing money and people at the problems. This conversation inevitably moves to testing, test processes, and the value stream of test.

Testing is often treated as janitorial work, a necessary step to be minimized rather than a strategic element of software development–one that can open up new avenues for velocity, user happiness, and even revenue. We at Sauce Labs recently released Sauce Orchestrate, a tool that makes test a strategic part of your release cycle–a profound piece of innovation by the company that invented Continuous Testing.

Teams spend thousands of hours developing tests, and these tests are usually slow and unreliable. This has always been a by-product of running such tests at scale, due to the number of elements that are outside your teams’ control (browsers, network latency, mobile device hardware, and more). Historically it’s been difficult to do functional testing that meets your desired criteria:

Most testing tools on the market cover 2-3 of these traits, but none of them does everything–there is always a trade-off.

Many organizations don’t even understand the full impact or ROI they get from testing, or how much time they’re spending on it as compared to the gains they realize. These programs are built ad hoc, usually by staff that’s under-trained and under-resourced. The only thing management knows for sure is that it’s expensive.

In a multi-year analysis of more than 5 billion tests run on Sauce Labs, we’ve been working toward a way to eliminate the parts of test automation that slow execution, introduce instability, and impede scale. What conclusion have we reached?

The point of origin for test automation is in the wrong place.

Teams currently use their CI/CD platform to control the entirety of the test process, rather than hand off that control to a platform that was purpose-built for test–in an optimized environment that provides deep insights. Given the distance between the CI/CD agent and the test platform, this introduces latency that has been impossible to eliminate.

Until now.

With Sauce Orchestrate, we invite you to move the “brain” of your testing into our data center, so that your tests execute in the same room as the device under test–not hundreds or thousands of miles.

Your system only needs to let us know when to start testing, and we handle the rest. We’ll send the results back to you for analysis the same way we always have. By obviating the vast majority of network traffic traveling back and forth between your CI/CD system and our data centers, we are able to help you move faster, increase stability, minimize risk, and realize massive ROI on your relationship with Sauce Labs.

Through our work with dozens of Beta customers, we determined that this simple change can speed up automated functional tests up to 70%, depending on the complexity of the tests. The time savings alone can save thousands of hours per year spent on CI/CD compute, but the real benefit is how much earlier you can get signals about your product. 

This has a direct impact on your ability to get to market faster.

Instead of developers waiting days or hours to see test results, now they’re waiting minutes. Instead of QA personnel spending months and years optimizing for the difference between remote and local test execution, they can just act as though everything is local.

When you tighten up the feedback loop between developers and testers, you move faster. When you have higher confidence in the results of a test suite, you have higher confidence in your product. When your team has fast feedback and high confidence, your velocity soars.

CI/CD systems were not built to handle test automation at the scale that modern applications and teams require. Now we have a solution in Sauce Orchestrate.

We hear from customers constantly that their biggest bottleneck–the biggest barrier to release–is testing. This has led many organizations to lower their testing requirements, to cut funding to automation programs–trading off the extra time at the cost of higher risk. People want fewer vendors, fewer tools, fewer barriers between themselves and production. 

We agree.

Your tests should be as fast, as accurate, and as reliable with a cloud vendor as they are when run locally.

With Sauce Orchestrate, now they are.