Since deploying Sauce Labs, the run time of the OpenDNS Selenium test suite has dropped to 20 minutes

The company is using Sauce Labs to execute a suite of 250 Selenium tests per build seven times a day.

With Sauce Labs, the organization no longer needs to manage its testing infrastructure, and it has reduced test runtime from one hour to 20 minutes.

OpenDNS is the world’s largest Internet-wide security network. Founded in 2005, the company has evolved from a provider of fast, reliable Domain Name System (DNS) services into a leading network and mobile security innovator that protects two percent of the Internet’s active users every day.

The company’s mission is to allow its customers – businesses large and small – to connect with confidence anywhere, anytime. OpenDNS does that by providing powerful security solutions that are simple to deploy and easy to manage.

Software testing is a critical part of the organization’s processes, and OpenDNS uses the Selenium open-source testing framework to run automated tests of its web application. However, as a small, growing company, OpenDNS did not want its engineering team spending their time managing an internal Selenium test farm.

“We needed a new testing tool that could automate testing and run tests across many browsers,” says Nick Guilford, QA Automation Lead, OpenDNS.

Additionally, the company wanted a better way to manage the growing number of tests it was conducting. “The more test development we do as a company, the more tests get written,” says Guilford. “These test suites can grow quickly and really slow down the build, so managing concurrency quickly enough to keep the build as fast as possible is always a challenge.”

While searching for a new testing solution, OpenDNS discovered Sauce Labs, a cloud-based automated testing platform offering support for Selenium. “Above all, we look for test tools that are reliable and fast, and allow our engineers to author and execute tests as simply as possible,” says Guilford. “Sauce Labs allowed us to run our tests concurrently across as many browsers as we wanted without us having to manage our own test farm.” OpenDNS also liked the fact that Sauce Labs records a video of each test.

Now, a team of 15 OpenDNS web developers use Sauce Labs daily. The developers write tests in PHP and Python using the Selenium framework and execute those tests using Sauce Labs and a continuous integration (CI) system based on Jenkins and GitHub. When a developer merges code into a master, Jenkins automatically kicks off a build, which deploys code across several test machines. If the tests pass, the build is promoted to a staging environment where the developer can do final testing before deploying the build to production.

Using 10 Sauce Labs virtual machines (VMs), OpenDNS executes a suite of 250 Selenium tests per build approximately seven times a day, for a total of 1,750 tests against Chrome and five other browsers. OpenDNS also kicks off the suite of tests anytime a pull request is issued, so that functionality is validated before the master can be polluted. This adds another 2,500 tests per day on average.

By using Sauce Labs, OpenDNS can take advantage of a cloud-based testing infrastructure that requires no internal management. “The ability to run Selenium tests in various browsers without having to worry about the magic behind bringing up and managing a test farm or having to hire engineers with a focus on building and maintaining a Selenium test farm is huge for us,” says Guilford.

In addition, the company is saving significant time by executing its Selenium test suite across multiple browsers. “Being able to run tests concurrently without having to worry about bringing up our own VMs is pretty awesome,” Guilford says.

With Sauce Labs, OpenDNS has also been able to reduce testing time. Previously, builds in the company’s Selenium test suite took about one hour each to test. But since deploying Sauce Labs, the run time of those tests has dropped to 20 minutes.