As a cutting edge development team, the Dollar Shave Club QA team enabled DevOps and continuous delivery by augmenting manual testing with automated testing.
The company started using Sauce Labs, taking advantage of 50 virtual machines to run hundreds of daily tests for its web and mobile applications.
Dollar Shave Club reduced its testing time from 6 hours to less than 10 minutes, saved 30 hours a week on testing time, and considerably lowered its testing costs.
The Challenge
Unlike a traditional brick-and-mortar store, online retailer Dollar Shave Club considers its website and android and ios mobile apps to be its storefront. That’s why the fast-growing company, which delivers razors and other personal grooming products to millions of members every month, strives to offer the best possible user experience. “Because our digital products are our storefront, we strive to make sure they meet the excellent standards we have defined as a company,” says Tim Nabhani, Director of Engineering, Quality, Dollar Shave Club.
To ensure those standards are met, the small Dollar Shave QA team relies on software testing as part of a DevOps approach. Unfortunately, the team was spending too much time managing tests. “We were doing a mix of manual testing and some automation. For the website, it would take us more than 6 hours to do a full pass of manual regression tests across four browsers,” says Nabhani. “And the automation suite took up to 4 hours to run.” Additionally, the automation solution wasn’t scalable. “As we added more tests, the suite took longer and longer to complete,” he says. “Given that we deploy code more than fifteen times per day, our testing must provide feedback to developers as quickly as possible. At our prior rates, it was difficult to do full coverage for every code deploy. We ended up reducing the scope of tests or limiting it to a few tests, which means we ended up missing some bugs.”
To increase its test coverage while reducing test execution time, Dollar Shave decided to fully enable its DevOps model by searching for a new automated testing solution.
The Solution
After considering several options, including building its own solution, Dollar Shave Club chose Sauce Labs, a cloud-based testing solution that supports the organization’s Selenium and Appium tests.
“Sauce Labs is an out-of-the-box solution, and the Selenium and Appium support was a big selling point,” says Nabhani.
Using 50 Sauce Labs concurrent instances, the Dollar Shave Club QA team runs automated functional, regression, and configuration tests in parallel for its web and mobile applications. Each day, the team runs more than a hundred unique tests on desktop web, mobile web, native iOS and native Android. Each test runs several times a day, on multiple browsers and platforms. “Running multiple VMs concurrently is a huge win for us,” says Nabhani. “The bottleneck used to be the number of tests we could run at the same time, and that bottleneck is gone thanks to Sauce Labs.”
The Results
Using Sauce Labs, Dollar Shave Club is enabling DevOps, which requires automated testing. Because Sauce Labs integrates with Dollar Shave’s Jenkins continuous integration system, the company can commit code and have it automatically tested. “That is definitely key, because that leads to continuous delivery, which is what we’re aiming for,” says Nabhani. As a result, the QA team can deploy code 15 times a day and more quickly deliver software updates to customers.
Dollar Shave Club has also significantly reduced its testing time. “It used to take us 6 hours to do a full regression suite, and now we can do it in less than 10 minutes using parallelization in Sauce Labs,” says Nabhani. “So we’re now getting results back within 10 minutes of a commit, which is great. We’re saving about 30 hours of developer time a week that used to be spent on testing time.”
Because the company can catch bugs earlier in the testing process, it is saving a substantial amount on testing costs. “In total, we are saving hundreds of thousands of dollars a year using Sauce Labs, because of the reduction in testing time and time savings for our developers,” says Nabhani. “These savings allow us to invest more in areas that have a direct impact on creating memorable experiences for our members.”
As its business continues to grow, Dollar Shave Club plans to expand its use of Sauce Labs, particularly for its native mobile applications. “Getting our mobile test automation up to the same scale as our web testing will be critical for our future,” says Nabhani. “We are excited to do that with Sauce Labs.”
