Engineers saved 10 developer hours per week per person and reduced build time by 86 percent. In addition, debugging is now five times faster.
The Challenge
Yahoo, one of the world’s most visited Internet companies, is focused on connecting people to what matters most to them, across devices and around the world. The company offers a popular web portal, search engine, the Yahoo Mail email service, and additional services for hundreds of millions of people.
The organization is committed to streamlining its processes to maximize efficiency and ensure quality across all platforms. For the Yahoo Mail department, that commitment includes a focus on automated testing to support its continuous delivery (CD) and continuous integration (CI) emphases. All of the department’s front-end engineers run tests on various browsers locally and in the CD pipeline, using the Selenium open-source testing framework.
Before Sauce, the Mail team managed 20 VMs manually, which meant only 20 tests could be run in parallel. Needing to scale its testing environment to ensure excellent build quality, the team bumped its VM count to 100, but soon found that managing the internal infrastructure was too time-consuming. Even after the department started testing in parallel, it still lacked the scalability it needed, and the team could only test at scale in parallel with a few browser and OS combinations. While the grid had some IE 10 VMs, it did not have enough VMs to support Chrome and Firefox. And by hosting its own grid, the team was bombarded with manual tasks like having to upgrade or close browsers when builds were interrupted.
The Solution
Yahoo Mail chose Sauce Labs, an automated, cloud-based testing solution that offers support for Selenium and gives testers a range of features, such as the ability to test different OS and browser combinations. Previously, the team relied on QA testers to do manual tests. “They spent a lot of time writing different test cases and scenarios. It was a very primitive and inefficient way of doing things,” says Ratan Hudda, Senior Director at Yahoo Inc. However, Yahoo Mail was able to convince Yahoo management that a modern model based on automated testing would be better.
After moving to Sauce Labs, the team’s testers increased the VMs they were using in order to speed up builds and run more tests in parallel. With the time saved, Yahoo Mail developed a new functional testing framework based on open-source technologies Cucumber, Watir-WebDriver, Page Objects, and Sauce Labs, eliminating the hassle of performing updates, writing code to support new technologies, or staying up-to-date with new browsers. Today, all the Mail engineers run more than 10,000 integration and functional tests per day in parallel in 100 builds per day across Chrome, Firefox, and IE 10 against the server. If the builds pass testing, they’re pushed to production. “We have a test case running on Sauce Labs for literally every feature we have in Yahoo Mail,” says Mohit Goenka, Senior Software Development Engineer and Product Manager at Yahoo Inc. “We test different scenarios multiple times throughout the day, which gives us a clear picture of what end users might be doing.”
The Results
With Sauce Labs, Yahoo Mail no longer struggles with maintaining its test infrastructure, and developers save an estimated 10 hours per week each because they do not have to handle VMs or internal grid issues. “By moving to Sauce and making some changes to our build pipelines, we can deliver quality code at a much faster pace. Now, we are able to focus on writing the actual code and tests for the code, rather than worrying about where and how to run the tests,” says Goenka.
Sauce Labs has also reduced testing time on builds from three hours to less than 25 minutes, and the critical debugging process is also much shorter. “With Sauce Labs, debugging is five times faster because it is automated, and we can use the screenshots and other features to see exactly what’s going on with a build,” says Goenka.
Yahoo is also taking advantage of the solution’s automated cross-browser testing, which engineers use to test any browser and OS combination. Sauce Labs also fits in perfectly with the organization’s CI practice. “Sauce Labs works well with our Continuous Integration flow because we have production-replica environments where we run test cases against different browser and OS combinations, and we can ensure each and every feature we can track is fully tested,” says Hudda. “We know everything will work for our end users, and that gives us more confidence in the product we ship.”
