Prior to working with Sauce Labs, Mozilla had built own their own Selenium grid, but finds it costly and time-consuming to build out to capacity, and problematic to maintain reliably.
With Sauce’s cloud powering Mozilla’s tests, the QA team is able to dramatically reduce time and energy spent worrying about infrastructure, and run more reliable test-suites at a much larger scale. Running tests on Sauce allows them to reduce the test-execution time by up to 80%, and to identity and troubleshoot problems more easily and quickly
The Challenge
Mozilla, a leader in the open source community, strives to promote openness, innovation, and opportunity on the Internet. As a non-profit, Mozilla is dedicated to advancing the goals of the Open Web. As the creator of such products as the Firefox browser, and Firefox OS, Mozilla needed to quickly and reliably find failures and scale their testing.
Mozilla’s Web QA team has built a local Selenium grid, but don’t have a way to dynamically restart licensed VM’s, and those, along with the browser instances themselves, often crash. With team members around the globe, many employees have to use VNC over long-distance, slow-performing connections to access machines for restarts or maintenance, which is time-consuming and annoying.
The test-execution concurrency they get using py.test as a runner can only scale horizontally, which takes time and capital to build out additional nodes. Even with a very large in-house grid, running tests locally was slow and unreliable. When tests failed, they had to examine stack traces, which made it difficult to determine issues.
The Solution
To reduce time and resources spent maintaining their grid and increase test reliability, Mozilla’s Web QA team turned to Sauce Labs and its Open Sauce plan, which provides unlimited testing resources for OSS projects.
Using Sauce helped the Web QA team become a leader in the testing community, in part by giving them access to reporting features they didn’t have with a local Selenium grid, especially videos. Sauce Labs’ HTML reports and screenshots were sought-after features that the team wanted so much that they built them into their own infrastructure. Being able to look at tests more in-depth allowed them to identify issues faster and facilitated communication by allowing the global team to view videos wherever they were.
Moreover, using Sauce allowed Mozilla to not only grow their own testing capabilities, but to support the open source community as well. When Mozilla developed their pytest plugin, they included hooks for Sauce, which allowed members of the community to do exactly what Mozilla does for testing. Building in hooks for Sauce gave developers in the community the same flexibility and scaling that Mozilla had.
The Results
Testing on Sauce saves the Mozilla QA team significant time and stress by freeing them from worrying about infrastructure. With 18 projects and 80 suites, Mozilla runs thousands of tests a day, and has found that using Sauce is the most reliable way to test. Not only does that save time and resources, but their tests also run faster. Tests on Sauce run up to 80% faster than local tests, with test suites that took 8 minutes locally running in just 2 minutes on Sauce, thanks to greater parallelization. Despite having a very large grid, they find Sauce gives them better performance and is hassle-free.
Through Sauce, Mozilla is able to quickly expand the number of browsers and OS versions they test on, allowing them to ensure they are maintaining their dedication to high quality code for the community, all with simple command lines. They are also able to test on mobile more reliably, thanks to Sauce’s Android support, and for future testing needs, iOS, too.
