Zillow Cut Testing Time from 8 Hours to 30 Minutes with Sauce Labs
To increase browser variation test coverage, Zillow started using Sauce Labs, a cloud-based testing solution, to run 60 tests per minute catching bugs earlier in the development cycle.
The company reduced its testing time from eight hours to 30 minutes and gained three times as much test coverage of browser variations.
Zillow is a leading provider of online real estate–listing websites. Each day, millions of U.S. home buyers and sellers, renters, agents, and property managers use Zillow sites to browse available homes and apartments.
Delivering a quality experience for its users is critical to the company’s success. “The quality of our product is a direct reflection on us,” says Yusuke Tsutsumi, Senior Software Developer, Zillow. “That’s why software testing and verification are hugely important for us. We need to catch a bug before it reaches a single user.” However, the organization’s internal Selenium automated testing solution, which tested the website against different browsers and operating systems, presented significant challenges. “Trying to keep up to date with all the latest Firefox and IE versions was very time-consuming, and we didn’t have anywhere near the test coverage we wanted for those variations,” says Tsutsumi.
Additionally, developers spent too much time maintaining Selenium machines. “We had a lot of difficulty maintaining our internal cluster, and the cost of doing it just didn’t make sense anymore,” Tsutsumi says. To solve its challenges, Zillow decided to look for a cloud-based testing solution.
It’s mind-boggling to think that we were only surviving on two or three variations of Firefox and IE. Now, we can support the top 10 variations, and we can do it seamlessly with Sauce Labs.
Zillow decided to implement Sauce Labs. “Sauce Labs was the only Selenium cloud-based solution that offered automated cross-browser testing, so we could test any OS and browser combination,” says Tsutsumi. Zillow was also drawn to the solution’s ease of deployment. “The fact that Sauce utilizes the Selenium interface so directly was important,” says Tsutsumi. “We’ve written a lot of tests that use a Selenium cluster service, and the only change we had to make was replacing a URL. Once we did, we were able to access all the features Sauce Labs provides.”
More than 100 developers at Zillow now use Selenium and Sauce Labs for automated website tests. Overall, Zillow supports 13 different test environments that run 42,000 tests every 15 minutes across all environments. Specific to Sauce Labs, Zillow uses 35 Sauce Labs virtual machines in parallel to execute 60 tests per minute. “Supporting all the popular browser variations is critical, so we rely on Sauce Labs to test those,” says Tsutsumi. “When we have a key use case or scenario that we need to ensure works on each variation, that’s where we use Sauce Labs exclusively. Every test environment we have executes Sauce Labs continuously to catch bugs.” Zillow is also taking advantage of the on-demand capabilities of Sauce Labs. “The fact that we can bring browser variations up on demand is a big plus,” Tsutsumi says. “Compared to our internal cluster, which required us to provision a specific number of machines per browser variation, we have a lot more flexibility in testing now.”
Using Sauce Labs, we catch bugs much earlier in the developer environment now, instead of in the about-to-be-released phase. As a result, we can get new features out to end users about a day faster than we could before.
Zillow has significantly reduced its testing time by using Sauce Labs. “With our previous Selenium cluster, our automated test process took about eight hours to run. We’ve been able to bring that down to 30 minutes, and a lot of it has to do with Sauce Labs,” says Tsutsumi. The company’s developers are also able to test more browser variations than they could before. “It’s mind-boggling to think that we were only surviving on two or three variations of Firefox and IE,” Tsutsumi says. “Now, we can support the top 10 variations, and we can do it seamlessly with Sauce Labs.”
Perhaps even more importantly, Zillow can now conduct more tests using different browser variations at a time. “We get about three times as much test coverage as we did before,” says Tsutsumi. “Previously, we didn’t even have test coverage for a lot of browser variations, like the four different IE versions that are popular right now. To have that is huge for us.” For Zillow, that increased test coverage is even more important than faster testing time, because it means that developers can catch bugs sooner. “Using Sauce Labs, we catch bugs much earlier in the developer environment now, instead of in the about-to-be-released phase,” says Tsutsumi. “As a result, we can get new features out to end users about a day faster than we could before.”
Ultimately, Zillow can confidently deliver higher quality code. “We care a lot about giving our users a quality product, and with Sauce Labs playing a major part in our test infrastructure, we can do that,” says Tsutsumi.