HomeAway Scales Testing with Sauce Labs In Shift To Continuous Delivery

HomeAway is a family of 50 websites and hundreds of apps that collectively provide a network of vacation rental properties to more than 65 million travelers each month. To maintain app quality, they are on the path towards continuous delivery through automation and continuous integration.

Over 20,000 tests are deployed in parallel per day to ensure mobile and web app quality in an effort to meet market demand for faster delivery times.

HomeAway combines their own infrastructure and Sauce Labs’ secure, reliable, and highly scalable cloud to test cross-browser, with plans to test cross-platform.

HomeAway is moving towards continuous delivery, Carl says, adding that the only way to do that is with CI and automation. To keep with industry pace, Carl surrounds himself with a team who believes in innovation. They found that Sauce Labs aligned with the direction they wanted to take their automation.

HomeAway leverages Sauce Labs for production monitoring using an internal tool named Green Screen. Developers built Selenium scripts to execute primary customer user flows through Sauce Labs against their family of vacation sites. The objective is for these tests to always be green; however, if a test fails, they receive an alert from Sauce Labs and the company responds.

Additional Selenium tests are executed from Quick Build, but they also developed a layer of abstraction for tests executed on Sauce Labs from their local machines or Team City, their primary CI system. Their builds fire 150 scripts every ten minutes, continuously, across all their applications for a total of over 20,000 tests per day that run in parallel on up to 25 Sauce VMs. A portion of the tests execute through their firewall and against their staging environments using Sauce Connect.

HomeAway is focused heavily on building up their mobile automation efforts. The team chose Appium, an open source, cross-platform test automation framework supported by Sauce Labs, since the rest of their testing is done with Selenium. “Appium is Selenium for mobile, so it was an easy transition for us. It works with Android and iOS, so we can write one suite of tests and run our application on both devices with very little additional work,” says Carl.

Most of the mobile automation to date has been relative to their native app, with future plans to look into mobile web. Currently HomeAway executes mobile tests against an internal Jenkins server with a small set of device form factors hard wired to the server. Maintaining this environment is complex, so HomeAway is working to shift to the cloud for mobile testing to reduce maintenance cost, extend the device coverage, and increase reliability.

HomeAway has found the biggest value gained from using a combination of Sauce Labs’ infrastructure, real-user monitoring, and running their Selenium and Appium frameworks continuously is that quality isn’t compromised.

“Sauce Labs gives us a highly scalable and reliable solution in a diverse, robust environment that we can leverage. Our developers can run their automation on a local machine, on our internal grid, and on the Sauce cloud. We love that flexibility,” says Carl. “Sauce gives us confidence that we are creating the experience we want to create for our customers.”