Today Puppet Labs’ testing process is extremely automated. Since using Sauce Labs, their GUI testing time went from 30 days to 3 days, a time savings of over 99%.
To achieve the same results in-house with their own Selenium grid, the estimated cost is a half million in headcount, not including infrastructure costs.
The Challenge
Puppet Labs is the leading provider of IT automation software. Their flagship commercial product, Puppet Enterprise, automates the configuration and ongoing management of services and applications that run on physical, virtual, or cloud infrastructure. Customers range from Fortune 500 companies to startups across all industry verticals, such as finance, technology, and government.
Puppet Labs attempted to host and run an in-house Selenium grid in 2012, but Dominic Maraglia, Engineering Manager, said they had several failures and false positives that arose from their unstable Selenium grid versus their code. While some failures were due to normal challenges associated with a grid running hundreds of nodes, others were due in part to low familiarity with Windows, since the company’s primary focus was UNIX and Linux that year.
The Solution
After initially experimenting with Sauce Labs, Dominic automated a handful of tests. "I couldn't believe it," he said. "I didn’t have to manage Windows boxes, browser versions, and security settings for Windows machines." Because virtually no one on his team had Windows experience at the time either, he knew Sauce would be well received.
"In a startup environment, often everyone reinvents the wheel with their favorite tool, and then you have to support multiple tool chains," Dominic said. He found no backlash when it came to Sauce, however. "Every developer and QA person said, 'this is amazing.' Within a week everyone was using Sauce at Puppet Labs. I turned off our Selenium grid and deleted the VM that it was running on." After moving more tests to Sauce, the team was able to reduce testing time from 30 days to 3 days, a time savings of over 99%.
Today Puppet Labs' test and development environments are highly automated. QA testing for Puppet Enterprise is complicated because of its large matrix of supported operating systems. They use 400 to 600 UNIX, Linux, and Windows VMs to run more than 1,000 UI tests in parallel. In addition to these VMs, they use 20 machines from Sauce to run 20 test suites in parallel continuously with Jenkins, a popular CI system. They use Sauce Connect to work behind their secure firewall, and Sauce is the final check for quality across all browsers. Because they have so many other pieces of their process to manage, Dominic said, "I do not want my team spending time on browser or image maintenance. It’s too expensive and it changes too quickly. I love that Sauce handles that for us."
He estimates 40 engineers currently use their Sauce Labs account. Their large team finds the video debugging tool extremely helpful as it aids communication. Said Dominic, "With Sauce, we experienced an epiphany. It works right out of the box by recording video so we don’t have to look at log files. It helps us debug instantly."
The Results
"Sauce immediately allowed my team to scale our testing from literally the day we rolled it out," Said Dominic. It also allows him to access browsers he hadn't thought about using previously, such as Safari and older versions of Internet Explorer, which would have required the team to manage infrastructure they were not equipped to do. "We’re not browser savvy," he said. "Avoiding this, thanks to Sauce, is a huge win."
To achieve the same results in-house they get by using Sauce, Dominic estimates he'd need to employ several systems engineers to set up and maintain the grid, for a total cost of approximately a half million annually in headcount.
Lastly, he’s found that avoiding the hassle of managing unfamiliar systems has helped retain top engineering talent within his team at Puppet Labs. "If you tell your team of UNIX and Linux people they’ll have to maintain Windows systems for IE use, that could be an issue for some folks," he said. "So I think I saved people from leaving Puppet because Sauce handles this for us."
