Eventbrite Catches Issues Sooner and Debugs Faster Using Sauce Labs
Eventbrite, an event management platform that processes hundreds of millions in financial transactions each year, needed a reliable automated testing solution that could ensure the quality of their web applications across multiple platforms and browsers.
The Eventbrite QA team chose to use the open source project, Selenium, for their automated testing framework.
The Eventbrite team selected Sauce Labs as the provider of cloud-based Selenium infrastructure. Unique features to Sauce, such as video and screenshot capabilities plus an easy integration with Jenkins, allowed Eventbrite to drastically speed up test and debugging time while ensuring quality across its website and payment forms.
Eventbrite is an online event management platform that enables easy organization and online selling of tickets to events of any size. Since launching in 2006, Eventbrite has processed more than 80 million tickets through its platform and in the summer of 2012, crossed the $1 billion mark in ticket sales.
Given the heavy flow of financial transactions that pass through the Eventbrite website each day, Quality Assurance (QA) has played an important role in the company from the beginning. As their business has changed over time, so have the QA initiatives. Early testing efforts were done manually, but in late 2010, Eventbrite decided to replace manual QA with an automated system to ensure better crossplatform test coverage and to support a more frequent deploy schedule.
The team chose Selenium as their automated testing tool, but it wasn’t long before the burden of running tests on local machines became too great to ignore. Eventbrite’s QA team wasn’t interested in using resources for setting up and maintaining a Selenium infrastructure themselves, and they didn’t want to be responsible for adding new browsers and versions to the VMs themselves.
Before Sauce, we would have to look at the stack trace and try to figure it out. That usually meant having to run it again. Now if there’s a failure we see the video and screenshots right here. It helps greatly with fixing tests faster.
Eventbrite decided to turn to Sauce Labs and its cloud-based Selenium testing infrastructure for help.
After setting up their Sauce Enterprise account and creating subaccounts so that everyone on the QA team could use it, they began pointing their tests to the Sauce cloud instead of running them locally. With their laptops now free from the overhead of running Selenium test suites, the team could focus exclusively on test scripting and architecture. And they now had access to all the browsers on all the OSes they needed without having to install or patch virtual machines.
In addition, Eventbrite added Jenkins to their testing environment using the SauceJenkins plugin to enable the CI system to kick off builds automatically. This allowed Eventbrite developers to automatically run tests every time they did a commit, helping them not only to catch issues but also debug those quicker than before.
It’s just easier to go with Sauce. If I can write a test and run it on a simulator, I don’t have to worry about installing anything like Xcode. I don’t want to worry about infrastructure or device setup. I just want to use it.
O’Neill says Eventbrite immediately noticed a difference in the speed of development, testing, and debugging time. Thanks to the video, screenshot and reporting capabilities that Sauce Labs offers, the Eventbrite QA team was able to feed the test artifacts directly into their CI report page and more easily identify why a test was failing.
These days, Eventbrite has a smoke test job of 20 tests that runs on every build and does a quick sanity check. If that passes, it gets handed to the acceptance test job, which runs nearly 700 Selenium tests. They test up to 35 concurrent threads on Sauce Labs at a time (and are working to increase that number), primarily against Chrome, Firefox and Internet Explorer. “Our stability has never been better,” says O’Neill.
Going forward in 2013, Eventbrite is increasing focus on mobile test automation to support the various mobile apps they released in 2012. For this, Eventbrite will continue to rely on Sauce for assurance.