The Motley Fool Speeds Testing 10X, Slashes Dev Wait Times and Gains Agile Velocity with Sauce Labs
The Opportunity
An important part of testing at The Motley Fool is providing valuable, timely and useful feedback to developers, but the relationship between QA and development is often problematic.
The QA Team at The Motley Fool had already solved one major problem. They had transformed a massive set of smoke tests and regression suites created using Selenium IDE that were inflexible and slow to run, because they were recorded instead of programmed. By shifting to a test coding methodology, the QA Team made their test suites much more flexible, allowing tests to be rapidly adapted as the web applications at The Motley Fool evolved in two week cycles. Flexibility was achieved, but speed - true agile velocity - still proved elusive.
"Being part of the QA team, you're responsible for meeting with the developers and making sure there are no assumptions and you also have to be able to write automated tests and provide feedback to developers that's timely", said Dave Haeffner, Senior Quality Analyst at The Motley Fool. "For the most part our testing cycle consisted of a lot of noise because of the way we were doing things-we needed to speed things up and part of that was looking into Sauce Labs."
The Solution
After attending a session about the history of Selenium at the Agile2009 Conference, Dave Haeffner met with Jason Huggins, Selenium creator and co-founder and CTO of Sauce Labs. Haeffner knew he had found the kind of automated web-testing tool that could potentially accelerate automated testing for his team-Sauce Labs, the cloud-based cross-browser testing solution.
The first potential advantage of using browsers in the cloud provided by Sauce Labs was the increased reliability of an environment specifically designed for testing.
Secondly, since Sauce Labs provided a cloud service, getting up and running would be quick and the initial investment and commitment was minimal.
Third, but most importantly, agile velocity seemed potentially achievable with Sauce. By running Selenium tests in parallel across multiple browsers, developers at The Motley Fool could save time and allow for true agile development and continuous integration.
After redirecting internal tests to browser instances running in the Sauce Labs cloud, Haeffner immediately saw that his team could change their entire QA process-and that the ROI would not just be financial. Often Saucecustomers see that server replacement offers the most clear cost savings, but for The Motley Fool the gains were more intangible: productivity and collaboration massively increased.
The Results
After four months of using Sauce, The Motley Fool completely offloaded their daily smoke testing to Sauce.
Since then, the entire development team at The Motley Fool has increased velocity and saves a large amount of time-every day. Their smoke test suite, which runs twice a day and is most important for base lining results, previously took up to 20 minutes. Now it takes 2 minutes - a 10x speedup. 20 minutes might not seem substantial, but with such a rapid feedback loop, QA was able to integrate with developers and provide timely feedback, improve tests, and move the team further towards the ideal of continuous integration.
"Currently, feedback is given to Developers and QA out-of-sync with their work", says Haeffner. "With this increase in speed, we will now be able to link test runs directly to web server deployments which will provide quick and meaningful feedback."
Productivity increases, time savings and cost reductions are traditional measures of return on investment—but there is highly intangible advantage gained from cohesive agile development and continuous integration. Traditional barriers break down between QA and development, and the ramp in collaboration brings about a true workplace evolution.
The Motley Fool has achieved a scalable, flexible process that is ready for the future of web application development.
"The future of testing will be more dynamic and involve everyone running tests", says Haeffner. "Our goal is to have only one person from QA be on hand for a software release; right now there's 6. In the future, we also hope to have evolved to the point where everyone - business, developers, etc - will help make tests and run them."
About Sauce Labs
Sauce runs Selenium tests in parallel across multiple browsers in the cloud. Sauce records a video of every test run, which developers can use to identify failures and see how the web app looks in different browsers. They can also automate browser tasks via an API and run tests in parallel, saving time and allowing for agile development and continuous Integration.
About Sauce Labs
Sauce Labs, the Selenium Company, provides the Sauce service that allows users to run automated cross-browser functional tests in the cloud. To date, nearly three million Sauce tests have been run in the cloud without QA teams having to maintain or update their own test infrastructure. The lead investor of Sauce Labs is the Contrarian Group, Peter Ueberroth's investment management firm. Sauce Labs is headquartered in San Francisco, California. For more information, visit http://saucelabs.com.