Parallelization helps Animoto decrease test suite runtime from over 3 hours to just 30 minutes.
New York City–based Animoto is the leading online video maker for consumers, educators, photographers, and businesses, who use Animoto web and mobile applications to create and share professional quality videos using pictures, video clips, words, and music.
Video is a very competitive climate, so the team aims to release new feature updates to their apps many times a day, continuously. Continuous integration (CI) and continuous delivery (CD) are vital to that effort. Also, an extensive set of tests are run daily using multiple browsers to ensure that the environment meets the company’s quality standards. “Occasionally, regressions would be found at the end of the development cycle that would result in postponing a release,” says Jay Sirju, Software Development Engineer in Test at Animoto. “We needed a solution to automatically frontload some testing to ensure that regressions are found as early as possible in the development cycle.” Animoto also wanted to find an automated testing solution for its iOS app, because its previous solution was too hands-on.
In mid-2013, the company ported its tests to Selenium and Appium and began using Sauce Labs. The company’s test engineers effectively recreated their tests from scratch. “We look for several attributes in a testing tool, including maturity, stability, portability, scalability, availability, community adoption and support,crossplatform applicability, and cost. Sauce met us in every way,” says Siddharth Jain, Vice President of Engineering at Animoto.
Animoto uses Selenium to run automated testing of its web application against various desktop browsers with Sauce Labs, and utilizes Appium to run automated tests for its iOS application, as well as to test the web application against mobile browsers. Armed with 15 VMs, Animoto runs 5 VMs at any given time during the working day and uses all of them in parallel for nightly full regression runs. As part of the nightly runs, the company’s web application test suite executes more than 200 tests that run on Sauce Labs against both the Chrome and Firefox browsers.
For its iOS application, Animoto runs 50 Appium regression tests against iPhone and iPad simulators. The company also executes eight long-running smoke tests against the iOS app as a core part of its CI workflow whenever a developer issues a pull request or commits a change to the master branch. These tests evaluate full end-to-end user scenarios. To run its tests continuously, Animoto integrates Sauce Labs into Jenkins, the company’s CI system.
Animoto has significantly reduced test time. "By using Sauce Labs to run 15 tests in parallel, we cut a lot of time out of the iOS testing process," says Sirju. "For our iOS app, we need to create and test a build to submit to Apple for certification". That process used to take us a full week, but with the automated regression and smoke tests we're doing with Sauce Labs, we've cut it down to one or two days. Also, the serial runtime of the iOS test suite, typically more than three hours, is only 45 minutes with parallelization.
Animoto can also catch regressions early in the development cycle, which gives developers fast feedback whenever a change has been committed. But the most important benefit is that Animoto can run more tests more frequently, so iOS developers spend less time fixing issues. "We don't have as many critical defects to address, so we don't need to do as many hotfixes and patches on the iOS side," Sirju says. "That gives us more time to focus on developing new features, which is ultimately what we want to be doing."