Running in slow motion?
Are you running, but can’t make your feet move as fast as you want them to? This is a common feeling among beginners, as well as experienced automation developers. As your regression suite grows, it takes longer to run tests, and soon you have a problem because your regression suite is running longer and longer. There are a few approaches to smarter testing: reduce the number of tests, only run the tests applicable to the change, or optimize the test execution.
Marching toward continuous integration
As your software development team marches toward Continuous Integration (CI), the process involves a lot of automated testing. Even automated testing consumes precious time, so you need to ensure your automated tests are designed to scale. If a particular change is going to cause one or more tests to fail, the team needs to know about it as quickly as possible. Developing scripts to be lean and independent allows for fast feedback to developers.
Let’s unleash the power of parallelization
Use parallelization to speed up slow automated UI tests, and set a standard to develop lean and independent starts. A single cucumber scenario can easily take minutes to run. When you have a lot of scenarios, they can quickly compound your suite and take several minutes or hours to complete. No one wants slow automated tests — tests so slow that they only run a couple of times per day. Everyone expects automated tests to be launched for every build and send feedback within minutes, not hours. Set a standard for every build: the test execution must complete and send feedback within 10 minutes.
One of my favorite quotes of all time:“Failure is not fatal, but failure to change might be” - John Wooden It’s time to change, and focus on how you optimize your test execution by unleashing the power of parallelization. Before starting, let’s take a step back and outline the design we will be using to achieve parallelization with cucumber, only indicating with "update" or "new" the necessary modifications to enable parallel testing using Sauce Labs.
my-regression |-- features/ |---- step_definitions/ |------ one_step.rb |------ two_step.rb |---- one.feature Set up for non-rails project by installing parallel_tests gem and running bundle from project directory:
(update) |---- two.feature
(update) |---- support/ |------ env.rb
(update) |------ sauce_helper.rb
(new) |-- Gemfile (update) |-- Rakefile (update)
$ gem install parallel_tests # go to your project dir $ cd ~/work/github/my-regression $ bundle update Testing in parallel within minutes for a single browser:
$ parallel_cucumber features/ -n 3 The regression suite has been parallelized, and now tests can be run in a fraction of the time. That’s only the beginning of great things to come. You can do more in parallel! What about cross-browser testing? It will take more than a few minutes to set up parallel testing locally with different browsers and environment configurations, and a checkout selenium grid. If you don’t want to build and support your own infrastructure grid consider a cloud-based vendor such as Sauce Labs. Just a few benefits of testing in parallel:
- Under the hood, parallelization automatically divides tests across processes
- Cross-browser and version compatibility
- Provides fast feedback within minutes, not hours
- Understanding the importance of lean and independent tests that will scale
Measure Single Thread vs Parallelization
Let’s measure the performance of ‘x’ cucumber scenarios from a single process and slowly add processes until it’s fully scaled. For example, if you have ‘x’ cucumber scenarios that take 30 seconds each, then with 10x virtual machines (VMs), the parallelism will spread the browser tests across 10 processes (evenly distribute them), so that each process spends an equal amount of time running tests. All of this happens automatically, without any extra setup required, and your tests will be continually rebalanced to run as quickly as possible, as your regression suite grows. This graph is a perfect example of the importance of parallelization: 20 feature files (2 scenarios per feature) with a total of 40 cucumber scenarios: As your regression suite grows, the great thing about parallelization in the cloud is that you are able to scale up your virtual machines within minutes.
- You’re no longer running in slow motion as long as you unleash the power of parallelization
- Increase throughput with parallel testing
- Parallel processing techniques reduce test execution
- Scriptless optimization
- Pushing to production more often with confidence in your application
- Solid cross-browser and OS platform validation
Greg Sypolt (@gregsypolt) is a senior engineer at Gannett and co-founder of Quality Element. He is a passionate automation engineer seeking to optimize software development quality, coaching team members how to write great automation scripts, and helping testing community become better testers. Greg has spent most of his career working on software quality - concentrating on web browsers, APIs, and mobile. For the past 5 years he has focused on the creation and deployment of automated test strategies, frameworks, tools, and platforms.