Automating Concurrent Tests

Set Yourself Up for Successful Concurrent Testing

The benefits of running tests at high concurrency are clear: the more tests you run at a time, the faster your build will complete. By "concurrency" of course we mean the number of tests that are active at any given time. Your concurrency is easy to determine: for Selenium and Appium tests, it is the number of browsers and/or mobile emulators or simulators you are able to run tests on at once. There are many challenges involved in running tests at high concurrency, from choosing the right test runner to ensuring you don't accidentally run a Denial of Service (DoS) attack against your stage environment!

Here are some useful tips to ensure that you're able to fully take advantage of the benefits of parallel testing:

  • Ensure your tests are logically independent. If your tests have dependencies (i.e., one test sets up state another test depends on), you will not be able to run them in parallel, because when you run in parallel, you're never sure exactly which test will execute when.
  • Ensure your test runner can handle running parallel tests. Test runners like Python's Nosetests, for example, support a --processes flag where you can simply pass it the number of tests you want to run in parallel. For other languages, like Node.js or Ruby, it can take some effort to find parallel support for your runner.
  • Ensure that your stage environment can handle the load of so many Selenium or Appium sessions hitting your servers. If you are running at 100 concurrency, but your server can only handle 25 connections, your tests will fail because they will take your server down!
  • Ensure that you have enough VMs, browsers, mobile simulators and emulators, or devices to handle your desired concurrency. It can be challenging to orchestrate all these pieces on your own, which is where using Sauce Labs comes in quite handy---we have hundreds of platforms available without you having to set up a thing!
  • Ensure that you are not doing more work than necessary to set up each test. If you have to create an independent user for each test case, create a private, build-only API endpoint that allows you to set up a user with whatever state is required, rather than putting that setup logic as Selenium or Appium steps, which will take vastly longer.

Keep these tips in mind and you'll be well on your way to successful massive parallelization of your test suite and experiencing all the benefits of parallel testing!