Running your Selenium tests in parallel: Python

This is the first post in our series "Running your Selenium tests in parallel", in which we're going to explain how to set up a concurrent execution environment and considerably reduce your testing times. The first client language we're going to address, as the title says, is Python. To start, let's get a set of Selenium Python tests to use: The tests are stored in a public github project. This set of tests validates our site. It checks basic structure, our login form, our signup form, and our feedback tab (UserVoice powered). The tests are grouped into Python files based on the functionality they address.

So,  the regular way to run these would be to run each file from the command line, like so:

$ python testIntegrity.py $ python testLogin.py $ python testSignup.py $ python testFeedback.py

As a first approach it's not that bad; your tests will run, and you'll get decent output with the results. Execution time: 16.2 minutes 16.2 minutes isn't that bad. The problem is that once your test base starts growing, the time and effort it will take you run these tests will increase considerably. We're talking about hours in some cases. The next step to take will be to write a small script that runs them automatically, so the only thing the user will have to do is to run a single Python script that takes care of the rest:

$ python ordered_run.py

The code in this script is:

import os import glob tests = glob.glob('test*.py') for test in tests: os.system('python %s' % test)

Easier to run, but not much faster... Execution time: 16 minutes

Let's improve this using subprocess, one of Python's multiple process libraries.

from subprocess import Popen import glob tests = glob.glob('test*.py') processes = [] for test in tests: processes.append(Popen('python %s' % test, shell=True)) for process in processes: process.wait()

Now our tests run concurrently, using a separate process per python file (4 processes total). Execution time: 8.3 minutes This is much faster! It reduces the whole execution time from the time it takes to run all the tests one after another to the time it takes the longest of the four sets of tests to end.

Note: One drawback to this method is that the output we receive isn't in any particular order. You can change that by setting the stdout parameter in the Popen instantiation and then concatenating the output in order.

We can further reduce the execution time by running all 14 individual test methods in parallel. The easiest way we've found to do that in Python is to use nose. First, install it:

$ easy_install nose==0.11 multiprocessing

Now, place yourself where your tests are, run nose and enjoy:

$ nosetests --processes=14

Much easier to run (no helper script for this), cleaner output... Execution time? 3 minutes! Do you have a better way to run your tests concurrently? Tell us about it in the comments.