Overcoming Challenges When Moving Toward Parallel Testing

As we know, parallel testing refers to the process of running multiple automated test scripts simultaneously against various configurations. This strategy provides numerous benefits that can assist a development organization in delivering a high-quality product without slowing the speed of delivery.

It’s easy to understand the benefits of parallel testing. By running multiple tests at once, you get faster results.

It’s also easy to set up basic parallelization. Some tests are straightforward to run in parallel.

Where things get more challenging is finding out ways to run as many tests as possible in parallel. To be sure, you probably won’t be able to run every single one of your tests at the same time. But if you are looking for ways to increase your rate of parallel testing, this article’s for you. Below, I explain the benefits of parallel testing, discuss common parallel testing pain points, and describe ways in which these pain points can be overcome to increase the adoption of parallelization in an organization’s testing model.

Parallel testing provides several benefits that are difficult to realize within a standard automated testing strategy — the first of which is a significant reduction in the time it takes to fully test your application. Historically, each instance of an automated test script would be run in a sequential manner. This means that each test script would be run one at a time for each environment configuration with which you intended to test. This could quickly become very time-consuming when integrating your test set with your build process, which could execute many times throughout the day.

Parallelization resolves this issue by running the same test, at the same time, in multiple VMs of varying environment configurations. By running the same test script simultaneously on multiple machines, the time it takes to complete execution of a single test script is reduced by nearly X times, where X represents the number of machines on which you are testing. For instance, if you are running a test against five specific browser/OS combinations sequentially, then the amount of time it will take to complete testing will be equal to the amount of time it takes for the script to execute five times. But with parallel testing, if you have five available VMs to account for all of these browser/OS permutations, then the time it takes to complete testing will merely be equivalent to the single longest-running execution of the script.

In addition to saving time, parallelization also allows for a significant uptick in test coverage. By making the move to parallel testing, you can utilize the reduction in testing time to execute these same scripts against additional configurations. This will help to ensure that your application functions as designed for all of your users, regardless of their environment.

Everything in life comes with challenges, and implementing parallel testing is no exception. It is likely that your organization is implementing parallel testing in some cases, but avoiding the move to parallelism in others. One of several notable challenges could be the reason why, which I discuss below.

Implementing an effective parallel testing strategy — one that comes with all the benefits described above — requires a commitment to making sure that your efforts are focused on the right things. Some organizations choose to implement their parallelization strategy by maintaining their own infrastructure and executing in parallel, utilizing their own setup. It’s possible that this is not the most efficient use of your time.

Maintaining your own infrastructure in this sense can place limits on how successful your parallel testing strategy can be. For instance, it’s unlikely that you will be able to build out an infrastructure that can provide the basis for test coverage similar to that of the top cloud-based testing providers. Companies such as Sauce Labs can reliably provide VMs configured with hundreds of browser/OS combinations. And even if the capability for test coverage within your own infrastructure was initially equivalent to that of a company like Sauce Labs, it may be more cost-effective to simply subscribe to a cloud-based testing grid rather than to allocate your resources to maintain and support such an infrastructure.

In short, outsourcing the responsibility of maintaining a testing grid with many available configurations can allow you to focus on what’s really important, such as getting as much out of your testing strategy as possible — that is, maximizing test coverage and reducing the time it takes to execute your tests.

Another common issue organizations encounter when moving towards parallel testing is running into scenarios where a reliance on certain test data makes it difficult to integrate these tests in a parallel testing strategy. Consider the following case:

You have a test script that was written to sign up an existing user for certain email alerts. This script makes some assumptions. It assumes that the user exists, and that they are not currently configured to receive these email alerts. But when running tests in parallel, it may not be possible to ensure that this user exists in such a state. Maybe a test script running in another VM configured this user to receive these alerts already, or maybe another test script was executed that removed the user from the database. This was not an issue when running these scripts sequentially, since you knew the order in which the scripts would execute and you knew the manner in which the data would be modified, allowing you to assume the state of the test data at each turn. Now, as you attempt to parallelize your test set, you can no longer assume the state of the data, and you’ve run into a roadblock for folding these scripts into your parallel testing process. That being said, a solution does exist.

In order to parallelize test scripts successfully, the scripts need to be truly independent of one another. One way in which you can ensure this is the case is to have your test scripts manage their own test data. The test script should begin by configuring the data that it requires in order to complete successfully, allowing you to once again be able to rely upon the state of the data as other test scripts execute in a parallel environment that utilizes the same database. Existing test scripts that used to run sequentially should be modified to be independent of one another in the same manner, allowing them to be folded into your parallel testing strategy.

Parallel testing can be an extremely effective and efficient way to ensure application quality, but it certainly comes with its own set of challenges. Analysis of your parallel testing strategy and consideration of potential pitfalls can go a long way towards furthering parallelization within your organization. This can involve getting in the weeds with efforts to refactor test scripts to ensure proper test data management, or even making higher-level decisions to utilize cloud-based testing grids rather than maintaining your own test infrastructure. Keep in mind that every automated testing strategy is a work in progress, and over time, you can continue to evolve yours to provide excellent test coverage while saving time and resources.

Scott Fitzpatrick is a Fixate IO Contributor and has 7 years of experience in software development. He has worked with many languages and frameworks, including Java, ColdFusion, HTML/CSS, JavaScript and SQL.