What Does Continuous Testing Mean in Practice?

Posted Jun 26th, 2018

The idea behind continuous testing is clear enough: It means you perform software tests throughout the CI/CD pipeline.

But what does continuous testing mean in practice? How do you actually achieve continuous testing? Let’s explore in this article.

What Is Continuous Testing?

Continuous testing is a way of applying the concept and practice of continuous software delivery to software testing.

Just as continuous delivery means that you write, build and deploy code on a continuous (or nearly continuous) basis, continuous testing involves performing software tests at every stage of the CI/CD pipeline.

Continuous testing is different from integrating testing into your pipeline but reserving it for only certain stages of the pipeline, such as pre-deployment. When you test continuously, you test on an ongoing basis, from unit and integration testing to production testing and live testing.

The benefits of continuous testing are similar to those of continuous software delivery: It provides greater agility, more visibility and an increased ability to scale. It also keeps testing routines in sync with the rest of the delivery pipeline, which mitigates the risk of delivery delays and software quality problems.

Continuous Testing in Practice

How do you translate the ideas and goals described above into actual practice? The following steps can help your DevOps team get started with continuous testing.

Identify Which Tests to Run Continuously

In practice, not all of your software tests will need to be performed continuously as part of the CI/CD pipeline. For example, it probably doesn’t make sense to perform user experience testing or exploratory testing on a continuous basis; those tests can be performed less regularly without undercutting application quality.

On the other hand, automated tests that verify application compatibility with different software environments, or that check for software quality problems in production, should be performed continuously in most cases.

Generally speaking, if even a minor change to your application code could change the outcome of a test, you should perform that test continuously. Higher-level tests that focus on issues not tied to specific code changes can be performed outside of the CI/CD pipeline.

Adopt Multiple Test Suites

In some cases, it can be helpful to implement multiple test suites to integrate into your CI/CD pipeline. You can set up a lightweight test suite to accompany minor software releases. If only a small amount of code changes, a lightweight set of tests is usually sufficient, and will save you time. For larger application changes, you can maintain a separate suite of tests that provide deeper coverage but take longer to complete.

Having multiple test suites ready to go allows you to tailor your continuous testing workflow to the size of each release that rolls down your CI/CD pipeline in order to strike the right balance between coverage and speed.

Use Simulators Wisely

In order to maintain the rapid pace of continuous delivery while performing continuous testing, you’ll want to take advantage of simulators for setting up test environments. Simulators will help you to complete tests more quickly.

This is not to say that real-device tests do not have any role to play in continuous delivery. In some cases, if the accuracy that only real-device tests can provide is important, then by all means, perform real-device testing.

By and large, however, simulators are an important enabler for continuous testing.

Design Scalable Tests

Scalability is an important component of continuous testing. You don’t want to integrate a series of tests into your CI/CD pipeline, only to find that they can’t scale to accommodate a larger code base or a more complex CI/CD workflow.

For this reason, when implementing continuous tests, design them for scalability. Automated testing will go a long way toward helping you to scale tests seamlessly, but other factors matter, too. For example, you’ll want to write your tests in such a way that you can modify them easily to accommodate a larger workload without having to rewrite tests from scratch. You’ll also want test infrastructure that can deliver as many resources as you need in case you need to run more instances of the same tests when you scale up.

Automate Testing Decisions

Effective CI/CD pipelines run on automation. This means not only that your tests themselves should run automatically, but also that decisions about what to do as a result of a test should be made automatically. You need to plan ahead of time whether to stop a release in response to a failed test, for example.

Trying to make these decisions manually will slow down your delivery pipeline and undercut the value of continuous testing.


Putting continuous testing into practice can be challenging, as is translating any abstract idea into reality. But with the right approach, it’s quite possible. The strategies listed above are just some of the practical strategies that QA teams can adopt in order to get started with continuous testing. For a more detailed look at getting started with continuous testing, check out the free whitepaper “Core Elements of Continuous Testing.”

Chris Tozzi has worked as a journalist and Linux systems administrator. He has particular interests in open source, agile infrastructure and networking. He is Senior Editor of content and a DevOps Analyst at Fixate IO. His latest book, For Fun and Profit: A History of the Free and Open Source Software Revolution, was published in 2017.

Written by

Chris Tozzi


Continuous testingSimulators