Continuous Testing In Practice

Cloud-based load testing service BlazeMeter released a white paper that discussed continuous testing last week. The ‘Continuous Testing’ platform leverages cloud and open source technology to bring better testing for DevOps and Agile practices enabling shorter release cycles with higher quality releases.

You might not normally consider back-end load testing as part of a continuous delivery process, but we believe in testing all the things. By pairing BlazeMeter’s service with Sauce Labs’ readily available test cloud used for functional and unit testing and others such as performance testing with New Relic, users get access to a full spectrum of tests that help ensure the utmost application quality- and that’s a big win for everyone.

To learn more about this continuous testing solution:

For more information about BlazeMeter's newest offering, see below - or read the full press release here.

Now, testing can be done in a matter of hours, not days, eliminating friction and human error, and allowing teams to immediately identify and address any problems.

Continuous testing as a service using BlazeMeter's testing platform, as well as industry standard tools and services such as Sauce Labs, Jenkins, GitHub and New Relic, employ the steps outlined below. Together with Sauce Labs, the two companies provide a complete continuous testing as a service solution for the entire testing lifecycle based on open source software and cloud based services.

Testing starts by the developer using BlazeMeter and/or Sauce Labs to develop a comprehensive set of performance and functional tests that completely validate the quality of the developed module. This set of tests will be used in later stages to continue the validation of the module in the continuous delivery process.

Tools, Scripting and Overall Test Creation BlazeMeter's performance and Sauce Labs' functional testing platforms support reading scripts and running tests with 'DevOps' friendly leading open source tools (e.g. JMeter, Selenium) and languages, such as Python, Ruby, or even Shell Scripts. The testing solutions provide libraries and APIs to create "homegrown" tests using common DSLs. With the new continuous testing service, the test creation and the running of the tests can be done with a robust API – everything can be automated regardless of the system you are using. Additionally, developers can replace scripts and use industry standard JSON files that all developers are familiar with, to configure the load test. This reduces the developer's need to learn a new language and allows them to perform tests much more quickly. Ideally, test scenarios and accompanying test codes are pulled directly from a source control tool such as GitHub.

Provisioning & Resources A set of tests can be run with different provisioning. For example, a developer can run a module test from behind the firewall with local resources. On production, the same test can come from multi-geographies from the public cloud with or without load. When a test configuration is executed in an environment (e.g. Dev, CI, Pre-Prod, Post-Prod), it can easily adopt a different provisioning scheme. With BlazeMeter's continuous testing as a service, users can now use the same tool and test definitions across the board, regardless of how they want to interact with the system. The company provides a single common platform regardless of how a user wants to interface with it, while accommodating both GUI and API access.

For an organization to run all required tests in parallel with zero time to test, sufficient resources need to be allocated for all teams according to their use case. BlazeMeter's continuous testing on any test configuration, as well as on-demand, using simple API calls and running in parallel supports:

Reporting BlazeMeter and Sauce Labs' testing environments seamlessly integrate into existing reporting solutions such as Jenkins Performance Trend and existing standard open source and cloud-based tools in the continuous delivery lifecycle. The combined solutions provide "pass"/"fail" trend reports and "deep dive" reports for diagnostics and analysis. The testing solutions also overlay data from 3rd party systems such as New Relic to present a comprehensive picture.

Alerts In Case of Failures The testing environments can indicate a "failure", gather all test artifacts and immediately send them to the group of interested people (e.g. the developers). The solutions provide an alerting scheme per developer, module and project that allows running the exact failed tests all over again to identify the cause for failure.

Modules, Builds, Release Candidates, Releases and Production Taking a snapshot of a build, release or production, provides different modules at different releases. To be able to comprehensively test every such snapshot, BlazeMeter can combine different test configuration fragments into one test configuration. The continuous testing solution now also granularly defines test result thresholds with pass/fail results saved in XML format and consumed by JUnit.

Version Control Friendly Incremental Testing BlazeMeter's continuous testing as a service framework is version control friendly and supports incremental testing to associate test configurations, set of tests and tests with versions (e.g code, build, RC, releases).

BlazeMeter's Continuous Testing as a Service platform is immediately available.