Getting Started With Visual Regression Testing Using Intern and Sauce Labs
- Full testing lifecycle support: Unit, functional, integration, and system testing
- Author: Modern JS (ES5, ES6+, TypeScript, etc.) with any toolkit or framework
- Multiple environments: Run tests in browsers, Node.js, and in the cloud (e.g. Sauce Labs, Selenium Grid)
- Code coverage: Full statement, line, branch, and function coverage data
- CI: Jenkins, SonarQube, TeamCity, Travis CI, Bamboo, Codeship, Codecov, etc.
- Visual and performance regression: Visual diffs and performance degradations
- Accessibility compliance: Verify compliance using Tenon and aXe
Thanks to a recent grant from the Mozilla MOSS program, Intern recently added support for performance benchmarking, accessibility testing, and visual regression testing. Here we will focus on visual regression testing with Intern while leveraging Sauce Labs.
Visual regression testing
Visual changes are notoriously difficult to test. Even the best functional test suite cannot tell you if a page looks correct. Instead, Visual Regression testing takes a screenshot of a web page (or state within a webapp) and compares it to a previously taken screenshot (referred to as a baseline) to see if the contents have changed, creating a report immediately revealing what is different. This helps engineers to:
- know when a change to CSS results in undesired visual changes elsewhere in the application
- ensure a page is rendered consistently across browsers
- put a quick set of tests around a legacy system to identify regressions, where it may be difficult to write tests for old code that is not well engineered
Sauce Labs with Intern
With recent versions of Intern, it’s now easy to use the intern-cli tool to get setup.
First, you’ll need an account with Sauce Labs. Assuming you have a Sauce Labs account, you can find your access key in your user settings page, under “Access Key”. It should be a 32-character string, like “1234abcd-12ab-34cd-12cd-1234abcd1234”.
Once you have a cloud testing account, you can use intern-cli to quickly create a test config and a few simple tests:
- Install intern-cli:
npm install -g intern-cli
- Create a new, empty project directory and install Intern:
npm install intern
- Use intern-cli to create some starter tests:
- Edit the intern config:
- Set the tunnel property to ‘SauceLabsTunnel’
- Set the tunnelOptions property to
If your Sauce Labs credentials are already provided via environmental variables, the
tunnelOptions properties for credentials is not necessary. In general and as a reminder, supplying these credentials via environment variables is advisable in order to avoid accidentally checking them into a source code repository. The username and access key are being added to the configuration file for example purposes only.
The project directory should now contain a tests directory with a starter Intern configuration, along with one unit test suite and one functional test suite.
To run the sample functional test suite with Intern, run the command
intern run -w. The first time you run the tests, you’ll see a tunnel download progress message, then something similar to the following:
These messages indicate that the service tunnel has successfully started and that a session was created on the remote testing service. At this point, Intern will start running tests on the test virtual machine. The default reporter will display the IDs of tests as they pass, showing a summary of the test results when the tests have completed.
And with that, you’re running Intern tests on Sauce Labs!
For further reading, the Intern guide contains a significant amount of information on setup and configuration.
Visual Regression testing is provided via a plugin to Intern. To get started, first use
npm install intern-visual to install the plugin. Second, configure the plugin as a package with your module loader.
Writing visual regression tests
Visual Regression tests are written for the WebDriver engine and can be written programmatically or via configuration.
Configuration-driven tests are ideal for situations where the goal is simply to verify that a web page hasn’t visually changed. These tests are very simple to author and are a great way to quickly create tests.
The above example instructs intern-visual to create a test that visits the provided url, resize the browser to the provided
height value, and create a baseline image snapshot if one does not already exist with the
However, there are times when we need more control over the page, the application we are testing, or when additional steps need to be taken during a test. These additional steps could include inputting and responding to text prompts, hiding parts of the DOM that change (e.g. ads), or interacting with the page. In these cases, the programmatic API provides a command helper that expects a screenshot as a parameter.
In the above example,
assertVisuals is a command helper that accepts a reference to the current test and various options, returning a function that expects a screenshot from the takeScreenshot command.
The programmatic code shown above does virtually the same thing as the configuration-based visual test, except the WebDriver commands are defined explicitly. Both of the above tests search for a baseline image on the local machine, generate one if necessary, and provide a pass/fail to Intern along with some additional information about the visual comparison.
Visual regression test reports
Reporters are Intern’s mechanism for providing test results from a variety of sources in a variety of formats. The intern-visual test and assert modules are part of the test and assertion layer of Intern, providing a pass or a fail notification. To get a visual HTML report, simply add the
VisualRegression report to Intern’s list of reporters:
The VisualRegression report will write an HTML report showing any visual regressions in the visual-test directory. It will not output any failures to the console, so be sure to include your preferred console reporter if you would like to see pass/fail information in the console. When a visual regression test fails, it will appear in the report with the differences overlaid in red, as shown in the below example.
This is an early release of intern-visual and we have plans to further improve our approach. Our plans include improving the configuration-based visual testing to make it easier and faster to write good visual tests using less code. Also, the current image comparison algorithms are best for comparing two identically sized images due to how image differences are identified. We want to improve these algorithms to be more tolerant of minor differences caused by anti-aliasing, allowing for minor differences between browsers or between different window sizes, ignoring all but relevant visual differences.
Visual regression tests provide a quick and easy way to establish baselines for how your application should appear. They are especially valuable when you need to put tests around a legacy application or to ensure that a CSS change has not caused undesired changes. For more information, take a look at intern-visual on Github. Visual regression test examples are available under the tests/visual directory.
- Accessibility Testing
- Appium Resources
- Best Practices
- Continuous Delivery
- Continuous Integration
- Continuous Testing
- Cross Browser Testing
- Guest Blog Posts
- Load Testing
- Machine Learning
- Mobile Development & Testing
- News & Product Updates
- Open Sauce
- Open Source
- Performance Testing
- Product Updates
- Quality Assurance
- Quality Engineering
- Sauce Product Info
- Security Testing
- Selenium Resources
- Software Development & Testing
- The Story of Sauce