Review of Visual vs. Functional Testing with pDiff and Applitools
As development teams eliminate manual testing and focus only on test automation, everyone lives and dies by automated testing. It is repeatable, faster (than manual), and should be preventing bugs, right? How does your automated test coverage compare to manual testing? You have more automated tests for Unit, API, Integration, and Functional UI. But the forgotten test coverage is often Visual testing! Typically, automated testing does not extend checks for visual components of your application.
I have found that teams writing automated Functional UI tests do not understand the difference between these and visual tests. Do you understand the difference? Your functional coverage could be that you've identified the main workflows in the app, and visual coverage validates how specific visual components appear in the app. Or it could be more. Without a clear picture of when to use what and why, you could be missing out, or even wasting time. It is important to separate the two levels of tests. Don't blend the two.
What is Visual Testing?
The goal of visual testing is to catch unintended visual bugs when updating UI components. This activity verifies that the UI appears correctly from the user’s perspective. There is no way to know that a visual component that has been updated breaks by only running a functional automated test. Here is my initial checklist when defining visual test coverage:
- Each UI component appears in the right color, shape, position and size
- Ensure that a component doesn’t hide or overlap any other UI elements
- Responsive content at different viewport sizes to ensure changes made at one screen size won’t break a layout on another screen size
- No broken images
What is Functional Testing?
The goal of functional testing is to cover all workflows of the system by fully exercising the GUI itself. Here is my initial checklist when defining Functional UI test coverage:
- Form validation, i.e. all mandatory fields
- Links on a page
- Search and display the correct results
- Pop-up or confirmation messages
- Create, edit, delete, and update (CRUD) tasks
To expand, check out this complete web application checklist.
Why do you need BOTH?
- Browsers. Vendors are pushing updates automatically and often.
- Screen Resolutions. People are browsing web applications from a variety of devices (mobile, tablet, desktop) for non-responsive applications.
- Responsive Designs. Content at different viewport sizes ensures changes made at one screen size won’t break a layout at another screen size.
Functional changes are easily detected. One UI component change can have unexpected consequences, making visual mistakes hard to spot with only automated Functional UI tests. Visual testing is as important as Functional UI testing. Users nowadays are using all types of devices to access web applications. So testing visual components and functional workflows is important because the user experience on mobile devices is quite different from desktops.
It turns out that there are a lot of Visual testing frameworks available. They all work in different ways — some take screenshots while others aren’t even visual (CSS attributes). Of the many frameworks out there, you’ll find, most notably, DPXDT, Applitools, and BackstopJS.
Applitools is a cloud-based automated visual testing solution that automatically validates all the visual aspects of web, mobile and desktop apps.
- Provides native SDK for development and testing
- Supports multiple browsers and multiple devices
DPXDT (pronounced Depicted) is a tool that compares before and after website screenshots for each release. It shows when any visual and/or perceptual differences are found.
- From Google, written in Python
- Support of PhantomJS by default
- Restful API style of development and testing
- Report supports manual diff confirmation
- Free and open source
- Galen Frameworks
- Rspec Page Regression
To satisfy your missing Visual test coverage, I recommend checking some of these out to see if they fit what you’re doing in your application. What do you want out of a Visual testing framework? I personally want to compare between iterations or releases to test our CSS, layout, graphics and other visual changes on our application.
Let’s see it in action
The perceptual testing frameworks compare between iterations or releases — the new iteration screenshot (staging environment) compared against the golden screenshot (production environment). The tool performs a graphical comparison by generating rendering differences, not using the traditional pixel comparison method. The release manager still needs to manually inspect all differences flagged as failures. The manager will confirm if failed differences are unexpected or pass them as expected changes (creating a new golden image). Takeaways As you can see, visual tests are an important part of test coverage. Every testing strategy should include visual testing. It is critical to coach team members on the difference between Visual and Functional UI testing.
Greg Sypolt (@gregsypolt) is a senior engineer at Gannett and co-founder of Quality Element. He is a passionate automation engineer seeking to optimize software development quality, while coaching team members on how to write great automation scripts and helping the testing community become better testers. Greg has spent most of his career working on software quality — concentrating on web browsers, APIs, and mobile. For the past five years, he has focused on the creation and deployment of automated test strategies, frameworks, tools and platforms.
- Accessibility Testing
- Appium Resources
- Best Practices
- Continuous Delivery
- Continuous Integration
- Continuous Testing
- Cross Browser Testing
- Guest Blog Posts
- Load Testing
- Machine Learning
- Mobile Development & Testing
- News & Product Updates
- Open Sauce
- Open Source
- Performance Testing
- Product Updates
- Quality Assurance
- Quality Engineering
- Sauce Product Info
- Security Testing
- Selenium Resources
- Software Development & Testing
- The Story of Sauce