Thanks again to those of you who attended our recent webinar with Applitools on automated visual testing. If you want to share it or if you happened to miss it, you can catch the audio and slides here. We also worked with Selenium expert Dave Haeffner to provide the how-to on the subject. Enjoy his post below.
In previous write-ups I covered what automated visual testing is and how to do it. Unfortunately, based on the examples demonstrated, it may be unclear how automated visual testing fits into your existing automated testing practice.
Do you need to write and maintain a separate set of tests? What about your existing Selenium tests? What do you do if there isn't a sufficient library for the programming language you're currently using?
You can rest easy knowing that you can build automated visual testing checks into your existing Selenium tests. By leveraging a third-party platform like Applitools Eyes, this is a simple feat.
And when coupled with Sauce Labs, you can quickly add coverage for those hard to reach browser, device, and platform combinations.
Let's step through an example.
NOTE: This example is written in Java with the JUnit testing framework.
Let's start with an existing Selenium test. A simple one that logs into a website.
1// filename: Login.java2import org.junit.After;3import org.junit.Assert;4import org.junit.Before;5import org.junit.Test;6import org.openqa.selenium.By;7import org.openqa.selenium.WebDriver;8import org.openqa.selenium.firefox.FirefoxDriver;910public class Login {1112private WebDriver driver;1314@Before15public void setup() {16driver = new FirefoxDriver();17}1819@Test20public void succeeded() {21driver.get("http://the-internet.herokuapp.com/login");22driver.findElement(By.id("username")).sendKeys("tomsmith");23driver.findElement(By.id("password")).sendKeys("SuperSecretPassword!");24driver.findElement(By.id("login")).submit();25Assert.assertTrue(";success message should be present after logging in",26driver.findElement(By.cssSelector".flash.success")).isDisplayed());27}2829@After30public void teardown() {31driver.quit();32}
In it we're loading an instance of Firefox, visiting the login page on the-internet, inputting the username & password, submitting the form, asserting that we reached a logged in state, and closing the browser.
Now let's add in Applitools Eyes support.
If you haven't already done so, you'll need to create a free Applitools Eyes account (no credit-card required). You'll then need to install the Applitools Eyes Java SDK and import it into the test.
1// filename: pom.xml2<dependency>3<groupId>com.applitools</groupId>4<artifactId>eyes-selenium-java</artifactId>5<version>RELEASE</version>6</dependency>7// filename: Login.java89import com.applitools.eyes.Eyes;10...
Next, we'll need to add a variable (to store the instance of Applitools Eyes) and modify our test setup.
1private WebDriver driver;2private Eyes eyes;34@Before5public void setup() {6WebDriver browser = new FirefoxDriver();7eyes = new Eyes();8eyes.setApiKey("YOUR_APPLITOOLS_API_KEY");9driver = eyes.open(browser, "the-internet", "Login succeeded");10}
Rather than storing the Selenium instance in the driver
variable, we're now storing it in a local browser
variable and passing it into eyes.open
-- storing the WebDriver object that eyes.open
returns in the driver
variable instead.
This way the Eyes platform will be able to capture what our test is doing when we ask it to capture a screenshot. The Selenium actions in our test will not need to be modified.
Before calling eyes.open
we provide the API key (which can be found on your Account Details page in Applitools). When calling eyes.open
, we pass it the Selenium instance, the name of the app we're testing (e.g., "the-internet"
), and the name of the test (e.g., "Login succeeded"
).
Now we're ready to add some visual checks to our test.
1// filename: Login.java2...3@Test public void succeeded() {4driver.get("http://the-internet.herokuapp.com/login");5eyes.checkWindow("Login");6driver.findElement(By.id("username")).sendKeys("tomsmith");7driver.findElement(By.id("password")).sendKeys("SuperSecretPassword!");8driver.findElement(By.id("login")).submit(); eyes.checkWindow("Logged In");9Assert.assertTrue("success message should be present after logging in",10driver.findElement(By.cssSelector(".flash.success")).isDisplayed());11eyes.close();12}13...
With eyes.checkWindow();
we are specifying when in the test's workflow we'd like Applitools Eyes to capture a screenshot (along with some description text). For this test we want to check the page before logging in, and then the screen just after logging in -- so we use eyes.checkWindow();
two times.
NOTE: These visual checks are effectively doing the same work as the pre-existing assertion (e.g., where we're asking Selenium if a success notification is displayed and asserting on the Boolean result) -- in addition to reviewing other visual aspects of the page. So once we verify that our test is working correctly we can remove this assertion and still be covered.
We end the test with eyes.close
. You may feel the urge to place this in teardown
, but in addition to closing the session with Eyes, it acts like an assertion. If Eyes finds a failure in the app (or if a baseline image approval is required), then eyes.close
will throw an exception; failing the test. So it's best suited to live in the test.
NOTE: An exceptions from eyes.close
will include a URL to the Applitools Eyes job in your test output. The job will include screenshots from each test step and enable you to play back the keystrokes and mouse movements from your Selenium tests.
When an exception gets thrown by eyes.close
, the Eyes session will close. But if an exception occurs before eyes.close
can fire, the session will remain open. To handle that, we'll need to add an additional command to our teardown
.
1// filename: Login.java2...3@After4public void teardown() {5eyes.abortIfNotClosed();6driver.quit();7}8}
eyes.abortIfNotClosed();
will make sure the Eyes session terminates properly regardless of what happens in the test.
Now when we run the test, it will execute locally while also performing visual checks in Applitools Eyes.
If we want to run our test with its newly added visual checks against other browsers and operating systems, it's simple enough to run your cross-browser tests on Sauce Labs.
NOTE: If you don't already have a Sauce Labs account, sign up for a free trial account here.
First we'll need to import the relevant classes.
1// filename: Login.java2... import org.openqa.selenium.Platform;3import org.openqa.selenium.remote.DesiredCapabilities;4import org.openqa.selenium.remote.RemoteWebDriver;5import java.net.URL;6...
We'll then need to modify the test setup to load a Sauce browser instance (via Selenium Remote) instead of a local Firefox one.
1// filename: Login.java2...3@Before public void setup() throws Exception {4DesiredCapabilities capabilities = DesiredCapabilities.internetExplorer();5capabilities.setCapability("platform", Platform.XP);6capabilities.setCapability("version", "8");7capabilities.setCapability("name", "Login succeeded");8String sauceUrl = String.format(9"http://%s:%s@ondemand.saucelabs.com:80/wd/hub",10"YOUR_SAUCE_USERNAME",11"YOUR_SAUCE_ACCESS_KEY");12WebDriver browser = new RemoteWebDriver(new URL(sauceUrl), capabilities);13eyes = new Eyes();14eyes.setApiKey(System.getenv("APPLITOOLS_API_KEY"));15driver = eyes.open(browser, "the-internet", "Login succeeded");16}17...
We tell Sauce what we want in our test instance through DesiredCapabilities
. The main things we want to specify are the browser, browser version, operating system (OS), and name of the test.
In order to connect to Sauce, we need to provide an account username and access key. The access key can be found on your account page. These values get concatenated into a URL that points to the Sauce Labs on-demand grid.
Once we have the DesiredCapabilities
and concatenated URL, we create a Selenium Remote instance with them and store it in a local browser
variable. Just like in our previous example, we feedbrowser
to eyes.open
and store the return object in the driver
variable.
Now when we run this test, it will execute against Internet Explorer 8 on Windows XP. You can see the test while it's running in your Sauce Labs account dashboard. And you can see the images captured on your Applitools account dashboard.
Both Applitools and Sauce Labs require you to specify a test name. Up until now, we've been hard-coding a value. Let's change it so it gets set automatically.
We can do this by leveraging a JUnit TestWatcher
and a public variable.
1// filename: Login.java2...3import org.junit.rules.TestRule;4import org.junit.rules.TestWatcher;5import org.junit.runner.Description;6...7public class Login {8private WebDriver driver;9private Eyes eyes;10public String testName;11@Rule12public TestRule watcher = new TestWatcher() {13protected void starting(Description description) {14testName = description.getDisplayName();15}16};17
Each time a test starts, the TestWatcher
starting
function will grab the display name of the test and store it in the testName
variable.
Let's clean up our setup to use this variable instead of a hard-coded value.
1// filename: Login.java2...3@Before4public void setup() throws Exception {5DesiredCapabilities capabilities = DesiredCapabilities.internetExplorer();6capabilities.setCapability("platform", Platform.XP);7capabilities.setCapability("version", "8");8capabilities.setCapability("name", testName);9String sauceUrl = String.format(10"http://%s:%s@ondemand.saucelabs.com:80/wd/hub",11System.getenv("SAUCE_USERNAME"),12System.getenv("SAUCE_ACCESS_KEY"));13WebDriver browser = new RemoteWebDriver(new URL(sauceUrl), capabilities);14eyes = new Eyes();15eyes.setApiKey(System.getenv("APPLITOOLS_API_KEY"));16driver = eyes.open(browser, "the-internet", testName);17}18...
Now when we run our test, the name will automatically appear. This will come in handy with additional tests.
When a job fails in Applitools Eyes, it automatically returns a URL for it in the test output. It would be nice if we could also get the Sauce Labs job URL in the output. So let's add it.
First, we'll need a public variable to store the session ID of the Selenium job.
1// filename: Login.java2...3public class Login {4private WebDriver driver;5private Eyes eyes;6public String testName;7public String sessionId;8...9
Next we'll add an additional function to TestWatcher
that will trigger when there's a failure. In it, we'll display the Sauce job URL in standard output.
1// filename: Login.java2...3@Rule4public TestRule watcher = new TestWatcher() {5protected void starting(Description description) {6testName = description.getDisplayName();7}89@Override10protected void failed(Throwable e, Description description) {11System.out.println(String.format("https://saucelabs.com/tests/%s", sessionId));12}13};14...
Lastly, we'll grab the session ID from the Sauce browser instance just after it's created.
1// filename: Login.java2...3WebDriver browser = new RemoteWebDriver(new URL(sauceUrl), capabilities);4sessionId = ((RemoteWebDriver) browser).getSessionId().toString();5...
Now when we run our test, if there's a Selenium failure, a URL to the Sauce job will be returned in the test output.
Connect to Applitools Eyes
Load an instance of Selenium in Sauce Labs
Run the test, performing visual checks at specified points
Close the Applitools session
Close the Sauce Labs session
Return a URL to a failed job in either Applitools Eyes or Sauce Labs
Happy Testing!
About Dave Haeffner: Dave is the author of Elemental Selenium (a free, once weekly Selenium tip newsletter that is read by hundreds of testing professionals) as well as a new book, The Selenium Guidebook. He is also the creator and maintainer of ChemistryKit (an open-source Selenium framework). He has helped numerous companies successfully implement automated acceptance testing; including The Motley Fool, ManTech International, Sittercity, and Animoto. He is a founder and co-organizer of the Selenium Hangout and has spoken at numerous conferences and meetups about acceptance testing.