How To Run Your Automated Web Tests on Any Browser

Posted Sep 25th, 2015

This is the second post in a 3-part series on getting started off with automated web testing on the right foot. You can find the first post here and third post here.

The Problem

In the last post we stepped through how to write an automated web test that uses automated visual testing to perform assertions. This is a strong first step. But the test as it's written will only work on one browser (Firefox) -- leaving you with limited browser coverage.

A Solution

Thankfully Selenium is built to work on all major browser and operating system combinations. Traditionally, you would tap into this functionality by standing up a series of machines (for each operating system and browser you care about) and orchestrating your tests across these machines with Selenium Grid.

This is an unnecessary amount of complexity and hardware to manage when there are third-party services that do the heavy lifting for us. By using a third-party cloud service like Sauce Labs, we're able to gain access to whatever browser & operating system combinations we need with just a few lines of code. And there are no changes required to make this work with our Applitools Eyes implementation either (it will automatically capture new baseline images regardless of the browser).

Let's dig in with an example.

An Example

NOTE: We're going to take the code from the last post and modify it. If you want a more in-depth walkthrough of what it's doing, take a look at the last post.

Here is the test code from the last post.

[code language="ruby"]
# filename: login_spec.rb

require 'selenium-webdriver'
require 'eyes_selenium'

describe 'Login' do

before(:each) do |example|
@browser = Selenium::WebDriver.for :firefox
@eyes =
@eyes.api_key = ENV['APPLITOOLS_API_KEY']
@driver = 'the-internet', test_name:
example.full_description, driver: @browser)

after(:each) do
@browser.quit end it 'succeeded' do
@driver.get '' @eyes.check_window('Login Page')
@driver.find_element(id: 'username').send_keys 'tomsmith'
@driver.find_element(id: 'password').send_keys 'SuperSecretPassword!' @driver.find_element(css: 'button').click
@eyes.check_window('Logged In')

end [/code]

To quickly recap: this test code creates an instance of Firefox, connects to Applitools Eyes, visits the login page, completes the login form, performs visual checks to make sure the page renders correctly, ends the Applitools Eyes session, and closes the browser. To add in Sauce Labs support and gain access to a browser other than Firefox, we'll need to make some changes in the before(:each) portion of our test code. First, we'll need to create an object to store our configuration parameters (e.g., browser name, browser version, operating system, etc.). In Selenium this object is referred to as Capabilities. Next we'll need to connect to the Sauce Labs end-point and pass it the Capabilities object.

Let's step through how to gain access an older version of Internet Explorer (e.g., IE 8).

NOTE: You can see a full list of available browser & operating system combinations Sauce Labs offers here and see what each code configuration looks like with their handy platform configurator. Also, if you don't already have a Sauce Labs account you can sign up for a free trial account here.

[code language="ruby"] # filename: login_spec.rb

# ...

before(:each) do |example|
caps = Selenium::WebDriver::Remote::Capabilities.internet_explorer
caps.version = '8'
caps.platform = 'Windows XP'
caps[:name] = example.metadata[:full_description]

@browser = Selenium::WebDriver.for( :remote,
url: "http://#{ENV['SAUCE_USERNAME']}:#
{ENV['SAUCE_ACCESS_KEY']}", desired_capabilities: caps)

@eyes =
@eyes.api_key = ENV['APPLITOOLS_API_KEY']
@driver = 'the-internet', test_name: example.full_description, driver: @browser)

# ...

At the top of before(:each) we create a Capabilities object for Internet Explorer (e.g., Selenium::WebDriver::Remote::Capabilities.internet_explorer) and store it in a local variable (e.g., caps). We then use this variable to add our configuration parameters for the browser version (e.g., caps.version = '8'), operating system (e.g., caps.platform = 'Windows XP'), and the name of the test that's being run (e.g., caps[:name] = example.metadata[:full_description]).

We then rework the @browser instance variable. Instead of creating a local instance of Firefox, we're now connecting to the Sauce Labs end-point, passing in our capabilities object, and in return getting a Selenium instance which is controlling a browser that lives on a machine in Sauce Labs' grid.

Then the rest is business as usual with connecting to Applitools Eyes. We create an object, specify the API key, and connect to Applitools Eyes (passing the name of the application we're testing, the name of the test, and the Sauce Labs browser instance).

NOTE: In this example the Sauce Labs Username, Sauce Labs Access Key, and Applitools Eyes API Key are passed in through the use of environment variables. Alternatively, you could hard-code these values in the test if that's your preference.

Expected Behavior

When you save and run this (e.g., ruby login_spec.rb from the command line) here is what will happen:

  • Open an instance of Internet Explorer 8 in Sauce Labs
  • Connect the browser instance to Applitools Eyes
  • Run the test, performing visual checks at specified points
  • Close the Eyes session
  • Close the instance of Internet Explorer 8 in Sauce Labs
  • Display failures (if any) in the console output

A Small Bit of Cleanup

Typically a failure can occur for two different reasons. Either Selenium is unable to complete an action in the test, or, Selenium completes all of its actions and the assertion fails (which in our case are visual checks from Applitools Eyes).

If the latter happens, then the URL for the Applitools Eyes job will be outputted to the console. But if the prior happens we won't be able to easily track down the issue since there's no immediate connection to the job in Sauce Labs (which has loads of helpful information for debugging test failures). But with a simple modification to our after(:each) we can make this connection.

[code language="ruby"]
# filename: login_spec.rb

# ...

after(:each) do |example|
unless example.exception.nil?¬
raise "Watch a video of the test at

# ... [/code]

In the line where we declare after(:each) do we need to add a block variable so we can access details about the test that's being run (e.g., |example|).

Now we can we can raise a custom exception (e.g., raise followed by a string of text) which contains the URL to the job in Sauce Labs. We'll want to do this before closing the session in Sauce Labs and ensure that the Sauce Job gets closed properly even when an exception has occurred (hence the ensure). We make the URL ourself by specifying the base URL plus the ID of the session (which is available within the @driver object through .session_id). And since we only want to know about the URL when there is a test failure, we wrap the output message in a conditional to check that a test exception has occurred.

Now when there's a test failure, we will get both a URL to the job in Sauce Labs AND the URL to the job in Applitools Eyes.



1) Login succeeded
Got 0 failures and 2 other errors:

1.1) Failure/Error: @eyes.close
'Login succeeded' of 'the-internet'. see details at
# ./spec/login_spec.rb:39:in `block (2 levels) in '

1.2) Failure/Error: raise "Watch a video of the test at{@driver.session_id}"
Watch a video of the test at
# ./spec/login_spec.rb:25:in `block (2 levels) in '

Finished in 34.74 seconds (files took 0.2732 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/login_spec.rb:32 # Login succeeded

When you visit the Sauce Labs job you will see a video available from Sauce Labs (amidst a bevy of other helpful debugging information). sauce_screenshot

And when you visit the Applitools Eyes job you can quickly see the visual anomaly that caused a failure in our test. Applitools_Screenshot


Now you're armed with Selenium fundamentals, powerful assertions (thanks to visual checks), and access to all of the browser and operating system combinations you could ever want. Now we're ready to add the missing link -- automating when the tests run, which I'll cover in the next (and final) post. And if you're interested in learning about cross-browser visual testing (namely, what it is and how to do it in a simple and automated way), be sure to check out this write-up.

Written by

Dave Haeffner


Cross-browser testingVisual Testing