Blog

Posted March 19, 2014

Getting Started with Selenium - Chapter 7: How To Scale Your Tests

This post is the seventh in a series of “Getting Started with Selenium Testing” posts from Dave Haeffner, a noted expert on Selenium and automated testing, and a frequent contributor to the Sauce blog and Selenium community. This series is for those who are brand new to test automation with Selenium and a new chapter will be posted every Tuesday (eight chapters in all).

How To Scale Your Tests

If you've ever needed to test features in an older browser like Internet Explorer 8 then odds are you ran a virtual machine (VM) on your computer with a "legit" version of Windows XP. Handy, but what happens when you need to check things on multiple versions of IE? Now you're looking at multiple VMs. And what about when you need to scale and cover other browser and Operating System (OS) combinations? Now you're looking at provisioning, running, and maintaining your own farm of machines and standing up something like Selenium Grid to coordinate tests across them. And all you wanted to do was run your tests on the browsers you cared about... Rather than take on the overhead of a test infrastructure you can easily outsource things to a third-party cloud provider - like Sauce Labs

A Selenium Remote, Selenium Grid, And Sauce Labs Primer

At the heart of Selenium at scale is the use of Selenium Grid and Selenium Remote. Selenium Grid lets you distribute test execution across several machines and you connect to it with Selenium Remote - specifying the browser type and version through the use of Selenium Remote's Capabilities. This is fundamentally how Sauce Labs works. Behind the curtain they are ultimately running Selenium Grid, and they receive and execute your tests through Selenium Remote - knowing which browser and operating system to use because of the Capabilities its users specify. Let's dig in with an example.

An Example

Part 1: Initial Setup

NOTE: You'll need an account to use Sauce Labs. Their free one offers enough to get you started. With Sauce Labs we need to provide specifics about what we want in our test environment, our credentials, and configure Selenium a little bit differently than we have been. Building on the last write-up, let's start by creating a new config file for cloud execution ('config_cloud.rb'). filename: config_cloud.rb

1
ENV['base_url']         ||= 'http://the-internet.herokuapp.com'
2
ENV['host']             = 'saucelabs'
3
ENV['operating_system'] ||= 'Windows XP'
4
ENV['browser']          ||= 'internet_explorer'
5
ENV['browser_version']  ||= '8'
6
ENV['SAUCE_USERNAME']   ||= 'your-sauce-username'
7
ENV['SAUCE_ACCESS_KEY'] ||= 'your-sauce-access-key'
8
Notice the use of a host environment variable. This is what we'll use in our spec_helper file to determine whether to run things locally or in the cloud -- and we'll use the other environment variables to populate the Capabilities. filename: spec/spec_helper.rb
9
require 'selenium-webdriver'
10

11
RSpec.configure do |config|
12

13
  config.before(:each) do
14
    case ENV['host']
15
    when 'saucelabs'
16
      caps = Selenium::WebDriver::Remote::Capabilities.send(ENV['browser'])
17
      caps.version = ENV['browser_version']
18
      caps.platform = ENV['operating_system']
19
      caps[:name] = example.metadata[:full_description]
20

21
      @driver = Selenium::WebDriver.for(
22
        :remote,
23
        url: "http://#{ENV['SAUCE_USERNAME']}:#{ENV['SAUCE_ACCESS_KEY']}@ondemand.saucelabs.com:80/wd/hub",
24
        desired_capabilities: caps)
25
    else
26
      @driver = Selenium::WebDriver.for :firefox
27
  end
28
end
29

30
  config.after(:each) do
31
    @driver.quit
32
  end
33

34
end
35

Notice that we've added a conditional to check on the host environment variable. If the host is set to 'saucelabs', then we configure the capabilities for Selenium Remote, passing in the requisite information that we will need for our Sauce Labs session. Otherwise, it will run our tests locally using Firefox. Now if we run our test suite (rspec -r ./config_cloud.rb) and navigate to our Sauce Labs Account page then we should see each of the tests running in their own job, with proper names, against Internet Explorer 8.

Part 2: Test Status

The only thing missing now is the pass/fail status of the job. In our local terminal window everything should be coming up green. But in the list of our Sauce jobs, the 'Results' panel for everything will just say 'Finished'. This will make our results less useful in the long run, so let's fix it. Thanks to Sauce Labs' sauce whisk gem, it's really simple to do. After we install it we will need to require it somewhere, and our 'config_cloud.rb' file seems like a logical place, since we will only need it when running our tests in Sauce. So let's add it to the top of the file. filename: config_cloud.rb

require 'sauce_whisk' ...

All that's left is to add an action to our after(:each) block in our 'spec_helper' file. Before we issue @driver.quit we will want to grab the job ID from our @driver object and set the job status based on the test result. Also, we'll want to make sure that it only executes when running tests against Sauce Labs -- so we'll want to wrap it in a conditional check against the host environment variable. filename: spec/spec_helper.rb

1
require 'selenium-webdriver'
2

3
RSpec.configure do |config|
4
  ...
5

6
  config.after(:each) do
7
    if ENV['host'] == 'saucelabs'
8
      if example.exception.nil?
9
        SauceWhisk::Jobs.pass_job @driver.session_id
10
      else
11
        SauceWhisk::Jobs.fail_job @driver.session_id
12
      end
13
    end
14

15
    @driver.quit
16
  end
17

18
end

Now when we run our tests (rspec -r ./config_cloud.rb) and navigate to our Sauce Labs Account page, we should see our tests running like before - but now when they finish there should be a proper test status (e.g., 'Pass' or 'Fail').

Speeding Up Your Tests

Now we can easily run our tests in Sauce Labs, but it's a real bummer that all of our tests are executing in series. As our suite grows, things will quickly start to add up and really hamper our ability to get feedback quickly. With parallelization we can quickly remedy this -- and there are a few ways to go about accomplishing it.

In code
Through a test runner
Through your Continuous Integration (CI) Server

Outro

For a deeper dive on how to scale your tests (e.g., running tests in parallel, testing applications behind a firewall, outputting Sauce Labs job information into your test results, etc.) then be sure to check out The Selenium Guidebook. Up next, in my final write-up, I'll help you tie everything together through the use of Continuous Integration.

Dave is the author of Elemental Selenium (a free, once weekly Selenium tip newsletter that is read by hundreds of testing professionals) as well as a new book, The Selenium Guidebook. He is also the creator and maintainer of ChemistryKit (an open-source Selenium framework). He has helped numerous companies successfully implement automated acceptance testing; including The Motley Fool, ManTech International, Sittercity, and Animoto. He is a founder and co-organizer of the Selenium Hangout and has spoken at numerous conferences and meetups about acceptance testing.

Bill McGee

Published:

Mar 19, 2014

Topics

Selenium

Share this post

EEA

CCPA

Cookie Settings