This is the third and final part of how to create Selenium scripts to run in parallel. Today we get into the meat of things with parallel execution, both locally and in OnDemand.

JUnit 4 doesn’t ship with a true parallel solution, but Harald Wellman wrote a blog post on running parameterized JUnit tests in parallel, which does just about everything we want. But instead of having a fixed thread pool side, I modified it to use a dynamically sized one. With this new class in your project, we only need to change the @RunWith line to use our new parallel, parameterized runner.

@RunWith(Parallelized.class)

Now the execution profile is as follows: For each item in the @Parameters Collection, launch a new thread to execute the current test method in parallel. Have two browser strings? Then two threads run in parallel. Have eight browser strings? Well, that is eight threads running in parallel. This means that however long it takes run a single browser through all your scripts, that’s how long it will take to run against any number of browsers.

Win!

Well, sure, but only if you have all those browsers on your machine or have an Se-Grid installation at your disposal. Most people do not have either luxury, so that is where Sauce OnDemand comes into the mix. Hosted in the cloud, you don’t need to have the browsers on your machine or even behind the firewall. With some minor modification to our example class, we can have a script that runs locally or in the OnDemand cloud.

package com.saucelabs.cc;

import java.util.Collection;
import java.util.List;
import java.util.LinkedList;
import java.util.Arrays;

import java.io.InputStream;
import java.util.Properties;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import static org.junit.Assert.*;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

import com.thoughtworks.selenium.DefaultSelenium;
import com.thoughtworks.selenium.Selenium;

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.FieldNamingPolicy;

import com.saucelabs.junit.Parallelized;
import com.saucelabs.ondemand.ConnectionParameters;

@RunWith(Parallelized.class)
public class TestParallelOnDemand {
    private Selenium selenium;
    private String browser;
    private String browserVersion;
    private String os;
    public static Properties browserProps = new Properties();
    private Properties parallelProps = new Properties();
    private String json;

    public TestParallelOnDemand(String os, String browser, String version) throws Exception {
        super();
        this.browser = browser;
        this.browserVersion = version;
        this.os = os;

        InputStream is = this.getClass().getResourceAsStream("/parallel.properties");
        parallelProps.load(is);

        if (parallelProps.getProperty("ondemand").equals("true")) {
          Gson gson = new GsonBuilder().setFieldNamingPolicy(FieldNamingPolicy.LOWER_CASE_WITH_DASHES).create();
          ConnectionParameters od = new ConnectionParameters();
          od.setBrowser(this.browser);
          od.browserVersion = this.browserVersion;
          od.jobName = this.getClass().getName();
          od.os = this.os;
          this.json = gson.toJson(od);
        }
    }

    @Parameters
    public static LinkedList browsersStrings() throws Exception {
      LinkedList browsers = new LinkedList();

      InputStream is = TestParallelOnDemand.class.getResourceAsStream("/browser.properties");
      browserProps.load(is);

      String[] rawBrowserStrings = browserProps.getProperty("browsers").split(",");
      for (String rawBrowserString : rawBrowserStrings) {
        if (rawBrowserString.indexOf(";") != -1) {
          String[] browserParts = rawBrowserString.split(";");
          browsers.add(new String[] { browserParts[0], browserParts[1], browserParts[2] });
        } else {
          browsers.add(new String[] { rawBrowserString, "", "" });
        }
      }
      return browsers;
    }

    @Before
    public void setUp() throws Exception  {
        if (parallelProps.getProperty("ondemand").equals("true")) {
          selenium = new DefaultSelenium("saucelabs.com", 4444, this.json, "http://saucelabs.com");
        } else {
          selenium = new DefaultSelenium("localhost", 4444, this.browser, "http://localhost:3000");
        }
        selenium.start();
        selenium.setTimeout("90000");
        selenium.windowMaximize();
    }

    @Test
    public void testSauce() throws Exception {
        this.selenium.open("/");
        assertEquals("Cross browser testing with Selenium - Sauce Labs", this.selenium.getTitle());
    }

    @After
    public void tearDown() throws Exception {
        selenium.stop();
    }
}

So what has changed?

The @Parameters are loaded differently again. When the script was only ever going to run locally, the browser string provided enough information. But since it’s also going to be run in OnDemand, there needs to be some extra information for the OS and browser version. Using the same pattern as part two, the browser string information has been moved from the test code and into an external properties file.

# order is important! - OS/browser/version
browsers=Windows 2003;*firefox;3.6.,Windows 2003;*googlechrome;3.,

OnDemand integrates with Selenium scripts by sending a JSON string with the configuration information to their server, which is why there is a new decision in the @Before method. An important design pattern when creating scripts that run locally or in some other environment is to add a switch in the script to determine where to go. You don’t want to have to modify code itself in order to run between different environments.

The JSON itself is an interesting challenge. Just as changing the source code didn’t make sense when parameterizing the individual browser strings, hard coding the JSON doesn’t make much sense either. To build the JSON in code, the Gson library was used. Here is the ConnectionParameters class that Gson is building the connection information from.

package com.saucelabs.ondemand;

import java.io.InputStream;
import java.util.Properties;

public class ConnectionParameters {
  // hyphenated strings like access-key and browser-version need to be camel-case
  public String username;
  public String accessKey;
  public String os;
  private String browser;
  public String browserVersion;
  public String jobName;

  // gson does not convert transient fields
  private transient String propertiesFile = "/ondemand.properties";
  private transient Properties ondemandProperties = new Properties();

  public ConnectionParameters() throws Exception {
    InputStream is = this.getClass().getResourceAsStream(propertiesFile);
    ondemandProperties.load(is);
    this.username = ondemandProperties.getProperty("username");
    this.accessKey = ondemandProperties.getProperty("access-key");
  }

  // ondemand browser strings cant start with the *
  public void setBrowser(String browser) {
    if (browser.startsWith("*")) {
      this.browser = browser.substring(1);
    }
  }

  public String getBrowser() {
    return this.browser;
  }
}

It should also be no surprise by now that the username and access key is loaded from an external file, as that can change and we wouldn’t want to recompile for something like that.

The only other thing of interest is the getter/setter for the browser string. Recall that the string can be used by either the local Se-RC server or OnDemand. Currently, OnDemand does not like having the * at the beginning, so we take care of that behind the scenes from the script.

With this bit of infrastructure in place, we can run our scripts against any OS/Browser/Version combination that OnDemand currently supports or you have handy on your own hardware, and it will automatically scale execution to run each method in parallel.

The example used here is a little extreme as the test itself is really only two lines, but in a real implementation scenario I would:

• Create a custom formatter for Selenium IDE (or Sauce IDE) so when you export the recorded script, it has everything your particular framework requires
• Create a class hierarchy for the scripts to move all the non @Test methods out of the class to make it a bit tidier

And so concludes the series on how to run Selenium scripts in parallel using JUnit 4. To see the full project, including the Maven pom, check out Parallel Test Examples in the Github repo.

In the first part of this series, we saw how to make use of the Parameterized runner that comes with JUnit 4 to execute our tests across multiple browsers. But that came with two penalties. The increasing test duration will be addressed in part three, but today we’ll address how to modify the browsers used without having to recompile our test.

As someone who sees a lot of different teams’ Selenium code, I’m appalled at how often a simple change, such as adjusting which browser(s) to run against, requires a recompile. To me, that is a nasty code smell. Things that do not materially affect the purpose of the script and are often changing should be externalized out of the code. In our case, we are going to use a Properties file inside the @Parameters method. Nothing else in the script has changed.

@Parameters
public static LinkedList browsersStrings() throws Exception {
  LinkedList browsers = new LinkedList();

  InputStream is = TestProperties.class.getResourceAsStream("/environment.properties");
  environmentProps.load(is);

  String[] rawBrowserStrings = environmentProps.getProperty("browsers").split(",");
  for (String rawBrowserString : rawBrowserStrings) {
    browsers.add(new String[] { rawBrowserString });
  }
  return browsers;
}

Nothing special here, just standard loading of a properties file that has a browsers key, which is a comma separated list of allowable Selenium browser strings. Now the same compiled code will run different browsers without actually change the Java code.

The result is much nicer, but still has to address the problem of a massive increase in execution. That will be the subject of part three of this series.

To see the full source code the snippet was pulled from, see TestProperties.java

Though not the first testing framework created, JUnit has garnered plenty of mainstream attention and created a wave of followers.

One of these is TestNG, which was created to address, among other things, the difficulty of passing parameters into scripts, as well as run scripts in parallel. For these reasons, the Selenium community has largely chosen TestNG as the framework of choice. But JUnit 4 can also address these topics using the @RunWith annotation and a custom runner.

This is part one in a series of posts illustrating how to create parameterized, parallel scripts that can be run either on your local infrastructure or on Sauce OnDemand.

Before we start into code, here’s a bit of context. I’ve been writing test frameworks and harnesses for a decade now and know the frustration of finding half answers online that have been stripped down for easy blog post write-ups. The intent of my post is to not inflict you with that experience. Instead, I’ll discuss parallel execution in the context of a full project – complete with Maven pom.xml and all headers in their proper places. There will be places in this post where they are omitted, so you are recommended to clone the Sauce Labs Parallel Test Examples github repo.

We begin our journey with @RunWith(Parameterized.class). This annotation will iterate over each test method with a set of parameters of your choosing. For Selenium scripts, the part that begs for parallelization is the browser, so that is what we will parameterize first.

Have a look at TestParameters.java, as there are a couple of interesting things beyond the standard @Before, @Test and @After.

package com.saucelabs.cc;

import java.util.Collection;
import java.util.List;
import java.util.Arrays;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import static org.junit.Assert.*;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

import com.thoughtworks.selenium.DefaultSelenium;
import com.thoughtworks.selenium.Selenium;

@RunWith(Parameterized.class)
public class TestParameters {
    private Selenium selenium;
    private String browser;

    public TestParameters(String browser){
        super();
        this.browser = browser;
    }

    @Parameters
    public static Collection browsersStrings(){
      return Arrays.asList(new Object[][] { {"*firefox"}, {"*googlechrome"} });
    }

    @Before
    public void setUp() throws Exception {
        selenium = new DefaultSelenium("localhost", 4444, this.browser, "http://localhost:3000");
        selenium.start();
        selenium.setTimeout("90000");
        selenium.windowMaximize();
    }

    @Test
    public void testSauce() throws Exception {
        this.selenium.open("/");
        assertEquals("Cross browser testing with Selenium - Sauce Labs", this.selenium.getTitle());
    }

    @After
    public void tearDown() throws Exception {
        selenium.stop();
    }
}

The first thing that sticks out is the @RunWith class annotation. While there are a number of different runners floating around, we are using the Parameterized one that comes with JUnit itself. With this in place, JUnit will look for a method annotated with @Parameters to use as arguments to the constructor. In this case, we currently only have one parameter, which we set as this.browser. Of course, this is then used when launching the browser in @Before.

And how does this address parallelization? Well, the @parameters method must return a Collection that is then iterated over, so you end up executing testSauce twice – the first time with “*firefox” and then with “*googlechrome.”

This is cool from a technical perspective, but has two pretty big flaws. First, the Java code needs to be edited to change browsers, and second, by executing serially, the time it takes to run the scripts increases 100% for every browser added. We’ll address both of those starting with external loading of our @Parameters in the next post of the series.