Unit Testing
Thousands if not millions of developers use JUnit for their Java testing. And what are they using it for? Overwhelmingly, the answer is unit testing, and nothing but unit testing.
But in fact, JUnit is much more than a unit testing framework. Did you know that within your regular JUnit test suite, you can include integration and acceptance tests too?
Let’s recall Mike Cohn’s testing pyramid:
Source: Martin Fowler, TestPyramid, based on Mike Cohn’s book Succeeding with Agile
At the bottom are regular unit tests. There are a lot of these because we aim to cover as much of our code as possible.
But unit testing only tests parts of our code in isolation - it can’t see how things play together. For that, we have those two layers above:
-
Service / integration testing - making sure components of our software interact correctly with each other. For example, that your application can successfully connect to the database.
-
UI / acceptance testing - making sure a feature in the software actually works from the user’s perspective. For example, your user can enter a known username and successfully log in.
Why are these layers important? Imagine you need to test a tax calculator on an eCommerce site. This is a crucial feature because if it fails, people can’t buy and the company loses money.
So you write lots of unit tests to make sure every possible input to the tax calculator produces the right output. With these tests in place, you feel confident that no matter which country, region, currency and purchase amount are entered, the tax amount will be correct.
After some tweaks, all the unit tests pass. Build is green! Ready to go!
But then you get a call from management (for dramatic effect, let’s say it’s 2am) saying the shopping cart interface is broken on the site.
You start a browser and go through the purchase process, and then at the last stage, when users need to view the tax amount,the amount isn’t shown at all. A browser issue caused another DIV to move out of place, and the tax amount is hidden by an ugly rectangle.
(We realize this example is a bit over the top. But things like this do actually happen.)
So even with all those unit tests, the user experience is broken. 100% code coverage is important - but just on it’s own, it can’t make sure that critical software functions actually work. That’s why integration and unit tests are needed.
Most of us know this in the back of our minds. Only a few of us actually venture out to write and execute these tests. But in fact,it’s much easier than you think.
Unit tests and Selenium tests - happy neighbors?
Here’s a JUnit test folder of a small Jenkins plugin we have written especially for this post. On first glance it looks like just any other test folder.
But take a closer look:
In this JUnit folder, living peacefully side-by-side, are tests from all three tiers of the testing pyramid. The fourth test listed is a regular unit test, but the top three tests in the list are actually based on Selenium 2.
The unit test file has regular stuff like this:
quickstart...
@Test
public void successResponse() {
SauceStatusHelper statusHelper = new SauceStatusHelper() {
@Override
public String retrieveResults(URL restEndpoint) {
return "{\"service_operational\" : true, \"status_message\" : \"Basic service status checks passed.\"}";
}
};
SauceStatusPageDecorator pageDecorator = new SauceStatusPageDecorator(statusHelper);
assertEquals("Sauce status mismatch", "Basic service status checks passed.", pageDecorator.getsauceStatus());
}
...
For example, here’s a bit of code that uses a real browser (e.g. Chrome) to click the link “Manage Jenkins”, then click “Manage Plugins”, then verify that the plugin is installed:
public void pluginIsListed() {
//Find and click on the 'Manage Jenkins' link
webDriver.findElement(By.linkText("Manage Jenkins")).click();
//It takes a few seconds for the page to load, so instead of running Thread.sleep(), we use the WebDriverWait construct
WebDriverWait wait = new WebDriverWait(webDriver, 30);
wait.until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("h1")));
//assert that we are on the Manage Jenkins page
assertEquals("Header not found", "Manage Jenkins", webDriver.findElement(By.cssSelector("h1")).getText());
//Find and click on the 'Manage Plugins' link
webDriver.findElement(By.linkText("Manage Plugins")).click();
wait.until(ExpectedConditions.visibilityOfElementLocated(By.linkText("Installed")));
//Find and click on the 'Installed' link
webDriver.findElement(By.linkText("Installed")).click();
assertEquals("Plugin link not found", "Jenkins Sauce Health Check plugin", webDriver.findElement(By.linkText("Jenkins Sauce Health Check plugin")).getText());
}
Cool huh? Let’s dive in a bit deeper and see how simple it is to cover all three tiers of the testing pyramid in one JUnit test suite.
Code example with all three testing tiers: Sauce Labs Health Check
Don’t worry - we’re not pushing our product (just yet ;) - we just based our example on a Jenkins plugin that interacts with our Sauce cloud service. But it could just as well be any Java code in any application.
(Oh, in case we haven’t met, we’re the team at Sauce Labs. We help thousands of organizations do Selenium testing on the cloud.)
The simple plugin we have written is called Sauce Health Check - see it on Github. Here’s what it does:
- Adds a line to the bottom of the Jenkins interface, like this:
- Performs a REST call to the Sauce Labs cloud service to see its status - up or down.
- Depending on the status, the message will be green and will say “Basic service status checks passed”, like in the screenshot; otherwise red and will say “Sauce Labs down.”
- The “Check now” link is an AJAX call. When clicked, it shows a little “working” icon, and re-checks the status of the service. We purposely made this an AJAX call because that’s something that can only be fully tested with browser/UI automation.
In case you’re wondering - why would you need to see the Sauce Labs status at the bottom of your Jenkins dashboard? No reason. It’s just the simplest piece of software we could think of that could help us demonstrate all three tiers of Java testing.
The code is simple and we won’t run you through it - suffice it to say that there’s a descriptor that tells Jenkins this is a plugin, a SauceStatusHelper which calls the Sauce Labs REST API to see if the service is up or down, and a PageDecorator which defines how the result is displayed on the Jenkins interface.
It’s showtime! See the tests
As promised, here are all three tiers of testing within one JUnit test suite, testing the Sauce Health Check Plugin.
1. Unit Tests
The unit tests reside in SauceStatusPageDecoratorTest.java. Here we have 3 functions that test which message is returned by PageDecorator for three possible statuses of the Sauce Cloud.
The REST API is mocked out to create a self-contained test of the code. If these unit tests pass, it means that the basic logic of the plugin is coded correctly.
Testing when Sauce is down:
@Test
public void errorResponse() {
SauceStatusHelper statusHelper = new SauceStatusHelper() {
@Override
public String retrieveResults(URL restEndpoint) {
return "{\"service_operational\" : false, \"status_message\" : \"Sauce Labs down\"}";
}
};
SauceStatusPageDecorator pageDecorator = new SauceStatusPageDecorator(statusHelper);
assertEquals("Sauce status mismatch", "Sauce Labs down", pageDecorator.getsauceStatus());
}
This preceding test is important because, in reality, the service is very rarely down. So it’s difficult to test this possibility in a UI automation test, but at least we make sure it’s supported in the code.
Now, testing when Sauce is up:
@Test
public void successResponse() {
SauceStatusHelper statusHelper = new SauceStatusHelper() {
@Override
public String retrieveResults(URL restEndpoint) {
return "{\"service_operational\" : true, \"status_message\" : \"Basic service status checks passed.\"}";
}
};
SauceStatusPageDecorator pageDecorator = new SauceStatusPageDecorator(statusHelper);
assertEquals("Sauce status mismatch", "Basic service status checks passed.", pageDecorator.getsauceStatus());
}
And when an empty string is returned by the REST call (status unknown):
@Test
public void invalidResponse() {
SauceStatusHelper statusHelper = new SauceStatusHelper() {
@Override
public String retrieveResults(URL restEndpoint) {
return "";
}
};
SauceStatusPageDecorator pageDecorator = new SauceStatusPageDecorator(statusHelper);
assertEquals("Sauce status mismatch", "Unknown", pageDecorator.getsauceStatus());
}
}
2. Integration Test
Next up we want to check that the “integration” of our plugin has succeeded - that the plugin is properly installed on Jenkins. We do this using a bit of Selenium 2 (A.K.A Selenium Webdriver) code. The integration tests are in IntegrationIT.java.
First off we define the WebDriver instance variable (WebDriver is the piece of Selenium that automates the browser), and define a JUnit rule that launches a local Jenkins instance with our plugin installed.
private WebDriver webDriver;
@Rule public JenkinsRule jenkinsRule = new JenkinsRule(); }
}
Now we start an actual Firefox browser, by creating a new FirefoxDriver instance (it is assumed that Firefox is installed on the local machine) and point it to the URL of the local Jenkins instance.
@Before
public void setUp() throws Exception {
webDriver = new FirefoxDriver();
URL url = jenkinsRule.getURL();
webDriver.get(url.toString());
}
That’s all it takes to fire up a real browser from within the code. Now we start the actual test - on the Jenkins dashboard, find and click the Manage Jenkins link:
@Test
public void pluginIsListed() {
//Find and click on the 'Manage Jenkins' link
webDriver.findElement(By.linkText("Manage Jenkins")).click();
Wait a few seconds for the page to load (remember we are on a real Firefox browser), using the
WebDriverWait
construct, but wait only until the H1 title of the page has rendered:
WebDriverWait wait = new WebDriverWait(webDriver, 30);
wait.until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("h1")));
Check the H1 title and ensure we are on the Manage Jenkins page:
assertEquals("Header not found", "Manage Jenkins", webDriver.findElement(By.cssSelector("h1")).getText());
Find and click on the Manage Plugins link:
webDriver.findElement(By.linkText("Manage Plugins")).click();
Find and click on the Installed link, as soon as it has rendered on the page:
WebDriverWait wait = new WebDriverWait(webDriver, 30);
wait.until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("h1")));
And finally - look for the link Jenkins Sauce Health Check plugin.
assertEquals("Plugin link not found", "Jenkins Sauce Health Check plugin", webDriver.findElement(By.linkText("Jenkins Sauce Health Check plugin")).getText()); }
If this link exists, our test passes and we conclude that the plugin is properly installed. By running this simple automated test we perform a sanity check of our basic integration.
3. Acceptance Test
We’re not stopping here - we’ll automatically test the plugin UI as well. The acceptance test is in AcceptanceIT.java.
Again, we instantiate the WebDriver instance (this automates the browser), and define a JUnit rule that launches a local Jenkins instance.
private WebDriver webDriver;
@Rule
public JenkinsRule jenkinsRule = new JenkinsRule();
@Before
public void setUp() throws Exception {
webDriver = new FirefoxDriver();
URL url = jenkinsRule.getURL();
webDriver.get(url.toString());
}
Testing plugin message while navigating through Jenkins pages
Now we’ll test that the plugin’s message is displayed as we navigate through the Jenkins user interface. We expect to always see a message like this at the bottom of the screen:
We find and click the Manage Jenkins link using the WebDriver findElement function, wait for the page to load, and make sure that the Sauce Status message appears at the bottom of the screen. This is done by selecting an element with ID equal tosauce_status:
@Test
public void navigation() {
...
webDriver.findElement(By.linkText("Manage Jenkins")).click();
WebDriverWait wait = new WebDriverWait(webDriver, 30);
wait.until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("h1")));
assertNotNull("Status not found", webDriver.findElement(By.id("sauce_status")));
Next we click through to another page, New Job, and and make sure the Sauce Status message appears there too:
webDriver.findElement(By.linkText("New Job")).click();
assertNotNull("Status not found", webDriver.findElement(By.id("sauce_status")));
Now we use an XPath expression to click the Jenkins link in the navigation bar, and verify the Sauce Status message is still displayed:
webDriver.findElement(By.xpath("//ul[@id=\"breadcrumbs\"]//a[1]")).click();
//wait until the 'Welcome to Jenkins' div is visible wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//td[@id='main-panel']/div[2][contains(text(), 'Welcome to Jenkins!')]")));
assertNotNull("Status not found", webDriver.findElement(By.id("sauce_status")));
Now we click through to a fourth page, UI Samples, this time using a CSS selector, and again check for the message at the bottom:
webDriver.findElement(By.cssSelector("div.task:nth-child(5) > a:nth-child(2)")).click();
assertNotNull("Status not found", webDriver.findElement(By.id("sauce_status")));
}
That’s it for the navigation test.
Testing the plugin message is shown with the correct color
Now another test that verifies the color of the Sauce Status message:
@Test
public void colourOfStatus() {
WebElement sauceStatus = webDriver.findElement(By.className("sauce_up"));
assertNotNull("Status not found", sauceStatus);
String colour = sauceStatus.getCssValue("color");
assertEquals("Colour not green", "rgba(0, 128, 0, 1)", colour);
}
Testing the AJAX call
And lastly, we check the AJAX call. Remember - this was the Check Now link that refreshes the Sauce Labs status at the bottom of the screen:
First we click the Check Now link. This can only be done with a browser automation framework like Selenium, because we are operating inside a real browser.
@Test
public void ajaxAction() {
WebElement sauceStatusProgressImage = webDriver.findElement(By.id("sauce_status_progress"));
WebElement sauceStatusMessage = webDriver.findElement(By.id("sauce_status_msg"));
assertFalse("Element is visible", sauceStatusProgressImage.isDisplayed());
WebElement checkNowLink = webDriver.findElement(By.id("sauce_check_status_now"));
assertNotNull("Status not found", checkNowLink);
//click the link
checkNowLink.click();
After the click, we verify that the loading image is displayed and the status has changed to Checking. We wait for a few seconds to give it a chance to load.
assertTrue("Element is not visible", sauceStatusProgressImage.isDisplayed());
assertEquals("Status text not 'Checking'", "Checking...", sauceStatusMessage.getText());
WebDriverWait wait = new WebDriverWait(webDriver, 30);
wait.until(ExpectedConditions.invisibilityOfElementLocated(By.id("sauce_status_progress")));
We assert that the correct text is displayed after the AJAX click:
assertEquals("Status text not expected", "Basic service status checks passed.", sauceStatusMessage.getText());
}
And lastly, cleaning up after ourselves - closing down the Firefox browser we used to perform the tests:
@After public void tearDown() throws Exception {
webDriver.quit();
}
Now let’s do the same thing on multiple browsers with Sauce
In the Selenium tests shown above, we only tested the plugin on one browser, Firefox. In a real application, you’d want to test the user interface on multiple browsers and operating systems.
Of course, this adds the complexity of having to install all those platforms on the computer that is running your tests - you might want to test on dozens or hundreds of different platforms. This complexity is part of the reason why many organizations are hesitant to do acceptance testing on a significant scale.
An easy way to run your Selenium tests on a large number of platforms is the Sauce Labs cloud service. We support JUnit, and maintain over 700 browser/OS combinations, including Android and iOS. So you can just specify which platform you want to run your tests on, within your JUnit test files, and we’ll handle it for you.
Even if your application is running locally behind a firewall, Sauce provides a secure tunnel called Sauce Connect, which allows our cloud servers to securely access your application behind the firewall.
Keep in mind - Sauce is a commercial service. We have a 14 day Free Trial which you can use to run automated or manual tests.
However, we’re completely free for open source projects - in fact, we are used by Mozilla, YUI, Travis, JQuery, and many other popular open source projects.
Running the same integration and acceptance tests on the Sauce cloud
In the JUnit test folder, there is another file called SauceIT.java. These are exactly the same Acceptance Tests as we described above, but they run on multiple browsers in the Sauce Cloud. Here’s how it works -
The following variables define the Sauce Connect tunnel (assuming you are testing behind a firewall), and the operating system, version and browser you want to test on.
private static final SauceConnectFourManager sauceConnectManager = new SauceConnectFourManager();
private final String os;
private final String version;
private final String browser;
Like in the previous acceptance test, we instantiate Webdriver and a local Jenkins instance:
private WebDriver webDriver;
@Rule public JenkinsRule jenkinsRule = new JenkinsRule();
We select the browser/OS combination to be used for the current test:
public SauceIT(String os, String version, String browser) {
super();
this.os = os;
this.version = version;
this.browser = browser;
}
@Parameterized.Parameters
public static LinkedList browserStrings() throws Exception {
LinkedList browsers = new LinkedList();
browsers.add(new String[]{"Windows 2003", null, "chrome"});
browsers.add(new String[]{"Windows 2003", "17", "firefox"});
browsers.add(new String[]{"linux", "17", "firefox"});
return browsers;
}
Here’s the code to start Sauce Connect - allows Sauce Labs machines on the cloud to seamlessly connect to a website or mobile site running on your local network.
public static void startSauceConnect() throws IOException { sauceConnectManager.openConnection("SAUCE_USER", "SAUCE_ACCESS_KEY", 4445, null, null, null, null); }
And lastly, we create a RemoteWebDriver instance that is configured to launch a specific OS / version / browser combination:
public void setUp() throws Exception {
DesiredCapabilities capabilities = DesiredCapabilities.firefox();
capabilities.setCapability(CapabilityType.BROWSER_NAME, browser);
capabilities.setCapability(CapabilityType.VERSION, version);
capabilities.setCapability(CapabilityType.PLATFORM, Platform.valueOf(os));
this.webDriver = new RemoteWebDriver(
new URL("http://SAUCE_USER:SAUCE_ACCESS_KEY@ondemand.saucelabs.com:80/wd/hub"),
capabilities);
URL url = jenkinsRule.getURL();
webDriver.get(url.toString());
}
That’s it! The rest of the code is the same as in the regular acceptance test above. Only now you can test against any number of OS/version/browser combinations in the Sauce cloud.
Summary
It’s been a long post. But we hope we’ve made our point that JUnit users should be doing more than just unit testing. And that really, it’s quite easy to add integration and acceptance tests with UI automation to your JUnit testing repertoire.
Our code example of a simple Jenkins plugin demonstrated how within one JUnit test suite, you can run regular unit tests, integration tests, and acceptance tests with full UI automation. We showed some pretty advanced browser automation flows, achieved in only a few lines of code.
Finally, we took the liberty of showing how the Sauce Labs cloud service fits in - helping you easily run the same Selenium tests on a large number of browser/OS platforms.
Here’s to covering the entire testing pyramid with JUnit! (And hopefully Sauce too :)