Unit Testing
Thousands if not millions of developers use JUnit for their Java testing. And what are they using it for? Even after 20 years of JUnit’s existence it is still mostly being used for unit tests. If you are lucky you may see an integration test or two.
JUnit is, in fact, much more than a unit testing framework. Did you know that within your regular JUnit test suite, you can even include acceptance tests?
Let’s recall Mike Cohn’s testing pyramid:
Source: Martin Fowler, TestPyramid, based on Mike Cohn’s book Succeeding with Agile
At the bottom are regular unit tests. There are a lot of these because we aim to cover as much of our code as possible.
But unit testing only tests parts of our code in isolation - it can’t see how things play together. For that, we have those two layers above:
Service / integration testing - making sure the components of our application are glued together properly. For example, ensuring that our logging in place and that our controller can deserialize a request into a saved entry in our database.- UI / acceptance testing - making sure a feature in the software actually works from the user’s perspective. For example, your user can click to add an item to their shopping cart and checkout.
The pyramid guides as to have a variety of tests of various amounts based on their level. To understand why let’s look at an example. Imagine you need to test a tax calculator on an eCommerce site. This is a crucial feature because if it fails, people can’t buy and the company loses money.
So you write lots of unit tests to make sure every possible input to the tax calculator produces the right output. With these tests in place, you feel confident that no matter which country, region, currency and purchase amount are entered, the tax amount will be correct.
After some tweaks, all the unit tests pass.
But then you get a call from management (for dramatic effect, let’s say it’s
You start a browser and go through the purchase process, and then at the last stage, when users need to view the tax amount, the amount isn’t shown at all. A browser issue caused another DIV to move out of place, and the tax amount is hidden by an ugly rectangle.
So even with all those unit tests, the user experience is broken. This may sound over the top but incidents like this happen frequently. Ensuring your fine-grained business logic is important- but just on
Most of us know this in the back of our minds. Only a few of us actually venture out to write and execute these tests. But in fact, it’s much easier than you think.
Unit tests and Selenium tests - happy neighbors?
Here’s a JUnit test folder of a small Jenkins plugin we have written especially for this post. On first glance, it looks like just any other test folder.
But take a closer look:
In this JUnit folder, living peacefully side-by-side, are tests from all three tiers of the testing pyramid. The fourth test listed is a regular unit test, but the top three tests in the list are actually based on Selenium 3.
The unit test file has regular stuff like this:
quickstart...
@Test
public void successResponse() {
SauceStatusHelper statusHelper = new SauceStatusHelper() {
@Override
public String retrieveResults(URL restEndpoint) {
return "{\"service_operational\" : true, \"status_message\" : \"Basic service status checks passed.\"}";
}
};
SauceStatusPageDecorator pageDecorator = new SauceStatusPageDecorator(statusHelper);
assertEquals("Sauce status mismatch", "Basic service status checks passed.", pageDecorator.getsauceStatus());
}
...
For example, here’s a bit of code that uses a real browser (e.g. Chrome) to click the link “Manage Jenkins”, then click “Manage Plugins”, then verify that the plugin is installed:
public void pluginIsListed() {
//Find and click on the 'Manage Jenkins' link
webDriver.findElement(By.linkText("Manage Jenkins")).click();
//It takes a few seconds for the page to load, so instead of running Thread.sleep(), we use the WebDriverWait construct
WebDriverWait wait = new WebDriverWait(webDriver, 30);
wait.until(driver -> driver.findElement(By.cssSelector("h1")).isDisplayed());
//assert that we are on the Manage Jenkins page
assertEquals("Header not found", "Manage Jenkins", webDriver.findElement(By.cssSelector("h1")).getText());
//Find and click on the 'Manage Plugins' link
webDriver.findElement(By.linkText("Manage Plugins")).click();
wait.until(driver -> driver.findElement(By.linkText("Installed")).isDisplayed());
//Find and click on the 'Installed' link
webDriver.findElement(By.linkText("Installed")).click();
assertEquals("Plugin link not found", "Sauce Health Check plugin", webDriver.findElement(By.linkText("Sauce Health Check plugin")).getText());
}
Does that not seem powerful? Let’s dive in a bit deeper and see how simple it is to cover all three tiers of the testing pyramid in one JUnit test suite.
Code example with all three testing tiers: Sauce Labs Health Check
We want to show you how easy it can be to integration and acceptance test with JUnit. To do this, we created a simple plugin we have written is called Sauce Health Check - see it on Github. The goal is to connect to our Sauce Labs multi-browser testing service
- Adds a line to the bottom of the Jenkins interface, like this:
- Performs a REST call to the Sauce Labs cloud service to see its status - up or down.
- Depending on the status, the message will be green and will say “Basic service status checks passed”, like in the screenshot; otherwise red and will say “Sauce Labs down.”
- The “Check now” link is an AJAX call. When clicked, it shows a little “working” icon and re-checks the status of the service. We purposely made this an AJAX call because that’s something that can only be fully tested with browser/UI automation.
In case you’re wondering - why would you need to see the Sauce Labs status at the bottom of your Jenkins dashboard? No reason. It’s just the simplest piece of software we could think of that could help us demonstrate all three tiers of Java testing.
The code is simple and we won’t run you through it - suffice it to say that there’s a descriptor that tells Jenkins this is a plugin, a SauceStatusHelper which calls the Sauce Labs REST API to see if the service is up or down, and a PageDecorator which defines how the result is displayed on the Jenkins interface.
It’s showtime! See the tests
As promised, here are all three tiers of testing within one JUnit test suite, testing the Sauce Health Check Plugin.
1. Unit Tests
The unit tests reside in SauceStatusPageDecoratorTest.java. Here we have 3 functions that test what messages the PageDecorator may return, depending on the Sauce Cloud’s status.
The REST API is stubbed out to create a self-contained test of the code. If these unit tests pass, it means that the basic logic of the plugin is coded correctly.
Testing when Sauce is down:
@Test
public void errorResponse() {
SauceStatusHelper statusHelper = new SauceStatusHelper() {
@Override
public String retrieveResults(URL restEndpoint) {
return "{\"service_operational\" : false, \"status_message\" : \"Sauce Labs down\"}";
}
};
SauceStatusPageDecorator pageDecorator = new SauceStatusPageDecorator(statusHelper);
assertEquals("Sauce status mismatch", "Sauce Labs down", pageDecorator.getsauceStatus());
}
We test this because the service is rarely down, so it is hard for us to test this via UI automation. But we ensure to diligently test against our edge cases, not only our happy paths. Now, testing when Sauce is up:
@Test
public void successResponse() {
SauceStatusHelper statusHelper = new SauceStatusHelper() {
@Override
public String retrieveResults(URL restEndpoint) {
return "{\"service_operational\" : true, \"status_message\" : \"Basic service status checks passed.\"}";
}
};
SauceStatusPageDecorator pageDecorator = new SauceStatusPageDecorator(statusHelper);
assertEquals("Sauce status mismatch", "Basic service status checks passed.", pageDecorator.getsauceStatus());
}
And when an empty string is returned by the REST call (status unknown):
@Test
public void invalidResponse() {
SauceStatusHelper statusHelper = new SauceStatusHelper() {
@Override
public String retrieveResults(URL restEndpoint) {
return "";
}
};
SauceStatusPageDecorator pageDecorator = new SauceStatusPageDecorator(statusHelper);
assertEquals("Sauce status mismatch", "Unknown", pageDecorator.getsauceStatus());
}
}
2. Integration Test
Now we want to check that our code integrates with Jenkins correctly by ensuring that the plugin is properly installed on Jenkins. We do this using a bit of Selenium 3 (A.K.A Selenium Webdriver) code. The integration tests are in IntegrationIT.java.
First off we define the WebDriver instance variable (WebDriver is the piece of Selenium that automates the browser
private WebDriver webDriver; @Rule public JenkinsRule jenkinsRule = new JenkinsRule(); } }
Now we start an actual Chrome browser, by creating a new ChromeDriver instance and point it to the URL of the local Jenkins instance.
@Before
public void setUp() throws Exception
{this.webDriver = Drivers.localResourceChromeDriver();
URL url = jenkinsRule.getURL();
webDriver.get(url.toString());
}
Our tests are well-factored, so we use a helper method to new up the source-controlled driver without duplicating the code all over our tests:
public class Drivers {
static ChromeDriver localResourceChromeDriver() {
URL resource = Drivers.class.getClassLoader().getResource("chromedriver.exe");
System.setProperty("webdriver.chrome.driver", resource.getPath());
return new ChromeDriver();
}
}
That’s all it takes to fire up a real browser from within the code. Now we start the actual test - on the Jenkins dashboard, find and click the Manage Jenkins link:
@Test
public void pluginIsListed() {
//Find and click on the 'Manage Jenkins' link
webDriver.findElement(By.linkText("Manage Jenkins")).click();
Wait a few seconds for the page to load (remember we are on a real Chrome browser), using the
WebDriverWait
construct, but wait only until the H1 title of the page has rendered:
WebDriverWait wait = new WebDriverWait(webDriver, 30);
wait.until(driver -> driver.findElement(By.cssSelector("h1")).isDisplayed());
Check the H1 title and ensure we are on the Manage Jenkins page:
assertEquals("Header not found", "Manage Jenkins", webDriver.findElement(By.cssSelector("h1")).getText());
Find and click on the Manage Plugins link:
webDriver.findElement(By.linkText("Manage Plugins")).click();
Find and click on the Installed link, as soon as it has rendered on the page:
WebDriverWait wait = new WebDriverWait(webDriver, 30);
wait.until(driver -> driver.findElement(By.cssSelector("h1")).isDisplayed());
And finally - look for the link Jenkins Sauce Health Check plugin.
assertEquals("Plugin link not found", "Sauce Health Check plugin", webDriver.findElement(By.linkText("Sauce Health Check plugin")).getText()); }
If this link exists, our test passes and we conclude that the plugin is properly installed. By running this simple automated test we perform a sanity check of our basic integration.
3. Acceptance Test
We’re not stopping here - we’ll automatically test the plugin UI as well. The acceptance test is in AcceptanceIT.java.
Again, we instantiate the WebDriver instance (this automates the browser), and define a JUnit rule that launches a local Jenkins instance.
private WebDriver webDriver;
@Rule
public JenkinsRule jenkinsRule = new JenkinsRule();
@Before
public void setUp() throws Exception {
webDriver = new FirefoxDriver();
URL url = jenkinsRule.getURL();
webDriver.get(url.toString());
}
Testing plugin message while navigating through Jenkins pages
Now we’ll test that the plugin’s message is displayed as we navigate through the Jenkins user interface. We expect to see a message like this at the bottom of the screen, independent of the page we are on:
We find and click the Manage Jenkins link using the WebDriver findElement function, wait for the page to load, and make sure that the Sauce Status message appears at the bottom of the screen. This is done by selecting an element with ID equal tosauce_status:
@Test
public void navigation() {
...
webDriver.findElement(By.linkText("Manage Jenkins")).click();
WebDriverWait wait = new WebDriverWait(webDriver, 30);
wait.until(driver -> driver.findElement(By.cssSelector("h1")).isDisplayed());
assertNotNull("Status not found", webDriver.findElement(By.id("sauce_status")));
webDriver.findElement(By.linkText("About Jenkins")).click();
assertNotNull("Status not found", webDriver.findElement(By.id("sauce_status")));
Now we use an XPath expression to click the Jenkins link in the navigation bar, and verify the Sauce Status message is still displayed:
webDriver.findElement(By.xpath("//ul[@id=\"breadcrumbs\"]//a[1]")).click();
//wait until the 'Welcome to Jenkins' div is visible wait.until(driver ->
driver.findElement(By.xpath("//div[@id='main-panel']/div[2]/h1[contains(text(), 'Welcome to Jenkins!')]")).isDisplayed());
assertNotNull("Status not found", webDriver.findElement(By.id("sauce_status")));
Now we click through to a fourth page, UI Samples, this time using a CSS selector, and again check for the message at the bottom:
webDriver.findElement(By.cssSelector("div.task:nth-child(5) > a:nth-child(2)")).click();
assertNotNull("Status not found", webDriver.findElement(By.id("sauce_status")));
}
That’s it for the navigation test.
Testing the plugin message is shown with the correct color
Now another test that verifies the color of the Sauce Status message:
@Test
public void colourOfStatus() {
WebElement sauceStatus = webDriver.findElement(By.className("sauce_up"));
assertNotNull("Status not found", sauceStatus);
String colour = sauceStatus.getCssValue("color");
assertEquals("Colour not green", "rgba(0, 128, 0, 1)", colour);
}
Testing the AJAX call
And lastly, we check the AJAX call. Remember - this was the Check Now link that refreshes the Sauce Labs status at the bottom of the screen:
@Test
public void ajaxAction() {
WebElement sauceStatusProgressImage = webDriver.findElement(By.id("sauce_status_progress"));
WebElement sauceStatusMessage = webDriver.findElement(By.id("sauce_status_msg"));
assertFalse("Element is visible", sauceStatusProgressImage.isDisplayed());
WebElement checkNowLink = webDriver.findElement(By.id("sauce_check_status_now"));
assertNotNull("Status not found", checkNowLink);
//click the link
checkNowLink.click();
After the click, we verify that the loading image is displayed and the status has changed to Checking. We wait for a few seconds to give it a chance to load.
assertTrue("Element is not visible", sauceStatusProgressImage.isDisplayed());
assertEquals("Status text not 'Checking'", "Checking...", sauceStatusMessage.getText());
WebDriverWait wait = new WebDriverWait(webDriver, 30);
wait.until(driver -> !driver.findElement(By.id("sauce_status_progress")).isDisplayed());
We assert that the correct text is displayed after the AJAX click:
assertEquals("Status text not expected", "Basic service status checks passed.", sauceStatusMessage.getText());
}
And lastly, cleaning up after ourselves - closing down the Firefox browser we used to perform the tests:
@After public void tearDown() throws Exception {
webDriver.quit();
}
Now let’s do the same thing on multiple browsers with Sauce
In the Selenium tests shown above, we only tested the plugin on one browser, Firefox. In a real application, you’d want to test the user interface on multiple browsers and operating systems.
Of course, this adds the complexity of having to install all those platforms on the computer that is running your tests - you might want to test on dozens or hundreds of different platforms. This complexity is part of the reason why many organizations are hesitant to do acceptance testing on a significant scale.
An easy way to run your Selenium tests on a large number of platforms is the Sauce Labs cloud service. We support
Even if your application is running locally behind a firewall, Sauce provides a secure tunnel called Sauce Connect, which allows our cloud servers to securely access your application behind the firewall.
Keep in mind - Sauce is a commercial service. We have a 14 day Free Trial which you can use to run automated or manual tests.
However, we’re completely free for open source projects - in fact, we are used by Mozilla, YUI, Travis, JQuery, and many other popular open source projects.
Running the same integration and acceptance tests on the Sauce cloud
In the JUnit test folder, there is another file called SauceIT.java. These are exactly the same Acceptance Tests as we described above, but they run on multiple browsers in the Sauce Cloud. Here’s how it works -
The following variables define the Sauce Connect tunnel (assuming you are testing behind a firewall), and the operating system, version
private static final SauceConnectFourManager sauceConnectManager = new SauceConnectFourManager();
private final String os;
private final String version;
private final String browser;
Like in the previous acceptance test, we instantiate Webdriver and a local Jenkins instance:
private WebDriver webDriver;
@Rule public JenkinsRule jenkinsRule = new JenkinsRule();
We select the browser/OS combination to be used for the current test:
public SauceIT(String os, String version, String browser) {
super();
this.os = os;
this.version = version;
this.browser = browser;
}
@Parameterized.Parameters
public static LinkedList browserStrings() throws Exception {
LinkedList browsers = new LinkedList();
browsers.add(new String[]{"Windows 2003", null, "chrome"});
browsers.add(new String[]{"Windows 2003", "17", "firefox"});
browsers.add(new String[]{"linux", "17", "firefox"});
return browsers;
}
Here’s the code to start Sauce Connect - allows Sauce Labs machines on the cloud to seamlessly connect to a website or mobile site running on your local network.
public static void startSauceConnect() throws IOException { sauceConnectManager.openConnection("SAUCE_USER", "SAUCE_ACCESS_KEY", 4445, null, null, null, null); }
And lastly, we create a RemoteWebDriver instance that is configured to launch a specific
public void setUp() throws Exception {
DesiredCapabilities capabilities = DesiredCapabilities.firefox();
capabilities.setCapability(CapabilityType.BROWSER_NAME, browser);
capabilities.setCapability(CapabilityType.VERSION, version);
capabilities.setCapability(CapabilityType.PLATFORM, Platform.valueOf(os));
this.webDriver = new RemoteWebDriver(
new URL("http://SAUCE_USER:SAUCE_ACCESS_KEY@ondemand.saucelabs.com:80/wd/hub"),
capabilities);
URL url = jenkinsRule.getURL();
webDriver.get(url.toString());
}
That’s it! The rest of the code is the same as in the regular acceptance test above. Only now you can test against any number of OS/version/browser combinations in the Sauce cloud.
Summary
We covered a lot of stuff in this post. We hope we’ve made it clear that JUnit users should be doing more than just unit testing. It is quite easy to add integration and acceptance tests with UI automation to your JUnit testing repertoire.
Our code example of a simple Jenkins plugin demonstrated how within one JUnit test suite, you can run regular unit tests, integration tests, and acceptance tests with full UI automation. We showed some pretty advanced browser automation flows, achieved in only a few lines of code.
Finally, we took the liberty of showing how the Sauce Labs cloud service fits in - helping you easily run the same Selenium tests on a large number of browser/OS platforms.
Here’s to