Web application testing turns out to be very similar to an onion. You keep peeling back layers and finding more testing that can, and probably should, be done. But you shouldn't only be concerned with testing the smooth outer layer that your users will see. There's still plenty of automated testing to be done on those inner layers, and it doesn't require much extra work. The testing I'm referring to is the JavaScript unit tests. For most development teams, this is an incredibly important step and should be done well before using Selenium for testing web UI and user flows.
Most people think about Selenium as the "click" here, "type" there project. And for many intents and purposes, that's correct. However, Selenium 2 is offering some new functionality that lends itself to a slightly bigger set of use cases. What I'm talking about is the 'execute' method. This allows you to execute arbitrary JavaScript in the AUT (application under test) scope, without having to deal with the frustrating complexities of 'getEval' in Selenium 1. Specifically because of this functionality, I wrote a driver for Jellyfish (jelly.io), a framework that allows you to write node .js scripts that run JavaScript in the browser. In order to do this, I implemented a tiny subset of the WebDriver JSON wire protocol in node.js and called it 'wd'. These projects are interesting all in their own right, but to get straight to the point: If you have tests in FooUnit, QUnit or any other JavaScript testing framework that run in the browser, we now have a way for you to do all of that in the Sauce OnDemand cloud.
For this example, lets use QUnit. In the hopes of covering all the bases, lets assume your QUnit tests are hosted on an internal web site to your company. Lets also assume that the way you currently run them is by opening a browser window and going to 'file open', or navigating to a local URL and sitting there watching the tests run. When they are done, you'll manually look through the tests and results to decide if you broke things or not. With Jellyfish and OnDemand, the way this now works is that you have a node.js script that runs after every check-in (perhaps by Jenkins) using different browser and platform combinations. When the tests are complete, an email will be sent to all your developers each time you have a failure.
So how do we do this? Start by installing Jellyfish, the directions for which are front and center at jelly.io. Next we will build a straightforward Jellyfish script that will start a Sauce Labs session, navigate to your QUnit tests URL, and assert the results when they finish. For my examples, I am going to be using the full 'JQuery Test Suite' that I got from 'jquery.com'. Lets talk about the contents of the script one piece at a time:
var jellyfish = require('jellyfish') , assert = require('assert');
This imports the jellyfish package and the assert module, which is a nice lightweight way to make assertions in node.js.
var sauce = jellyfish.createSauce(); sauce.opts.platform = "VISTA"; sauce.opts.browserName = "iexplorer"; sauce.opts.version = "9";
This instantiates a new Jellyfish object and lets it know that this one will be talking to Sauce Labs. The username and password for your Sauce Labs account can be passed through here directly, but in this case, I'm storing it in my ~/.jfrc file --, which is nice for simplicity. Next we'll specify the platform, browser, and version we want to use to run our test (more information about .jfrc can be found here).
sauce.on('result', function(res) { console.log(sauce.name + ' : '+sauce.tid + ' - \x1b[33m%s\x1b[0m', JSON.stringify(res)); });
This one is optional, but isn't it nice to get well-formatted output in your logs?
sauce.on('complete', function(res) { console.log(sauce.name + ' : '+sauce.tid + ' - \x1b[33m%s\x1b[0m', JSON.stringify(res)); sauce.js("window.testOutput", function(o) { try { assert.equal(o.result.failed, 0); sauce.stop(function() { process.exit(0); }) } catch(e) { console.log(e); sauce.stop(function() { process.exit(1); }) } }) });
This listener is very important since generally our communication with Selenium on Sauce OnDemand is one directional. However, that has been abstracted away. During the script, Jellyfish will poll Selenium every few seconds and ask 'has window.jfComplete been set to true'? If so, an event will be triggered called 'complete'. In the next section, you will see how the results are set, but at this point, I just have Jellyfish ask the page for the contents of 'window.testOutput'. Next we assert the value of the number of failures. In this case, I am catching that exception so that I can print it to the terminal and exit the script with the correct return code for CI.
sauce.go("http://myurl.com/jquery/test/index.html") .js("QUnit.done = function(res) { window.testOutput=res; window.jfComplete=true; }");
This is where the magic happens. This code navigates to the URL of the QUnit tests that start automatically, and as soon as we have navigated there, I redefine the QUnit done event to set the window.jfComplete flag. Then I set the results output of the test run in a place I can access it in the above code. Usually the first question asked here is how to access QUnit tests that are on an internal network. Fortunately, we have a product at Sauce Labs called Sauce Connect that allows our cloud's VMs to securely access whatever your test machines access on the internal network. It is incredibly simple to setup and you can do so by following the comprehensive documentation. After that tunnel is running (you can just leave it running), you can now update the start URL in 'sauce.go' to reflect the URL you use internally to access your tests in the browser. Jenkins has hooks for essentially any deployment platform or source control repository, and it also has post-deploy hooks that can be configured to send you an email. You can also use either a 'matrix project' or environment variables to make one job run your test on all of our available platform and browser combinations. The example script can be downloaded here from github and I also made one of my Jellyfish QUnit test runs public for your viewing pleasure. Happy testing!
