This is a guest blog post by Matthew of Bootstrap Online LLC. Matthew is a volunteer committer on Gollum, the Git powered wiki used on GitHub.com. Checkout Matthew's GitHub profile and hire him to work on your open source project.
Gollum is GitHub's open source wiki software used by GitHub.com, Nature, and others. Gollum uses Selenium 2 WebDriver with JUnit 4 (Java) to test its live preview feature, which requires a real browser. The WebDriver tests can be run locally with ChromeDriver for development, and are automatically executed by Travis-CI using ChromeDriver.
The first version of live preview used window.location.origin, which is undefined on Firefox. Fixing the issue was not difficult, but the tests should have caught the problem. The previous testing strategy for live preview consisted of waiting for users to report bugs. In one issue, a user said that if live preview was not properly tested, it would be removed from the wiki software. Clearly something had to be done and I emailed Sauce Labs for help.
Sauce Labs' Cloud enables testing across a variety of browsers. Following the Sauce documentation produces sequential tests using one file per operating system and browser combination. Three operating systems (Windows Server 2003, Windows Server 2008, Linux) each with two browsers (Firefox, Chrome) means six files.
JUnit's parameterization feature enables consolidating all of those tests into one. The problem with the default parameterization implementation is that the tests are named using only numbers. This makes identifying which test failed difficult. Parallel-webtest, created DescriptivelyParameterized, which enables meaningful test names while still using parameters.
To run the tests in parallel, parallel-webtest provides a ParallelRunner that extends DescriptivelyParameterized. Running cross browser parameterized tests in parallel is not helpful unless you have more than one browser to test on. I could either setup a local cloud, which realistically was not an option, or use a hosted cloud to run the tests. Leveraging Sauce Labs’ OnDemand service enabled Gollum’s tests to complete in nearly 40 seconds when run in parallel. Without the hosted cloud option, I would not have been able to easily run the tests across different environments.
As tests run in the cloud, the results (success or failure) must be reported. I accomplished this by using behrica’s test watcher and saucerest-java. The finished Gollum tests now work on three environments - local, Travis-CI, & the Sauce Cloud - with no duplication of test logic. The code is available on GitHub.
