Getting Started With Selenium - Chapter 8: Tying It All Together

This post is the eighth and final in a series of “Getting Started with Selenium Testing” posts from Dave Haeffner, a noted expert on Selenium and automated testing, and a frequent contributor to the Sauce blog and Selenium community. This series is for those who are brand new to test automation with Selenium. 

You'll probably get a lot of mileage out of your test suite in its current form if you just run things from your computer, look at the results, and inform people on your team when there are issues. But that only helps you solve part of the problem. The real goal in all of this is to find issues reliably, quickly, and continuously -- and ideally in sync with the development workflow you are a part of. In order to do that, we want to use a Continuous Integration (CI) server.

Continuous Integration (a.k.a. CI) is the practice of merging code that is actively being worked on into a shared mainline (e.g., trunk or master) as often as possible (e.g., several times a day). This is with the hopes of finding issues early and avoiding merging and integration issues that are not only considered a special kind of hell, but can dramatically slow the time it takes to release software. The use of a CI server (a.k.a. build server) enables this practice to be automated, and to have tests run as part of the work flow. The lion's share of tests that are typically run on a CI Server are unit (and potentially integration) tests. But we can very easily add in our automated acceptance tests. There are numerous CI Servers available for use today, most notably:

In order to get the most out of our test runs in a CI environment, we want to break up our test suite into small, relevant chunks and have separate jobs for each. This helps keep test runs fast (so people on your team will care about them) and informative. Gojko Adzic refers to these as 'Test Packs'. The workflow is pretty straightforward. The CI Server pulls in the latest code, merges it, runs unit tests, and deploys to a test server. We then have the CI Server kick off a new job to run a subset of our acceptance tests -- our critical ones (e.g., smoke or sanity tests). Assuming those pass, we can have another job run to promote the code to another environment (e.g., for manual testing) or fire up the remaining (longer running) tests. Adam Goucher refers to this strategy as a 'shallow' and 'deep' tagging model.

In order to make the test output useful for a CI Server we need to generate it in a standard way. One format that works across most CI Servers is JUnit XML. Your test runner should be able to accommodate this either out of the box, or through the addition of a third-party library or plugin (e.g., like rspec_junit_formatter for RSpec. If you're using Sauce Labs, then you will get all of the human readable test output you need to diagnose a test failure at a glance. You will just need to find a way to tie the Sauce Labs job to the failed test. In RSpec, this can be accomplished through creating custom failure messages that will output the Sauce Labs job URL. Alternatively, you can use a CI server plugin to help accomplish the same thing.

Here are the basic steps that go into setting up a simple Jenkins job that will be trigger dynamically. 1. Create a Job 2. Pull In Your Test Code 3. Set up Build Triggers 4. Configure Build steps 5. Configure Test Reports 6. Run Tests & View The Results 7. Set Up Notifications (e.g., e-mail, chat, etc.)

Your Selenium journey may prove challenging at times. But by adhering to the principals in this course, you are better equipped to start off on the right foot and avoid common Selenium pitfalls. You are now ready to continue your Selenium journey on your own. For a full list of additional Selenium resources -- check out this write-up. Happy Testing!

Dave is the author of Elemental Selenium (a free, once weekly Selenium tip newsletter that is read by hundreds of testing professionals) as well as a new book, The Selenium Guidebook. He is also the creator and maintainer of ChemistryKit (an open-source Selenium framework). He has helped numerous companies successfully implement automated acceptance testing; including The Motley Fool, ManTech International, Sittercity, and Animoto. He is a founder and co-organizer of the Selenium Hangout and has spoken at numerous conferences and meetups about acceptance testing.