So you’re looking to integrate Selenium WebDriver with Jenkins, in order to run full end-to-end / UI automation testing as part of your build process? Well, we have good news and bad news:
The good news is that it’s relatively easy to run Selenium tests as part of the Jenkins build, assuming you’re already using a testing framework like JUnit or TestNG. We’ll explain a few ways to achieve it further on this page.
The bad news is that there are quite a few challenges to consider:
Builds with Selenium tests take much longer to run.
You’ll find it extremely hard to cover all the relevant browsers and platforms.
Once it runs, u expect ongoing maintenance work as browsers and operating systems become out of date, update themselves, perhaps beyond the support of the current webdriver.
Scaling up your testing will be complex and you’ll need to deploy Selenium Grid and/or rewrite your tests to support multi-threading.
We at Sauce Labs run over a million Selenium tests every day on our cloud testing platform. We have an easy-to-use Selenium Jenkins Plugin that allows you to run your Selenium tests, with effortless scalability and cool features. (Keep in mind Sauce is a commercial service, but we have a free trial and is free for open source projects.)
If you’d like to do Selenium with Jenkins yourself, using open source software, check out our summary table of Jenkins Selenium solutions, or visit the sections below to see how it’s done and the pros and cons of each solution:
Option 1: Selenium on one machine- headless testing - running Selenium directly on your Jenkins workstation with a headless browser, probably the easiest option. Headless browsers do not pain the Graphical User Interface on the screen, and are much faster - though there may be rendering issues or race conditions the headless browser can not find.
Option 2: Selenium on one machine - real browsers - running Selenium on your Jenkins workstation with real browsers, limited to the capacity of your workstation. Note this can be extremely slow - best to focus on a few of the most powerful tests.
Option 3: Selenium Grid local- real browsers on multiple machines - running Selenium on several machines on premises, using Selenium Grid, and activating them via Jenkins. To do this you’ll register the machines that execute tests with your own server/coordinator, which might or might not be your jenkins machine.
Option 4: Sauce Labs Jenkins Plugin how to setup the plugin and run Selenium Jenkins tests on over 250 browser/OS combinations in the Sauce cloud, with no need to install browsers and set up special infrastructure in house.
Heads Up: Challenges You Might Face When Running Selenium Tests with Jenkins
It’s not fun reading about problems. But it’s even less fun running into them further along your project! So we hope to save you a bit of pain with our take on Selenium Jenkins integration challenges.
(This discussion is based on our experience with over 500 million Selenium tests run on the Sauce Labs cloud testing platform - many of them through CI platforms.)
Challenge 1: Selenium tests take a long time to run
We've mentioned this before. It’s likely that Selenium tests take much much longer to run than all your unit tests put together,.Selenium tests run on a real browser, they need to perform actual browser operations and often wait for an HTTP server to respond. Plus starting and stopping the browser takes time. As new tests are added for every story as part of the development process, the time to run them increases ... you get the picture.
The implication is that if you include Selenium tests as part of your build, the build will take much longer to run - so if currently, you’re running a build on every commit, or several times a day, you may have to resort to running the build overnight, and you might need to upgrade your Jenkins workstation or even add more machines to your Jenkins cluster. There are two ways to achieve this:
Setting up a local Selenium Grid.
Running your tests on Sauce Labs with the Sauce Jenkins Plugin.
Challenge 2: Limited browser coverage
If you choose to run your Selenium tests on real browsers (as opposed to headless browsers), you will actually need to install multiple browsers on your Jenkins workstation. You will only be able to run Selenium tests against browsers that are actually installed:
If the workstation is Windows, it is difficult to test on OSx. If it’s Mac, you can’t test on Internet Explorer - except by running a Windows VM, which can be exceeding complex to coordinate.
With a Windows machine, you won’t can’t test how Firefox behaves on Linux, how Chrome behaves on Mac, etc.
Because tests take a long time, even if you have installed numerous browsers, you probably won’t be able to run all your tests on all of them. That will make the build even longer.
You might be thinking:
No biggie, I’ll do headless testing! Sure, and we explain below how it’s done. But keep in mind it’s not a true test of the browsers used by your visitors. In the end, someone has to test the application on actual Firefox, Chrome, etc.
I’ll use Selenium Grid! This way you can run many more browsers and reduce the build time. But it can turn out to be quite complex.
Let’s do multi-threading! Okay, but you might need to rewrite all your tests. See our series of blog posts explaining how to do multi-threaded testing with JUnit 4. This will definitely speed up the build but it will still take a long time compared to non-Selenium testing.
Or - you can use the Sauce Labs Jenkins plugin to run the tests seamlessly on the Sauce cloud. This will give you access to a comprehensive and growing list of over 800 browser/OS combinations.
Challenge 3: Updating browsers
If you go down the “real browser” route (as opposed to headless testing), you’ll need to worry about updating your browsers. Each time a new version of IE / Firefox / Chrome / Safari comes out, you’ll need to install it on your local machine/s. And uninstall old browsers that are no longer relevant. And deal with browsers that want to auto-update themselves, while you want to test specifically on an older version. Auto-updates can happen during a test run, invalidating the run. All of this creates a big ongoing maintenance effort.
At Sauce Labs, a big part of our service is taking care of this maintenance effort for you. We can set up a cloud VM for you on the fly which has practically any combination of browser and operating system, and we work hard all the time to add the latest platforms.
Challenge 4: Difficult to scale up
It’s important to consider how you’ll scale up your testing in the future. If you add more Selenium tests - for example, to test more functionality or new features - this will increase the build time and the system resources required. Also, if you want to test on additional browsers/platforms, this will increase build time.
Keep in mind that your test running capacity will be limited by either:
The capacity of your Jenkins workstation (unless you distribute builds with Jenkins slave nodes), or
The number of machines available in your local Selenium Grid if you set one up.
One approach to solving this is headless testing, which runs much faster than real-browser testing. Another approach is our Sauce Labs Jenkins Selenium plugin. Sauce Labs does Selenium testing on the cloud for thousands of organizations. We have an easy-to-use Jenkins integration that lets you run your Selenium tests on our cloud service, with effortless scalability and cool features.
4 Approaches to Integrating Jenkins with Selenium - Pros and Cons4 Approaches to Integrating Jenkins with Selenium - Pros and Cons
Solution | Complexity | Pros | Cons |
. Run the Selenium tests directly on your Jenkins workstation. Use headless browsers to save time and at least approximate GUI behavior. | Low | Tests take much less time to run (compared to real browsers) No need to update browsers - no real browsers used Easier to scale, system resources used Open source | Not testing real browsers and platforms Still increases build time compared to unit testing |
. Run Selenium tests directly on your Jenkins workstation against real browsers. Build takes longer, but you’ll be able to test real browser reactions. | Medium | Testing real platforms Open source | Tests take long to run Difficult to scale Limited browser coverage - only what you have installed on Jenkins workstation Need to install, update, maintain browsers |
. Run a battery of Selenium tests quickly and on a wide range of browser/OS combinations, by investing in infrastructure. Set up Selenium Grid to distribute test execution across multiple nodes. | High | Tests take much less time to run (compared to Selenium on one machine) Testing real platforms (unlike headless testing) Ability to support more browsers and operating systems Open source | Need to update browsers Difficult to scale further - need to add more machines and configure them cost of hardware and ongoing maintenance Browser coverage still limited to the number of computers / operating systems you can physically set up. There are more than 800 OS/browser combinations in use today. |
Sauce Labs Jenkins plugin. Outsource the infrastructure to Sauce Labs, run tests quickly on over 800 browser/OS combinations, without disrupting your CI process and without installing and maintaining browsers locally. | Low | Tests take much less time to run Full browser coverage Testing real platforms Ability to scale effortlessly Browsers are updated for you | Commercial service - beyond the free plan limit, priced per minute. (Sauce is free for open source projects.) |
Option 1. How to Run Selenium on One Machine - Headless Testing
The simplest option is to run the Selenium tests directly on your Jenkins workstation. In such a configuration, running tests serially against full-fledged browsers is going to take a lot of time and computer power. One way to economize is to run them using headless (“GUI-less”) browsers. You’ll save time configuring and running these emulators, and at least approximate GUI behavior.
There are several ways you can set headless Selenium testing, depending on your browser needs and operating system. Some popular headless browsers are HTMLUnit, Xvfb, Spynner, Cabybara, Ghost, and Headless Chrome. As an example, we’ll show how to perform functional tests on Web applications using Selenium and headless Chrome.
Chrome Headless runs a full browser stack with JavaScript access, only without the GUI. Chrome Headless can run on practically all platforms. It is suitable for automated testing as part of a CI system like Jenkins because it can be installed directly on the Jenkins node and executed from the command-line.
In Chrome headless is simply a selenium run-time option. In Ruby, this is as easy as adding this code when creating a WebDrive object:
chromeOptions.addArguments("--headless");
Paste that code into line 11 of css_focus.rb and see it run yourself. Any existing code that uses a CI pipeline will continue to execute under Chrome Headless -- just faster.
Pros and cons of headless Selenium testing
Pros
The main advantage of headless browser emulators is that tests run much faster than traditional GUI tests.
Headless browsers are easy to run and configure, you can run one on practically any platform.
They are easy to automate as part of a continuous integration flow, because they run from the command line and return plain ascii pass/fail results.
New technologies are emerging to package headless browsers in extremely small containers - making test spin-up in two or three seconds.
“Closer to the metal”, headless tests are more likely to exercise small components and work within the development cycle, instead of after it.
Cons
The browser is abstract, does not simulate user actions on real platforms. Headless browsers can mask many problems that only real browsers encounter.
With no visibile interface, there are no videos available to help with debugging.
Headless browsers have partial JavaScript/AJAX support. So, you cannot fully test web applications which include Javascript and AJAX elements.
Option 2: Selenium on one machine - real browsers
Let’s say you want to run Selenium tests directly on your Jenkins workstation against real (non-”headless”) browsers. Your build will take longer, as it runs full-blown browsers and full UI automation, but you’ll be able to measure real browser reactions on the most common browsers used by site visitors.
Following is one method of achieving WebDriver integration with Jenkins, using Maven.
Here are the general steps:
Install, on the Jenkins workstation, all the browsers you want to test your application on (e.g. Chrome, Firefox, Internet Explorer).
Ensure Jenkins is up and running.
Ensure Java and Maven are installed.
Run Tests locally first (4 steps)
Create a folder with this structure:ProjectName-- pom.xml-- src----- test----- java----- YourTest.java
Create a pom.xml file
Put your Selenium scripts in Java folder: [/ProjectName/src/test/java]
Open a terminal, and cd to [/ProjectName], then run: mvn test
You should see your tests running and the results once they are complete.
5. Now, integrate with Jenkins (9 steps):
Create a “Build a free-style software project“ Jenkins job.
Configure newly created job.
Reference your source code using any available version management system (Git, SVN, CVS or other).
Run the Jenkins job against your local directory.
Select ‘Invoke Top-Level Maven-targets’ from the Build dropdown box.
Enter ‘test’ as Goals.
Save the configuration.
Run the job. You should expect a failure first time round.
Now copy all you data under your project directory to the Jenkins workspace and run the job again. Now you can see tests running.
For more details, see this blog post from Anoop Philip.
Pros and cons of Selenium on one machine with real browsers
Pros
Testing real platforms - in this method you can test at least a few of the most popular platforms used by your visitors.
Open source, no infrastructure costs - Selenium is free and since the browsers will typically run on the existing Jenkins workstation, there is no additional investment in infrastructure.
Cons
Tests take long to run - because Selenium tests involve starting up real browsers, waiting for one or more web pages to load, and performing actions on them, these tests take significantly longer to run than regular unit tests. If you were previously used to running builds on every commit, or several times a day, you will be able to run much less frequently, or possibly resort to a nightly build. It is still possible to run builds during the day, skipping the Selenium tests.you will probably install more than a few browsers on the local machine, and then, you need to install new versions of the browsers as they come up, uninstall old versions, and deal with browsers wanting to automatically update themselves, while you are interested in testing specifically the old version. This becomes a large, ongoing maintenance effort.
Difficult to scale - if you want to scale up the number of tests, you might need to add more machines, and this will require using Selenium Grid see how to do this below.
Limited browser coverage - this scenario involves testing on browsers physically installed on the local Jenkins workstation. There are only so many browsers you can install on one machine, and if, for example, you are running on Windows, you can’t test Safari, nor can you test Internet Explorer on a Mac, without running virtual machines. So this scenario is inherently limited in the type and number of platforms you can test.
Need to install, update, maintain browsers - you will probably install more than a few browsers on the local machine, and then, you need to install new versions of the browsers as they come up, uninstall old versions, and deal with browsers wanting to automatically update themselves, while you are interested in testing specifically the old version. This becomes a large, ongoing maintenance effort.
Option 3: Selenium Grid local - real browsers on multiple machines - under Jenkins
If you need to run a battery of Selenium tests quickly and on a wide range of browser/OS combinations, you will need to invest in infrastructure in the form of additional machines, installed with different operating systems and browsers, to cover the additional scenarios. Then, set up Selenium Grid to distribute test execution across multiple nodes.
To get set up, you'll need:
Standalone jar for selenium.
Chrome driver
Both can be downloaded here: https://code.google.com/p/selenium/downloads/list
These are the general steps to get it to work:
Set up two or more computers on which you want to run Selenium tests with real browsers. This is your Selenium Grid cluster.
Create a directory that looks like this:
(dir) chrome-<ver_1>
(dir) chrome-<ver_2>
(dir) chrome-drive-zipfiles
(dir) jdk android-server
chromedriver
grid-startup-batch
selenium-server-standalone
Write a batch file which spawns a WebDriver node as well as the Selenium Grid hub (we’ll call it grid-startup-batch.bat). Run the batch file on one of the Selenium Grid machines.
Open the Grid console, by firing up a web browser of your choice and type: http://localhost:4444/grid/console
Inspect the grid console. Move your mouse over the browser icons to check browser names,
