Additional Configuration :: Sauce OnDemand
In Sauce, we refer to an individual run of a single Selenium test as a "job". For example, if your test suite contains 100 Selenium tests, and you run the whole suite 3 times, OnDemand will keep records of 300 jobs, one for each browser session. The following are additional settings you can use to annotate your jobs and configure Sauce OnDemand on a per-job basis. This is done differently depending on the Selenium version you are using: Selenium 1 or Selenium 2 or with our alternative job annotation methods.
Selenium 1 tests: The JSON Configuration
In Selenium 1 tests, Sauce-specific settings are given inside Selenium's "browser" parameter. This is generally a string in the form "*browser" (e.g. "*iexplore", "*firefox"), but will now need to be a full JSON object like this:
'{"username": "your username here",
"access-key": "your access key here",
"os": "Linux",
"browser": "firefox",
"browser-version": "3"}'
Any key-value pair specified in this documentation can be set through this JSON object.
Selenium 2 tests: Desired Capabilities
In Selenium 2 tests, Sauce specific settings are provided using the Desired Capabilities object that Remote Webdriver libraries provide. Basically any key-value pair specified in this documentation can be set through this hash-like object.
Find more about RemoteDriver and the Desired Capabilities object in Selenium's RemoteDriver wiki.
Name Your Jobs
To make it easier to find and identify individual tests, use the name setting to record test names on your jobs:
| Key | name |
| Value type | str |
| Example | "name": "my example name" |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Record the Build
When looking through test results on our website, you'll probably want to know which version of your app the test was running against. Use this setting to annotate test jobs with a build number or app version. Once you set the build info on your job, it will be displayed on the job results page.
| Key | build |
| Value type | str |
| Example | "build": "build-1234" |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Tag Your Jobs
To filter and group jobs more easily, users can provide tags for them.
| Key | tags |
| Value type | list |
| Example | "tags": [
"tag1",
"tag2",
"tag3"
] |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Record Pass/Fail Status
Selenium handles sending commands to control a browser, but doesn't report to the server whether a test passed or failed. To record pass/fail status in OnDemand, set the passed flag on the job.
Since you can't know in advance whether a test passed or failed, this flag can't be set in the initial configuration. Instead, you'll need to use one of our alternative job annotation methods.
| Key | passed |
| Value type | bool |
| Example | "passed": true |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Record Custom Data
To give you an extensible way to annotate and differentiate tests, OnDemand provides a custom annotation you can set that will accept any valid JSON object.
| Key | custom-data |
| Value type | object |
| Example | "custom-data": {
"release": "1.0",
"commit": "0k392a9dkjr",
"staging": true,
"execution_number": 5,
"server": "test.customer.com"
} |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Make Your Jobs Public
If you want to share your job's result page and video, you can make it public. This can be done manually from the job result page, but also programatically using the public setting:
| Key | public |
| Value type | bool |
| Example | "public": true |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Restrict What's Shown in Public Jobs
If you want to share your job's result page and video, but keep the logs only for you, you can certainly do so. This flag will hide the fancy job log as well as prohibit access to the raw Selenium log, so that anonymous users with the link will be able to watch the video and screen shots but won't be able to see what's being typed and done to get there.
| Key | restricted-public-info |
| Value type | bool |
| Example | "restricted-public-info": true |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Disable Video Recording
By default, OnDemand records a video of every test you run. This is generally handy for debugging failing tests, as well as having a visual confirmation that certain feature works (or still works!) However, there is an added wait time for screen recording during a test run. You can avoid this by optionally disabling video recording with this setting:
| Key | record-video |
| Value type | bool |
| Example | "record-video": false |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
As an alternative, the video-upload-on-pass setting will let you discard videos for passing tests identified using the passed setting. This disables video post-processing and uploading that may otherwise consume some extra time after your test is complete.
| Key | video-upload-on-pass |
| Value type | bool |
| Example | "video-upload-on-pass": false |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Disable Step By Step Screenshots
Sauce OnDemand captures step-by-step screenshots of every test you run. Most users find it very useful to get a quick overview of what happened without having to watch the complete video. However, this feature may add some extra time to your tests. You can avoid this by optionally turning off this feature.
| Key | record-screenshots |
| Value type | bool |
| Example | "record-screenshots": false |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Enable HTML source captures
In the same way Sauce OnDemand captures step-by-step screenshots, we can do the same with HTML source captures for you. Even though this feature is off by default, you can turn it off anytime and find the sources in your job result page:
| Key | capture-html |
| Value type | bool |
| Example | "capture-html": true |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Enable Selenium 2's automatic Screenshots
Selenium 2 captures automatic screenshots in most command failures. We strip these to reduce traffic back and forth from your tests and therefore speed them up as much as possible. You can turn these screnshots back on by optionally turning off this feature.
Notice: this setting only affects Selenium 2 tests.
| Key | strip-se2-screenshots |
| Value type | bool |
| Example | "strip-se2-screenshots": false |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Disable Sauce Advisor
Sauce Advisor analyzes your tests and suggests ways to make them faster and more robust. It may add a small amount of extra time to your tests. To disable this feature, use the following setting:
| Key | sauce-advisor |
| Value type | bool |
| Example | "sauce-advisor": false |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Single Window Mode
By default, to get the most out of videos and screenshots, our tests run in multi-window mode. You can change that by setting the single-window setting to true:
| Key | single-window |
| Value type | bool |
| Example | "single-window": true |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
User Extensions
User extensions are available for custom Selenium functionality on the OnDemand service. Given the URL of a file on an accessible HTTP or FTP server (public or connected with Sauce Connect), OnDemand will download it and use it in your test. A list of several extensions can be provided.
| Key | user-extensions-url |
| Value type | list |
| Example | "user-extensions-url": [
"http://saucelabs.com/ext/flex.js",
"ftp://username:password@server.com/bleh.js"
] |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Custom Firefox Profiles
Custom Firefox profiles allow you to configure the browser running in our cloud on a per-job basis. This includes both plugins and any particular setting your tests may need.
To use this feature, a zip file with the contents of the Firefox profile directory you wish to use needs to be provided. Given the URL of a file on an accessible HTTP or FTP server (public or connected with Sauce Connect), OnDemand will download it and use it in your test.
For more info on Firefox profiles, you can check Mozilla's knowledge base.
| Key | firefox-profile-url |
| Value type | str |
| Example | "firefox-profile-url": "http://saucelabs.com/example_files/notls.zip" |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Note: If you actually zip the directory, it will not work. The zip file needs to contain the contents of the profile, not a directory with the contents of it.
Maximum Test Duration
As a safety measure to prevent broken tests from running indefinitely, OnDemand limits the duration of tests to 30 minutes by default. You can adjust this limit on per-job basis. The value of this setting is given in seconds.
| Key | max-duration |
| Value type | int |
| Example | "max-duration": 300 |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Command Timeout
As a safety measure to prevent Selenium crashes from making your tests run indefinitely, OnDemand limits how long Selenium can take to run a command in our browsers. This is set to 300 seconds by default. The value of this setting is given in seconds.
| Key | command-timeout |
| Value type | int |
| Example | "command-timeout": 300 |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Idle Test Timeout
As another safety measure to prevent tests from running too long after something has gone wrong, OnDemand limits how long a browser can wait for a test to send a new command. This is set to 90 seconds by default. You can adjust this limit on a per-job basis. The value of this setting is given in seconds.
| Key | idle-timeout |
| Value type | int |
| Example | "idle-timeout": 60 |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Avoid Proxy
By default, OnDemand routes all traffic from browsers through the Selenium RC proxy server so that HTTPS connections with self-signed certificates work everywhere. Sometimes, though, the Selenium proxy server can cause problems for some users. If that's the case for you, you can configure Sauce OnDemand to avoid using the proxy server and have browsers communicate directly with your servers.
Note: Avoid Proxy only works with Safari, Firefox, and Internet Explorer on Windows using *firefox/*iexplore/*safari modes.
| Key | avoid-proxy |
| Value type | bool |
| Example | "avoid-proxy": true |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Disable Popup Handler
By default, Sauce OnDemand automatically clicks through some types of browser popup windows, to let tests continue. If that's a problem for one of your tests, you can set this flag to disable the popup handler.
| Key | disable-popup-handler |
| Value type | bool |
| Example | "disable-popup-handler": true |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
Use a specific Selenium version
We keep our service up to date with the current state of the Selenium project. For the cutting edge improvements from the project to make it to our service, we periodically update the default version of Selenium that we are running in our test servers.
If you find any problems with a particular version of Selenium or for any other reason you'd like to keep your tests running on a specific version without keeping up with our updates, you can do so using this key.
| Key | selenium-version |
| Value type | str |
| Example | "selenium-version": "2.8.0" |
Learn more about how to configure your tests with these settings in Selenium 1 or Selenium 2 tests.
The current version being used as default is: 2.6.0.
The list of supported versions you can choose from:
2.0.0
2.0b3
2.0rc3
2.1.0
2.10.0
2.11.0
2.12.0
2.13.0
2.14.0
2.14.1
2.15.0
2.16.0
2.16.1
2.17.0
2.18.0
2.2.0
2.3.0
2.4.0
2.5.0
2.6.0
2.7.0
2.8.0
2.9.0
Alternative Job Annotation Methods
As an alternative to the settings you can provide in advance,
Sauce OnDemand has two additional methods by which tests can set a
subset of the job settings described earlier.
These are generally used to update the job with information that couldn't
be foreseen at the time the test was created, like the pass/fail status of
a test. The methods are:
- Selenium 1's setContext()
- Our REST API
Both of these receive a JSON object and accept the subset of settings described below:
Accepted Keys
In both setContext and the REST API, the JSON object to update the job's information accepts the following keys and values:
Here's an example of the JSON you can send with either of these methods:
{
"name": "my job name",
"passed": true,
"public": true,
"tags": ["tag1", "tag2", "tag3"],
"build": 234,
"custom-data": {
"release": "1.0",
"server": "test.customer.com"
}
}
setContext()
setContext is a Selenium 1 command that is not meant to do anything but send information to the server showing what's going on. When the value passed to setContext starts with sauce:, Sauce OnDemand intercepts the command and parses it for job annotations.
We allow two formats for setContext: basic and advanced. The basic format lets you set tags, name, and pass/fail status for jobs. The advanced format lets you set more fields, and you can set them all in a single command.
Basic format:
this.selenium.setContext("sauce:job-tags=tag1,tag2,tag3")
this.selenium.setContext("sauce:job-name=My awesome job")
this.selenium.setContext("sauce:job-result=passed")
this.selenium.setContext("sauce:job-result=failed")
Advanced format:
Our advanced format involves submitting a JSON-encoded dictionary as the value of sauce:job-info. You can set as many or as few of the fields as you wish. For example, in Java, include the following code in your test to add information to a job:
this.selenium.setContext("sauce:job-info={\"name\": \"my job name\"," +
"\"tags\": [\"tag1\", \"tag2\", \"tag3\"]," +
"\"passed\": true,"+
"\"build\": \"103\","+
"\"custom-data\": {\"field\": \"value\"}"+
}");
Update jobs via our REST API
Our REST API provides a way to set the same additional information in jobs via a JSON object sent with an HTTP PUT command.
Selenium 2 tests, which can't leverage the setContext command, can use this API and some custom code to set job info even after the test is over. Here's an example of setting job info using curl, from the command line:
$ curl -H "Content-Type:text/json" -s -X PUT -d '{"name": "my job name 2"}' http://<username>:<key>@saucelabs.com/rest/v1/<username>/jobs/<job-id>
If you were to use this from your tests, you would probably want to build a simple set of functions that do the request for you. We've built a Java library and here's a Python example you can use and would love to see users share libraries for other languages.
Reading more about our REST API.
Getting Help
We're still working to expand our documentation. If anything here has left you with questions, please ask for help or contribute to our our user forums.