The official Sauce Labs Blog
Posts Tagged ‘sauce ondemand’
In case you missed the Bamboo webinar yesterday, here’s a short video that walks you through integrating Bamboo with Sauce OnDemand. Thanks to Jonathan Doklovic and Systems Bliss for creating the Bamboo Sauce Plugin (available for download here), and for providing us with a great tutorial!
The Sauce Labs ecosystem continues to happily grow with the addition of a new CI plugin built specifically for Sauce OnDemand.
Bamboo Sauce, created by Systems Bliss engineer Jonathan Doklovic, enables Bamboo users to easily integrate with Sauce OnDemand to run builds across multiple browsers in parallel. Sign up for the live demo this Thursday at 11 am PST to see how this pairing will speed up your test cycles and reduce the need for costly test infrastructure, while leveraging your existing investment in Bamboo, the popular Atlassian CI server.
Setup is simple, needing only a one-time admin configuration, and builds can be customized to test on a variety of OS/Browser combinations. Bamboo Sauce takes care of Firewall problems by creating a direct SSH tunnel from the cloud to a specific server (when you use Sauce OnDemand, we never see your code – rather, it stays behind whatever guard you have in place), and it also lets you pick which tests you want recorded via Sauce TV.
To find out more, tune in to the live demo, led by Systems Bliss and Sauce Labs. Jonathan of Sysbliss will walk you through setting up Bamboo Sauce, and also answer any questions. To follow along step-by-step, download the plugin and sign up for a free 30 day trial of Sauce OnDemand.
(To note – Bamboo Sauce is still in beta. We’d love to hear any feedback you have, so please let us know!)
Here at Sauce Labs, we work with, quite literally, millions of real-world Selenium tests. A handful of major issues with Selenium come up again and again with our customers. Some are browser configuration details we can tweak in our cloud, while others are beyond the reach of Selenium 1.
The good news for the Selenium community is that Selenium 2 aims to solve these latter issues in a comprehensive way. Most current Selenium users, however, have yet to make the leap to Selenium 2. In the meantime, we’ve been working around some of the biggest issues to make things work well for users of Sauce OnDemand.
First, we’ve brought Sauce OnDemand up to the best Selenium can be for SSL support and Unicode handling. On top of that, we’ve implemented a cross-browser file upload command that works in all versions of Internet Explorer, Safari, and Firefox, along with an automatic OS-level popup killer that deals with Internet Explorer’s tendency to break JavaScript testing environments with modal dialogs.
Testing with SSL is challenging. Testing with Selenium and SSL can seem downright impossible. But it doesn’t have to be so! Selenium users encounter two major issues with SSL: Selenium’s proxy mode interfering with the SSL connection and invalid certificates in staging environments. The solution to both is to use the Selenium project’s “CyberVillains” Certificate Authority.
In browsers other than Internet Explorer and Firefox, Selenium proxies all HTTP and HTTPS traffic. Browsers correctly recognize that the HTTPS connection is being interfered with and prevent Selenium from doing its job. The solution that the Selenium project came up with is to have Selenium’s proxy server sign HTTPS connections itself, using the CyberVillains CA. Once you install this certificate to the Trusted Root Certificate Authorities on Windows, Selenium can re-sign all HTTPS connections with a certificate signed by CyberVillains, allowing Selenium to drive your browser to HTTPS URLs. By default, it will do its best to validate the certificate used to sign the connection Selenium makes to your AUT, but that, too, can be dealt with.
Many teams use SSL certificates in testing that aren’t entirely valid. This presents a problem for in-browser testing, because browsers will complain about the SSL certificates being invalid on HTTPS URLs. CyberVillains to the rescue again! When browsers are configured to use Selenium to proxy all content, Selenium can be launched with -trustAllSSLCertificates, which turns off Selenium’s SSL certificate validation and accepts anything. In *iexplore and *firefox modes (AKA *iehta and *chrome, respectively), browsers are not configured to use Selenium as a proxy server, which usually means the CyberVillains approach doesn’t work. In Sauce OnDemand, we now automatically configure Internet Explorer to use Selenium as a proxy server, even in *iexplore mode. For Firefox, we’ve implemented custom profiles, which give you even more control over your certificate environment.
You can read more about our HTTPS support in the Sauce OnDemand documentation.
We have now enabled full Unicode support for all browsers in Sauce OnDemand. We’ve installed East Asian Language support on all of our Windows robots, so your Unicode will render correctly in screenshots/videos, and you can now make assertions in your tests using unicode characters.
File upload has been a persistent problem for Selenium users. We are excited to announce our preliminary support for cross-browser file upload in Sauce OnDemand. We’ve stepped out of the browser environment to interact directly with the operating system (only Windows for now), downloading files you specify from a URL and entering them into file input fields. We have this working on all versions of Internet Explorer, Safari, and Firefox on Windows, and we look forward to open-sourcing our work when we’ve flushed out the last few issues with it.
We’ve implemented file upload support using the existing attachFile command in Selenium, so you don’t even need to change your tests that already work in Firefox. Just call attachFile with a URL to the file you want uploaded and a Selenium locator for the file input box, and we’ll take care of the rest!
Using the same OS-level hooks we built for cross-browser file upload, we’ve built a popup eliminator that will automatically click “OK” or “Yes” in any browser-level popups that appear in Internet Explorer. For example, even with the CyberVillains certificate, IE 6, 7, and 8 will pop up warnings when opening a page with mixed HTTP/HTTPS content. Our popup eliminator takes care of these and more popups. As we gain confidence in this approach, we look forward to pushing these changes upstream into the Selenium project or open-sourcing them as a standalone wrapper for Selenium.
At Sauce Labs, we’re all about providing the best Selenium experience possible. If you have suggestions for how we could improve these features, let us know on our forums!
Guest post by Aytekin Tank, co-founder of Interlogy Internet Technologies
If you are in SaaS business, one of the best things you can do for yourself is to setup a continuous automated browser testing environment. It will save you a lot of headache, money, and customers down the road.
As a happy new Sauce OnDemand user, I’d like to share our story with automated browser tests and how we integrated a PHP based web app with Sauce OnDemand.
Our web application, JotForm, is a web based form builder. Using its drag and drop WYSIWYG form editor, you can create web forms quickly and easily. Since our core application is written in JavaScript, it is extremely important for us to do automated cross-browser tests. We accomplish this by using Selenium.
We use Hudson to automate our releases. We do continuous integration and deployment. We release our product ten times a day every day. When a developer makes a commit, Hudson executes hundreds of unit, integration, and cross browser tests. Our service has over 300,000 users and they can be very loud when we mess up. That’s why we care a lot about our tests. If we mess up once, we investigate how we missed it and make sure to add new tests so something like that never happens again.
Why Automated Selenium Tests?
If you are developing a web app, you must have already discovered that unit tests don’t work well. They can only really cover the server side and little bit on the JavaScript, but when it comes to testing your code on a multi-browser environment, they are pretty much useless. Browsers constantly change. Just this week, our developers spent a lot of time making our app work properly on IE 9 Beta.
Ideally, you would want to test your app on all browsers and all platforms. Selenium is the best tool for the job. You can easily record tests using Selenium IDE and integrate them with your continuous integration tools, such as Hudson or Cruise Control.
Why NOT Automated Selenium Tests In-house?
The biggest problem with running Selenium tests is creating and managing many testing environments. Usually if you are running Selenium tests constantly on a machine, it cannot be used by anyone. So in practice, they have to be on dedicated machines. And if your releases depend on the Selenium tests, you have to make sure they are always on and running smoothly. It is a lot of maintenance headache.
The other problem with Selenium is that it is slow. It takes Selenium a long time to launch up the browser to do the tests. Unless you have many dedicated machines, the tests have to run sequentially.
To speed up our Selenium tests, I looked for ways to run them in parallel. My first approach was to use Amazon EC2, since I am already pretty familiar with it. That turned out to not be a great solution. We still had to maintain many selenium environments. Keeping many EC2 instances always on can be quite expensive. Launching new instances on demand turned out to be too slow for our case.
Final Solution: Sauce OnDemand
Then, I found out about Sauce Labs’ Sauce OnDemand service. This is basically an on demand service that lets you run Selenium tests instantly on the cloud. It has many good code samples. So, setting it up was pretty straightforward. The great thing about Sauce OnDemand is that you can choose from many browser setups and run your tests on them instantly. No need to maintain dedicated machines.
Unfortunately, there was no documentation about running tests in parallel in PHP. So, we first looked into using PHPUnit’s new parallel processing features: @runTestsInSeparateProcesses and @runInSeparateProcess. They turned out to be too buggy to be usable at this time. So, we hacked together a simple script that runs the tests on the shell as background processes.
A Simple Example
I am a big Perl junkie, but since our existing unit and integration tests were written in PHPUnit, we wanted to keep our Selenium tests consistent and write them in PHP. Below, you can find a simple example showing how you can run automated cross-browser Sauce OnDemand tests on PHP. We hacked this together quickly so it is not documented or optimized. But it should give you a good jump-start:
1. BrowserTests.php: Includes actual Selenium tests and simplified for this example with a single test. On our production version, we have all the tests here, since we prefer to save time and run all tests at once on a specific browser. You can create these tests using Selenium IDE or Sauce IDE.
2. allBrowserTests.php: You can run this on the shell then add it to Hudson or any other Continuous Integration tool. Basically, this script runs Selenium tests on the background in parallel. It then checks their log files to see if they finished successfully or failed.
3. SeleniumTestHelper.php: This is the integration script with Sauce Labs. You should change the domain name, Sauce username and access key to your own.
Download all files files here. You must first have phpunit installed to use them.
Happy testing!
Aytekin Tank is a co-founder of Interlogy Internet Technologies. Their flagship product is JotForm online form builder.
If you’re a regular reader of this blog, you’ve no doubt seen posts about the San Francisco Selenium Meetup group. Organized by Sauce Labs, we invite speakers to present to our enthused attendees once a month on a variety of topics, including integrating Hudson with Selenium, adopting the Page-Model Framework, and using Robot Framework to drive your Selenium tests.
For this last meetup, we were particularly excited about our speaker since he’s the creator of Selenium and an original Saucier. Jason Huggins, co-founder of Sauce Labs, joined us at LinkedIn in what could only be described as a “QA session on steroids.” Watch the video below to hear Jason’s take on things, including the topic on everyone’s mind, Selenium 2.0.
Our next meetup is Sept. 29 at Trulia. Davis W. Frank, of Pivotal Labs, will discuss Jasmine + Selenium, so make sure to join #SFSE today!
Let’s imagine your team is making a chat application. Nothing too fancy, just something simple in which you need to make sure multiple users get together, talk, and read each other.
Doesn’t sound that bad, huh? But how about coordinating those browsers? You need to have different sessions running at the same time in coordination and interacting with each other through your web app.
Now, when we think about parallelization and multiple browsers, Sauce OnDemand is, of course, our first answer, but here, there can be a small limitation. Sauce OnDemand has a special timeout that waits for new commands for no more than 90 seconds*. If it doesn’t get any new command, it will kill the job and assume something went wrong.
This effects us, as we need to get multiple browsers running in parallel and in coordination. As we request them one after another, depending on the amount of browsers and the load our service is put under, the first ones might timeout while waiting for the others to come.
To avoid this, the best solution is to write a multi-threaded script, in which browsers will send heartbeat commands while they are waiting for the rest. Multi-threading can be done with every programming language, but I decided to write my example in Python because I think it’s understandable for almost anyone, even if you’ve never seen Python before. The complete example is in github as a gist, and here is a brief description of what each interesting part is doing:
def get_browser_and_wait(browser, browser_num):
print "starting browser %s" % browser_num
browser.start()
browser.open("/") # if we get here, OnDemand already gave us the browser
browsers_waiting.append(browser)
print "browser %s ready" % browser_num
while len(browsers_waiting) < len(browsers):
print "browser %s sending heartbeat while waiting" % browser_num
browser.open("/")
time.sleep(3)
get_browser_and_wait is the function that will take care of the ‘start the browser’ request in OnDemand and wait till it comes back. Once the server is ready, it will keep sending open commands as heartbeats and waiting until the other requested browsers are ready and waiting too.
thread_list = []
for i, browser in enumerate(browsers):
t = Thread(target=get_browser_and_wait, args=[browser, i + 1])
thread_list.append(t)
t.start()
for t in thread_list:
t.join()
This is the magic multithreading part, in which we iterate over every Selenium instance and call get_browser_and_wait with it. Once we send each browser to a thread, the main thread will continue and get to the t.join() part. By this method, the main thread will wait for every browser thread to complete in order to proceed with the rest of the code.
Again, the full example is up in github: http://gist.github.com/511658. If you’re interested in learning what you can use this kind of test for, stay tuned!
* This helps avoid running (and charging) tests that get disconnected or crash.
Watch Rob Fan, founder and CTO of Sharethrough, talk about how instrumental Sauce OnDemand has been in solving many of their testing woes. Sharethrough, a popular video distribution network, has been a Sauce Labs customer since the beginning, and we’re just thrilled to be part of their testing process!
As part of our featured webinar series, Kohsuke Kawaguchi, creator of Hudson and founder of InfraDNA, recently led an interactive demonstration on using Hudson with Selenium and Sauce OnDemand. In case you missed the live presentation on September 1, check out the recorded video to learn more about:
1. Setting up the Hudson plugin for Sauce OnDemand
2. How using Hudson for CI will drastically speed up your testing process
3. Tips for integrating Hudson with Selenium Grid
4. Tips for integrating Hudson with Sauce OnDemand
Sauce Labs is based on open source and we use a lot of it. We also do our best to contribute back to the community by working on Selenium, organizing #SFSE meetups, answering questions on the messages boards, and more.
Today we are happy to announce selenium.saucelabs.com, a searchable archive for the Selenium IRC channel. selenium.saucelabs.com is powered by “selbot,” and it hangs out at #selenium on freenode.
“selbot” has two main functions:
“selbot” is made of open source components, mainly Supybot and xapian. An open source project itself, you can find the code on github.
“selbot” is a humble start, and we have great plans for it in the future. For now, log in to #selenium on freenode and have a look. And while you’re there, answer a few questions – it’s a great way to give back to the community.
A few weeks ago, we announced the Sauce Flash/Flex Testing System, a new and innovative way to automatically test your apps built with Flash and Flex.
The first functional testing product to support test automation for hybrid web apps, Sauce Flash/Flex Testing System bundles Flex-Pilot’s object explorer API into Sauce IDE, and uses Sauce OnDemand to run tests across multiple browsers in the cloud.
Flex Pilot co-creator Adam Christian, a developer here at Sauce Labs, recently led an informative webinar demonstrating how to setup and use this complete testing system. Watch the video below to learn how easy it is to start testing your Flash and Flex apps today!
