Parallel Automated Browser Testing in PHP using Selenium and Sauce OnDemand

Posted Sep 23rd, 2010

Guest post by Aytekin Tank, co-founder of Interlogy Internet Technologies

If you are in SaaS business, one of the best things you can do for yourself is to setup a continuous automated browser testing environment. It will save you a lot of headache, money, and customers down the road. As a happy new Sauce OnDemand user, I’d like to share our story with automated browser tests and how we integrated a PHP based web app with Sauce OnDemand. Our web application, JotForm, is a web based form builder. Using its drag and drop WYSIWYG form editor, you can create web forms quickly and easily. Since our core application is written in JavaScript, it is extremely important for us to do automated cross-browser tests. We accomplish this by using Selenium. We use Hudson to automate our releases. We do continuous integration and deployment. We release our product ten times a day every day. When a developer makes a commit, Hudson executes hundreds of unit, integration, and cross browser tests. Our service has over 300,000 users and they can be very loud when we mess up. That’s why we care a lot about our tests. If we mess up once, we investigate how we missed it and make sure to add new tests so something like that never happens again.

Why Automated Selenium Tests? If you are developing a web app, you must have already discovered that unit tests don’t work well. They can only really cover the server side and little bit on the JavaScript, but when it comes to testing your code on a multi-browser environment, they are pretty much useless. Browsers constantly change. Just this week, our developers spent a lot of time making our app work properly on IE 9 Beta. Ideally, you would want to test your app on all browsers and all platforms. Selenium is the best tool for the job. You can easily record tests using Selenium IDEand integrate them with your continuous integration tools, such as Hudson or Cruise Control.

Why NOT Automated Selenium Tests In-house? The biggest problem with running Selenium tests is creating and managing many testing environments. Usually if you are running Selenium tests constantly on a machine, it cannot be used by anyone. So in practice, they have to be on dedicated machines. And if your releases depend on the Selenium tests, you have to make sure they are always on and running smoothly. It is a lot of maintenance headache. The other problem with Selenium is that it is slow. It takes Selenium a long time to launch up the browser to do the tests. Unless you have many dedicated machines, the tests have to run sequentially. To speed up our Selenium tests, I looked for ways to run them in parallel. My first approach was to use Amazon EC2, since I am already pretty familiar with it. That turned out to not be a great solution. We still had to maintain many selenium environments. Keeping many EC2 instances always on can be quite expensive. Launching new instances on demand turned out to be too slow for our case.

Final Solution: Sauce OnDemand Then, I found out about Sauce Labs’ Sauce OnDemand service. This is basically an on demand service that lets you run Selenium tests instantly on the cloud. It has many good code samples. So, setting it up was pretty straightforward. The great thing about Sauce OnDemand is that you can choose from many browser setups and run your tests on them instantly. No need to maintain dedicated machines. Unfortunately, there was no documentation about running tests in parallel in PHP. So, we first looked into using PHPUnit’s new parallel processing features: @runTestsInSeparateProcesses and @runInSeparateProcess. They turned out to be too buggy to be usable at this time. So, we hacked together a simple script that runs the tests on the shell as background processes.

A Simple Example I am a big Perl junkie, but since our existing unit and integration tests were written in PHPUnit, we wanted to keep our Selenium tests consistent and write them in PHP. Below, you can find a simple example showing how you can run automated cross-browser Sauce OnDemand tests on PHP. We hacked this together quickly so it is not documented or optimized. But it should give you a good jump-start:

1. BrowserTests.php: Includes actual Selenium tests and simplified for this example with a single test. On our production version, we have all the tests here, since we prefer to save time and run all tests at once on a specific browser. You can create these tests using Selenium IDE or Sauce IDE.

2. allBrowserTests.php: You can run this on the shell then add it to Hudson or any other Continuous Integration tool. Basically, this script runs Selenium tests on the background in parallel. It then checks their log files to see if they finished successfully or failed.

3. SeleniumTestHelper.php: This is the integration script with Sauce Labs. You should change the domain name, Sauce username and access key to your own.

Download all files files here. You must first have phpunit installed to use them. Happy testing! Aytekin Tank is a co-founder of Interlogy Internet Technologies. Their flagship product is JotForm online form builder.

Written by

The Sauce Labs Team


SeleniumCross-browser testing