Okay, honesty time: browser testing sucks. It’s stuck in the stone age. Browser tests are great when they’re working well, but they’re way too hard to get right, way too expensive to maintain, and just overall not a lot of fun. That’s because existing tools lead us to build brittle, sequential tests that can be clobbered by tiny application tweaks. What starts as straightforward test creation ends up with data, actions, and selectors thrown together in a big mess. We paper over the mess with abstractions like Cucumber and Capybara, but we haven’t addressed the fundamental issues.

We’re the cavemen of browser tests.

Fundamentally, these are the problems with browser tests as they’re written today:

• Getting started is hard. It’s important to structure your tests in a way that will work out well long-term, but guidance is scarce.
• Browser tests are brittle. Small changes in the DOM, in application performance, or in other tests often generate a lot of menial labor to repair tests. A single DOM change in your application can easily cascade into hundreds of broken tests — and cost staff days to fix them.
• Browser tests are slow. The solution is to run many tests in parallel, but current testing frameworks lead us to write tests that don’t parallelize well.

All this makes writing and maintaining tests an expensive, and more importantly unpleasant activity.

We can do better. It’s time for a new way. As an industry, we know how to do this a lot better than we’re managing to in most cases. It’s time to accept a standard, move on, and get things done.

Test::Right aims to be that standard.

The ultimate goal of Test::Right is to make it impossible to write bad tests. That’s right, read it again. Impossible to write bad tests. That’s a serious goal, but we’re serious about it.

Are we there yet? No. But we’re a whole hell of a lot closer than anything popular out there today.

Our philosophy in creating Test::Right is to directly address the problems with browser testing from the start:

• To make it easy to get started, test authors need an opinionated framework that chooses a thoughtful structure for tests based on our extensive experience working with large test suites.
• To make browser tests less brittle, we have decoupled intentions from mechanics. Stable intentions like “log in” and “check account status” are disentangled from volatile mechanisms like “click the DOM element identified by this CSS selector”. We isolate the parts of tests that change frequently in a single place instead of scattering and duplicating them throughout tests, making tests robust and maintenance easy.
• To make browser test suites run faster, we parallelize your tests from day one. Your builds will always be fast, and because every build checks that your tests run in parallel, you’ll never develop performance problems that will bite you later.

Test::Right is an idea whose time has come, born of dozens of on-site integrations of our service in household-name companies and some not-so-household name places. We as a small team have seen more browser test badness than any sane person should have to, and the tools are at fault. Test::Right is designed to address the mainstream case we’ve had the privilege to experience over the past 16 months.

We’re releasing it as a work in progress, but it already has a ton of useful functionality. We’ve put a stake in the ground with a whole slew of opinions that are sure to get a discussion started. And that’s really where we need you.

We built Test::Right by borrowing concepts from the Page Object model, but went even further by enforcing clean separation of concerns. With Test::Right, your test cases are defined in terms of actions and properties on widgets. A widget is a piece of functionality present on one or more pages of your application. A single page can have many widgets, and multiple copies of a widget may appear on the same page. Tests are grouped by feature, and tests don’t have direct access to the underlying Selenium object.

• Widgets bring the Page Object concept into the Ajax age and decouple tests from DOM, making it far easier to adapt to frontend tweaks. They give your tests an application-specific vocabulary that maps directly to what users do with your app, and let tests make assertions about high-level aspects of your app, instead of low-level DOM state.
• Automatic spin asserts ensure that your assertions work as you expect them to, even with Ajax.
• Out-of-the-box parallelization gives you speed and scalability from day one.
• Random test ordering prevents you from writing bad tests that serially build up state, depend heavily on one another, and can’t be parallelized.
• Data factory generates isolated test data so that tests don’t clobber each other.

Test::Right is in Ruby because it seemed like a good place to start. We believe these ideas make as much sense in any other language.

Whether or not you use Test::Right, we want you to come away from it with ideas and opinions. The browser testing community needs to wake up and start writing good tests, and with Test::Right we’re going to make that happen.

And I, personally, look forward to hashing out the future of browser automation with you.

Check out the Test::Right source code on GitHub to get started. Let us know what you think, and enjoy!

So I heard you have a PHP website you want to test. I guess you already heard that Selenium is the way to test a website, and that Sauce OnDemand is the way to use Selenium. Since you want to do the right thing and use Sauce, I have two (2) pieces of good news for you:

1. I wrote this post on how to use Sauce and PHP to test your website
2. You’re reading it

Okay, PHP has this cool thing called pear that you can use to download useful php scripts. We’ll spend pretty much this whole blog post getting pear, then using it to get the Sauce OnDemand pear package and all of the other pear packages it depends on.

Things to know about pear:

• You download pear packages from channels.
• Most of the time, when you need a package, you first have to tell pear about the channel. Pear calls this “discovering” the channel.
• If a package depends on another package, it will tell you about the other package you need, but it won’t automatically go get it for you.
• There’s a new version of pear called Pyrus and we aren’t going to use it

Okay. So, before we can use pear, we have to go get it.

Get Pear

This part depends on what operating system you’re using. Below are the easiest ways I’ve found to install pear for each OS. If they don’t work for you, you can always follow the instructions on the pear website.

Mac

Get mac ports. Then:
$sudo port install php5 +pear
This might complain that you don’t have all of the stuff php5 needs to install. If it does, it should also tell you what you need to do to wrap that up.

Linux

$ sudo apt-get update
$ sudo apt-get install php-pear


Windows

There’s a php installer here you can use to reinstall php. One of the steps (shown here) lets you install pear along with php. That’s all you need to do.

Now that you have pear

The rest of your setup happens from the command line. If you’re a linux or mac person, you’re already there. Windows people, push windows key+R, type “cmd”, and push enter; when you see a black window with “C:\>” in it, you’re at a command prompt and you’re ready to proceed.

First, upgrade pear:

• pear upgrade pear

Next, discover and update all the channels we’ll need to install pear packages from:

• pear channel-discover pear.phpunit.de
• pear channel-update pear.phpunit.de
• pear channel-discover components.ez.no
• pear channel-update components.ez.no
• pear channel-discover pear.symfony-project.com
• pear channel-update pear.symfony-project.com
  

  

• pear channel-discover saucelabs.github.com/pear
• pear channel-update saucelabs.github.com/pear

Now, finally, install phpunit for Sauce OnDemand:

• pear install -a saucelabs/PHPUnit_Selenium_SauceOnDemand

Last, configure PHPUnit_Selenium_SauceOnDemand to use your Sauce Labs account and access key (which you can find at your account page):

• sauce configure <account name> <access key>

If you don’t have an account name and access key, then it’s time to sign up for Sauce OnDemand!

You’re ready to cook with hot sauce

You get to run tests with hot Sauce! Here’s an example script, and how to run it. Put the following PHP into a file named ExampleTest.php:

<?php

require_once 'PHPUnit/Extensions/SeleniumTestCase/SauceOnDemandTestCase.php';

class ExampleTest extends PHPUnit_Extensions_SeleniumTestCase_SauceOnDemandTestCase {

    function setUp() {
        $this->setOs('Windows 2003');
        $this->setBrowser('firefox');
        $this->setBrowserVersion('3.6.');
        $this->setBrowserUrl('http://example.saucelabs.com');
    }

    function test_example() {
        $this->open('/');
        $this->assertTitle('Cross browser testing with Selenium - Sauce Labs');
    }
}

Notice that the filename, ExampleTest.php, is the same as the name of the test class.
Get excited, because here’s where we finally use Sauce OnDemand:

• phpunit ExampleTest.php

Congratulations! You ran a job with Sauce OnDemand. Go to your jobs page to see it. From here, you can edit the example test to open your own website instead of example.saucelabs.com, and add clicks and assertions to your brain’s content.

We’ll have some scheduled downtime next weekend for stability and performance improvements.  One major theme of this set of upgrades is the introduction of graceful upgrade paths for major components, to reduce the need for scheduled downtime in the future. We expect to be down for 4 hours beginning at 1 PM PDT (2000 UTC) on Saturday August 15th.

We’ll have some scheduled downtime this weekend for upgrades and stability improvements.  We expect to be down for 4 hours beginning at 1 PM PDT (2000 UTC) on Sunday, July 5th.