If you’ve been following along with my other posts on building a Selenium testing framework, you know we’ve covered some of the high-level testing concepts and base classes so far. Now that we have defined all our working parts, let’s see how we would pull this above code into our test projects. To summarize specficially what we talked about in the last blog, we have defined the following classes.

• SeleniumTestBase.cs, which handles creating of the test browser, reading from the DB,  and handles changing of active windows or frames.
• SeleniumActionBase.cs, which can update the DB and also can handle changing of the active windows or frames.
• SeleniumPageBase.cs, which checks the starting number of the XPath browser index (IE uses 0 while all other browsers use 1).
• SeleniumPageElement.cs, which handles waiting for and interacting with elements on the page.

Since I mentioned the above classes should be in an external project from your testing project, I will refer to these set of classes in compiled form as the OurSeleniumFramework.dll. This .dll file should be referenced in your test project so test, action, and page classes can extend the base classes we created.

When creating a test project to test a website, it is a good idea to create a project base test that will create all of our action and page classes, as well as having a project base action class that will create all our page classes. Page classes will create SeleniumPageElements, which will be used by action and test classes.

Example Test Project

To demonstrate this structure, I will show how base classes are created and how this can be used to test a login page which has a username field, password field, submit button, login confirmation message that appears after successful login, and error message that appears after unsuccessful attempts. I will now show what each class required to perform a basic login test will look like. I will also add notes in the tests to show why something is being done. See below for all the example test project files. The files will be:

1. ProjectTestBase.cs: Handles NUnit setup, teardown, and page and action initialization functionality for tests.
2. ProjectActionBase.cs: Handles page initialization functionality for actions.
3. LoginTests.cs: Implements the tests for the login page.
4. LoginActions.cs: Implements the steps required to do Login testing.
5. LoginPage.cs: Handles the initialization for PageElement objects which will be used by actions and tests.

Example ProjectTestBase.cs

public ProjectTestBase : SeleniumTestBase
{
#region Action declarations
protected LoginActions _LoginActions;
#endregion

#region Page declarations
protected LoginPage _LoginPage;
#endregion

public ProjectTestBase (IWebDriver webDriver)
: base(webDriver) { }

public void CreateActions(IWebDriver webDriver)
{
_LoginActions = new LoginActions(webDriver);
}

public void CreatePages(IWebDriver webDriver)
{
_LoginPage= new LoginPage(webDriver);
}

[SetupTestFixture]
public virtual void setupTestFixture()
{
//Function is virtual so child test classes can have additional functionality.

//For this example project, nothing needs to be done on this level.
}

[Setup]
public virtual void setup()
{
//Function is virtual so child test classes can have additional functionality.
myDriver = StartBrowser(“firefox”);
CreateActions(myDriver);
CreatePages(myDriver);
LaunchUrl(“http://myhomepage.com”);
}

[Teardown]
public virtual void tearDown()
{
//Function is virtual so child test classes can have additional functionality.
myDriver.Quit();
}

[TeardownTestFixture]
public virtual void setupTestFixture()
{
//Function is virtual so child test classes can have additional functionality.

//For this example project, nothing needs to be done on this level.
}
}

Example ProjectActionBase.cs

public ProjectActionBase : SeleniumActionBase
{

#region Page declarations
protected LoginPage _LoginPage;
#endregion

public ProjectActionBase(IWebDriver webDriver)
: base(webDriver);
{
_LoginPage = new LoginPage(webDriver);
}
}

(I’d like to point out here that in this class, you can put common actions such as successful login so all action classes will be able to perform this step. This is just a nice-to-have, though, and not a must.)

Example LoginTests.cs

public LoginTests : ProjectTestBase
{

[Test]
public void successfulLoginTest()
{
_LoginActions.LoginAttempt(“Jason”, “Smiley”, true);
Assert.That(_LoginPage.ConfirmationMessage.Text, Is.EqualTo(“You are now logged in”).IgnoreCase, “You should have logged in successfuly but didn’t”);
}

[Test]
public void invalidLoginTest()
{
_LoginActions.LoginAttempt(“fake”, “fake”, false);
Assert.That(_LoginPage.ErrorMessage.Text, Is.EqualTo(“Invalid login”).IgnoreCase, “You should have gotten an error message but didn’t”)
}
}

Example LoginActions.cs

public LoginActions : ProjectActionBase
{
public void LoginAttempt(string username, string password, isValid = null)
{
LoginPage.UsernameTextField.Clear();
LoginPage.UsernameTextField.SendKeys(username);
LoginPage.PasswordTextField.Clear();
LoginPage.PasswordTextField.SendKeys(password);
LoginPage.SubmitBtn.Click();

if(isValid != null && isValid == true)
{
LoginPage.ConfirmationMessage.
WaitUntilVisibleAndPresent(“confirmation is not appearing after 5 seconds”);
}

if(isValid != null && isValid == false)
{
LoginPage.ErrorMessage.WaitUntilVisibleAndPresent(“error message is not appearing after 5 seconds”);
}
}
}

Example LoginPage.cs

public LoginPage : SeleniumPageBase
{
#region PageElement declarations
public PageElement UsernameTextField;
public PageElement PasswordTextField;
public PageElement ConfirmationMessage;
public PageElement ErrorMessage;
#endregion

public LoginPage(IWebDriver webDriver)
: base(webDriver)
{
UsernameTextField = new PageElement(By.Id(“username”), webDriver);
PasswordTextField= new PageElement(By.Id(“password”), webDriver);
ConfirmationMessage= new PageElement(By.Id(“success”), webDriver);
ErrorMessage= new PageElement(By.Id(“error”), webDriver);
}
}

• Make it easier to connect customers to service providers and vice versa
• Expand the use cases of Sauce’s browser cloud service and
• Show our appreciation for community cooperation in a concrete way

We’re excited to announce the formation of the Sauce Partner program. The program aims to support testing consultants, QA companies, and OEMs by providing infrastructure integrations, demo accounts, commissions, and more, as they enable software development teams to ship quality code faster using Selenium and the Sauce Selenium infrastructure.

The Sauce Partners program contains three different initiatives to cover the full breadth of needs in our expanding community. Read on to find out which one is right for you.

Authorized Partners

Authorized Partners are expertise centers for Selenium and/or Sauce who handle most, if not all, of the end-to-end QA process. This could include test script creation, test maintenance, Selenium implementation, QTP to Selenium training, etc. At Sauce, we get a number of people who write in saying they want to use our service, but only have manual test cases and need a company to write scripts for them. Or they need individualized help implementing Selenium and Sauce Labs. Since we don’t currently support this ourselves, we’re looking for 1-2 companies highly skilled in Selenium and Sauce who will benefit from some brand-name customer referrals. Requirements are that you pass an annual qualification demo and also pay a fee. In addition to the customer referrals, we’ll list you on our website as an authorized partner and also kick back commissions based on sales volume brought in.

Sauce Maître’ D

We designed the Sauce Maître’ D program for individual agile coaches or testing consultancies that help companies implement automation infrastructure, transition manual test practices to automated ones, adopt Selenium, implement Sauce, and more. The Sauce Maître’ D program is free to participant and includes an unlimited demo account for your consulting demonstration purposes. We’ll issue you a promo code to share with your clients that want to use Sauce. Your clients benefit in that your promo code entitles them to a 5% discount off Sauce’s published price schedule.  On a quarterly basis we’ll send you a 10% commission based on the aggregate spending of customers entering your promo code. Simply enroll here, and we’ll set you up.

Sauce OEM

Sauce Labs built the world’s first public browser cloud to meet the needs of agile shops using Selenium to go fast safely.  But now we’re being increasingly approached by OEMs looking to use our browser cloud for applications beyond Selenium and beyond functional testing (such as Security threat injection, deep web data mining, etc.).  So we created the Sauce OEM program to serve the needs of these emerging use cases in addition to those organizations wishing to private-label Sauce’s Selenium service for use with their own customers.

We want to hear from you!

To learn more about and to join any one of these programs, please visit our partner page and register. Or, email us directly at partners@saucelabs.com. We look forward to partnering with you!

(In case you’re wondering, we will resist the incredible temptation to call this new offering “apple sauce”).

If you’re willing to give us tons of feedback, and will be patient with growing pains, please sign up!

Like the rest of our cloud testing services, each Mac OS X session will include the browser testing features you’re familiar with. You’ll get a video of each session, and can take screenshots at any time. With every new test session, you get a clean-slate, freshly booted machine to control. And when the session is over, we delete the virtual machine, so there’s no lingering data left behind on the machine to worry about.

The Mac OS X private beta will initially be limited due to available systems and we’ll contact you as resourcing permits. First priority will be given to corporate users.

In you’re interested in joining the beta, complete the sign-up form, and we’ll email you soon with details.

This new release includes a mayor fix to an important bug affecting some native clicks on elements. You can check out the official changelog for more information.

Due to our new release process, there will be a testing period before we make this the default version in our service. (Once we’ve decided to do so, we’ll announce it in advance). In the meantime, we advise you to try out your tests in this new version using the following Desired Capabilities/JSON key-value:

"selenium-version": "2.1.0"

We’d love to hear if you see any issues after moving your tests to Selenium 2.1.0. And stay tuned, as we’ll be announcing 2.2.0 as well as other versions through our blog too!

Bob Remeika, senior engineer at Yammer, gave a spirited presentation that left no one questioning his stance on testing (his opening slide – “Test your shit” – really said it all). He gave us an inside look at how Yammer tests using a combination of Jellyfish and Sauce OnDemand, and gave some great advice on knowing what and how to test when you’re just starting out.

We also had Adam Christian, Sauce Labs’ Javascript Aficionado and the creator of Jellyfish, give two talks. The first, a lightning talk titled “Javascript Via Selenium: The Good, The Bad, The Obvious”, covered some of the lesser known things about Javascript testing via Selenium.

The second showed off how you can use Jellyfish, the open source Javascript runner that he announced a few weeks ago, to run your JS unit tests in any environment.

Thanks to Adam, Bob, and Yammer for making this quite the fun and memorable meetup. As always, the San Francisco Selenium Meetup group is free to join & we meet monthly at different venues around the Bay Area to talk all things testing. See you in August!

Selenium 2.0.0

We couldn’t be happier to hear that the 2.0 final release has landed. Everyone on the Selenium development team has done an incredible job moving this forward and making Selenium 2/Webdriver into what we believe is the best tool in the market for browser automation. As of last Friday, 2 hours after the release, we included Selenium 2.0 in our list of supported versions, allowing our users to start running tests on it by providing the right capability into their DesiredCapabilities object.

As of today, we’ve made 2.0.0 the default version for all users, as it’s proven to be the most stable and fast version.

Notice: We know some users were affected by this upgrade due to some newly unsupported commands in this release. We’ve since put in place new steps for making the upgrade process more apparent and painless for our users in the future.

Firefox 5

With the latest Selenium upgrade, support for new browsers was included as usual. And since it’s a fundamental part of our job to keep users up to date with cutting edge technology, we’ve included Firefox 5 support for both your Selenium 1 and 2 tests. Just go ahead and add Firefox version 5 to your list of browsers to test, and you should be good to go.

Scout users can also use Firefox 5 to manually test anytime. (Are you aware about Scout, our cool new tool? If not, you should!).

The new Chrome and Opera drivers

By now, you’ve hopefully seen the video of Simon Stewart presenting the new ChromeDriver during the closing keynote of  the 2011 Selenium Conference. If you were in attendance, you may recall the OH SHIT, THAT’S SO COOL! moment when attendees witnessed the new ChromeDriver running tests at blazing speeds as compared to the old version. But the importance of the new ChromeDriver and OperaDriver (which, as Simon mentioned, is just as fast and robust) is not only in their speed, but also in that they are no longer part of the Selenium codebase. They are now maintained by the right people: the browser vendors themselves. Right on, Opera and Google! We’re hoping the rest will follow along.

You can run tests using the new ChromeDriver by specifying it in your RemoteDriver’s DesiredCapabilities object. We’re currently working on getting support for the OperaDriver and will announce it as soon as it’s there. Here are the official release links in case you’re interested into getting these in your local setup too:

http://seleniumhq.wordpress.com/2011/02/09/operadriver_released/
http://seleniumhq.wordpress.com/2011/07/07/new-chromedriver/

For all of these releases, we owe a huge thank you to everyone on the Selenium development team. You guys are doing a great job, and your contributions to the project are constantly improving the quality of our service. For that (and a lot more), we humbly declare each of you the deserved owners of a Sauce t-shirt!

Happy testing!

Sauce Labs does not employ any artists

This is what our build looked like. He was not happy. Notice how his eyes are red. That’s because of failing tests, which as we know turn builds red. These failing tests weren’t normal failing tests, which are actually pretty happy, because they are telling you how to fix your broken stuff. No, these tests were intermittent. That’s largely the fault of the fact that they were selenium tests – go figure, the selenium shop that sells selenium as a service is having selenium issues, lol (but sadface).

Well, to get out of sadfaceland, we dug into one of the sadtests and saw something surprising. It was a case of selenium looking for an element before it appeared. Wait, what? We already know how to fix that. In fact, we already told the whole world how to fix it: with spinning! Basically, the way a spinassert works is by trying the assertion over and over until it passes. This is a big deal with selenium because trying to interact with an element on a page before the page has loaded is a frequent source of flakey tests. We even went so far as to write Test::Right (source on github), a ruby test framework which goes out of its way to make it impossible for you to do write selenium tests without spin asserts. So what happened?

Well, we don’t use Test::Right for all our own tests. Test::Right is ruby and our tests are written in the same language as our back-end; python. And while we were using spin-style asserts for many of our tests, some of them were written before we discovered the magic of spinning, and some were written when we just didn’t remember to use spinning. Also, there were some things that were not assertions but still required spinning; like, if you want to click on an element in a page, it makes sense to spin-wait for that element to be present before clicking on it. Test::Right does the right thing in all these cases; we just had to be more like Test::Right in-house, to prevent sadness from creeping back into the codebase again.

So the fix was clear: modify our own test framework to always spin-wait all the time for every DOM element to show up, every time anyone tried to interact with a DOM element, automatically. We did that, and it made the butler happy. All of a sudden we could trust the build again, and it didn’t take hours to wait for a nonflakey build so that we could deploy new changes. This in turn meant that we never had to hotpatch production when there was an issue, which meant we didn’t have to spend time cleaning up hotpatches when a deploy happened. It also unblocked people who wanted to push new changes to trunk. Basically the whole dev process got better.

So here’s the thing – our customers are having the same issue. Not everyone uses Test::Right. We know this because we have a support team and customers contact us all the time asking for help with flakey tests. In our over 3,000 support requests, race-condition flakey tests are the #1 cause of pain. The shocker is that except for Test::Right (and now our own internal python nose framework), nobody has this universal fix.

So we’re hoping to make test frameworks for all the common languages – PHP, Java, Python, at least – that do what Test::Right did. We already have our own PHPUnit extension (thanks Jan Sorgalla!) which will probably be the first to get the Test::Right makeover.

Sadly, we are a focused team, and we have many tasks to accomplish, so this fix to selenium client libraries will likely not be imminently released by us. Thus we decided to at least tell you all how to Fix Everything on your own side. Now that I’ve done that, back to programming! Good night and good luck.

Jason Huggins’ Opening Keynote

Dave Hunt & Andrew Smith: Automating Canvas Applications
Despite recent improvements to automated testing tools, there’s still a large gap when it comes to emerging technologies such as HTML5. Recent developments like the canvas element present an interesting dilemma for traditional automated testing as they expose little or no information to debug tools. In order to move forwards, both developers and testers will need to work together. Using Selenium, Java, and JavaScript we will demonstrate writing automated tests for a popular canvas game.

Dima Kovalenko: Selenium and Cucumber
Wouldn’t it be nice to have the BA’s write out the acceptance criteria in plain English, and then have those criteria run as tests? Join us for a beginner to intermediate walk through of Cucumber and Selenium. Learn how to write tests that are easy to understand and run. There will be plenty of examples and sample code to get you going in the right direction.

We had to do that recently, and it turns out there’s an easy way to do it thanks to GitHub and Fabien Potencier‘s Pirum. It’s pretty straightforward. Here come the lists. So many lists! Like 40. Still. Straightforward.

By the way, you’ll need to install pear and install git if you haven’t already.

Woo! Now you have a package and a channel. Next step is to

Add the package to the channel

1. Copy the .tgz file to your pear repository
2. Navigate to that directory
3. Add the package to the channel locally
   1. $ pirum add . [filename].tgz
4. Upload the changes to github! Note that you push to gh-pages and not to master.
   1. $ git add -A
   2. $ git commit -m "Added first version of my pear package. Sauce Labs is awesome"
   3. $ git push origin gh-pages

And you’re done! No more bullets! Or numbered lists. Now the whole world is exposed to your PHP. Hopefully that’s a good thing.

Here’s our Pear channel: https://github.com/saucelabs/pear

Here’s the source we distribute through it: https://github.com/saucelabs/phpunit-selenium-sauceondemand

Special thanks to Jan Sorgalla for showing me by example how to do all this.

Selenium tests sometime fail for no reason. You can eliminate most of their random failures with a special kind of assertion, the spin assert. The problem is that Selenium is overeager, and it needs to chill out and wait.

Selenium tests often fail because they’re too fast. Where a user might wait for a page to load for a few seconds and then click on a link, Selenium will interact with a page at the speed of code, before the page is ready. The way to fix this is to have Selenium repeat its actions and assertions until they work. If you don’t, Selenium races your browser.

You need Selenium to lose the race

Selenium tests pages. It tests them by loading them and then doing stuff on them like it’s a person. Isn’t that cute? Selenium thinks it’s people. Unfortunately for us, Selenium isn’t people, and in not being people, it’s too fast. Websites are made for people, with the knowledge that people are slow. When humans open a page, the humans wait for it to be loaded, then click on stuff. The clever secret of webpages is that they’re not really loaded when they look loaded. That’s because the web page’s authors know we take a second to take it all in before we start playing. So they make pages that look loaded right away, and then use javascript to do more real loading once the page is ‘loaded.’ This javascript sets up for playtime in the background while we’re taking it all in. Selenium has a bad habit of trying to use pages before they’re set up for playtime.

The way to solve that is not telling Selenium to pause for a fixed number of seconds. Pauses have two problems. They make your tests slower than they need to be, and they don’t really fix the problem, just make it less frequent. The best solution is to have Selenium keep trying until it works. If your test is trying to assert that clicking on an “email settings” button pops up an email settings dialog, Selenium should repeat a “click the button -> is the status dialog visible?” loop until the status dialog is visible.

Here’s the problem. When we tell Selenium to wait-for-page-to-load and then click, Selenium waits for the wrong thing. It can tell when all the page’s content has been downloaded and displayed, but it has no idea when the page’s custom javascript has finished loading. Sometimes that functional javascript goes back to the server and fetches more content to be displayed, so you can’t even count on required text being present yet. There are several ways for a page to be “loaded,” and Selenium doesn’t wait for the right one. It races your page’s loading logic, often “winning” the race and clicking on things before they’re ready to be clicked on. You need Selenium to lose the race.

Here’s how to do this

The solution is spin asserts. At Sauce Labs, when we want to verify that some text appears on a page, we use this function:

def wait_for_text_present(self, text, msg=None):
    msg = msg or " waiting for text %s to appear" % text
    assertion = lambda: self.selenium.is_text_present(text)
    self.spin_assert(msg, assertion)

Do you see it calling a function named spin_assert? That function is the key. spin_assert retries the passed-in test function, in this case a lambda expression, over and over until it works. Here’s what spin_assert looks like, minus some bells and whistles:

def spin_assert(self, msg, assertion):
    for i in xrange(60):
        try:
            self.assertTrue(assertion())
            return
        except Exception, e:
            pass
        sleep(1)
    self.fail(msg)

In this case, wait_for_text_present will repeatedly ask the Selenium server whether the text is present until it shows up. It could do several things in a loop, like click a button and then ask whether a popup has appeared. Almost all the asserts in our internal Selenium tests are a wrap around a call to spin_assert, and they’re all way more reliable than they were before the switch.