How to Lose Races and Win at Selenium

Selenium tests sometime fail for no reason. You can eliminate most of their random failures with a special kind of assertion, the spin assert. The problem is that Selenium is overeager, and it needs to chill out and wait.

Selenium tests often fail because they're too fast. Where a user might wait for a page to load for a few seconds and then click on a link, Selenium will interact with a page at the speed of code, before the page is ready. The way to fix this is to have Selenium repeat its actions and assertions until they work. If you don't, Selenium races your browser.

Selenium tests pages. It tests them by loading them and then doing stuff on them like it's a person. Isn't that cute? Selenium thinks it's people. Unfortunately for us, Selenium isn't people, and in not being people, it's too fast. Websites are made for people, with the knowledge that people are slow. When humans open a page, the humans wait for it to be loaded, then click on stuff. The clever secret of webpages is that they're not really loaded when they look loaded. That's because the web page's authors know we take a second to take it all in before we start playing. So they make pages that look loaded right away, and then use javascript to do more real loading once the page is 'loaded.' This javascript sets up for playtime in the background while we're taking it all in. Selenium has a bad habit of trying to use pages before they're set up for playtime.

The way to solve that is not telling Selenium to pause for a fixed number of seconds. Pauses have two problems. They make your tests slower than they need to be, and they don't really fix the problem, just make it less frequent. The best solution is to have Selenium keep trying until it works. If your test is trying to assert that clicking on an "email settings" button pops up an email settings dialog, Selenium should repeat a "click the button -> is the status dialog visible?" loop until the status dialog is visible.

Here's the problem. When we tell Selenium to wait-for-page-to-load and then click, Selenium waits for the wrong thing. It can tell when all the page's content has been downloaded and displayed, but it has no idea when the page's custom javascript has finished loading. Sometimes that functional javascript goes back to the server and fetches more content to be displayed, so you can't even count on required text being present yet. There are several ways for a page to be "loaded," and Selenium doesn't wait for the right one. It races your page's loading logic, often "winning" the race and clicking on things before they're ready to be clicked on. You need Selenium to lose the race.

The solution is spin asserts. At Sauce Labs, when we want to verify that some text appears on a page, we use this function:

def wait_for_text_present(self, text, msg=None): msg = msg or " waiting for text %s to appear" % text assertion = lambda: self.selenium.is_text_present(text) self.spin_assert(msg, assertion)

Do you see it calling a function named spin_assert? That function is the key. spin_assert retries the passed-in test function, in this case a lambda expression, over and over until it works. Here's what spin_assert looks like, minus some bells and whistles:

def spin_assert(self, msg, assertion): for i in xrange(60): try: self.assertTrue(assertion()) return except Exception, e: pass sleep(1) self.fail(msg)

In this case, wait_for_text_present will repeatedly ask the Selenium server whether the text is present until it shows up. It could do several things in a loop, like click a button and then ask whether a popup has appeared. Almost all the asserts in our internal Selenium testing are a wrap around a call to spin_assert, and they're all way more reliable than they were before the switch.