Hey, Selenium. Hurry The Hell Up.

Posted Feb 11th, 2011

I love automated browser tests, but I *really* love fast builds. The faster the better. So when a build that I'm waiting for starts taking over 5-10 minutes, all I can think to myself is... "Hurry the hell up!" And we all go through it, even here at Sauce. So here are a few things we've found to really help speed up Selenium testing:

1) Parallelism

The more tests you can run in parallel, the faster your build can finish. So when writing your tests, always ask yourself, "Can I run this test by itself and have it pass?" If you think to yourself, "No... another test has to run *before* for this one to pass," then stop. Put the shovel down, and start climbing your way out of that hole. Tests that rely on previous tests cause random, weird bugs to pop up, are hard to run when you want to reproduce those bugs, and are just slow. So keep your tests independent. It's how they were meant to be anyway.

When none of your tests are inter-dependent, that means they can all be run in parallel, and in theory, your entire build should only take as long as your single longest test (in practice, it's a bit longer, but not too much). Which brings us to the next point...

2) Short Tests

Write your tests to only test a single tiny bit of functionality. Keep the number of steps in your tests under 20. All the parallelism in the world won't save you from long-running tests, because a single test can't be parallelized. The best written tests we've seen generally run in under 10 seconds. Yeah, I know. That's short. If you're wondering how that's possible, check out step #3, below.

A pleasant side-effect of this is that when a build fails, you'll be able to narrow it down to a very specific area point and fix it all the faster.

3) Pre-populate the cookies

This is a really nice trick.

Every browser you get from Sauce Labs is in a clean state. This is great, and you really want this. Even if Fred says you don't want it, you want it. Tell Fred, "Fred, last night I spent an hour debugging tests, and found out nothing was wrong. You know what the problem was? Someone had installed the fuzzy_warm_critters_asciis browser extension, and it replaced every instance of my javascript with an ASCII cat. I want clean browsers every time." Do it. Fred will understand.

But this poses a bit of a problem with shorter builds, because now each browser session has to go through a login process, and possibly a few more steps depending on your application logic, just to get to where you want to test. Previously, I simply recommended abstracting the login process into a separate method so that the tests looked cleaner, but when you're waiting for the build to finish, that's not good enough. You want them clean *and* fast. One way to get this is to set the cookie via Selenium after the browser starts up.

Log in to your user via server-side logic (e.g., generate a row in your sessions table, or however you have it setup locally), generate a cookie for that login session, and push it into the browser via Selenium. Bam. You're done logging in, and your test can continue on as though it had already spent the 5 seconds logging in and the 10 seconds adding items to the shopping cart, all so you can just test the "confirm purchase" button works on your site as expected in IE7.

This isn't to say that you shouldn't test out your login form or your "add fuzzy critter to cart" logic via browser tests. Because you should. But you shouldn't do it for every test. Test it once, test it good, and leave it at that.

This works on anything that relies on state to be built up. Logins, Shopping Carts, Playlists, etc. You get a clean browser for every tests, and your tests are speedy. And you know what? Even Fred'll love it. And that - in and of itself - is worth it.

Got any tips on how you speed up your Selenium builds? Take us to school and share 'em. We're all ears! Oh, and for Fred's sake...

|\      _,,,---,,_
 /,`.-'`'    -.  ;-;;,_
 |,4-  ) )-,_..;\ (  `'-'
 '---''(_/--'  `-'\_)


The Sauce Labs cloud-based testing platform allows developers to automatically or interactively test mobile and web applications on more than 750 browsers and platforms including iOS, Android & Mac OS X. With built-in video recording of every test, debugging tools, and secure tunneling for local or firewalled testing, Sauce makes running, debugging and scaling test suites easier than ever. Sauce supports automated functional testing with Selenium and Javascript unit testing in all the popular programming languages, frameworks and CI systems.

Written by

The Sauce Labs Team