Automated browser tests are wonderful, almost as wonderful as fast-running build systems. Once we add the two together to create continuous integration, they seem to be at odd. We all go through it, even here at Sauce. So here are a few things we've found to really help speed up Selenium testing:
1) Parallelism
If all tests run in less than ten minutes individually, and it is possible to put each test on a different machine, then the total test run should be ten minutes or less … right?
Sadly, things don’t happen that way often enough. Generally, that is because of the design. hen writing your tests, always ask yourself, "Can I run this test by itself and have it pass?" If you think to yourself, "No... another test has to run *before* for this one to pass," then stop. Put the shovel down, and start climbing your way out of that hole. Tests that rely on previous tests cause intermittent issues, are hard to run when you want to reproduce those bugs, and are just slow. So keep your tests independent. It's how they were meant to be anyway.
When none of your tests are interdependent, that means they can all be run in parallel, with the entire test run taking as long as the longest test. Which brings us to our next point.
2) Short Tests
Write your tests to only test a single tiny bit of functionality. As a rule of thumb, keep the number of steps in your tests under 20. All the parallelism in the world won't save you from long-running tests, because a single test can't be parallelized. As another rule of thumb, e the best-written tests we've seen generally run in under 10 seconds.
Yes, that is short. If you're wondering how that's possible, check out step #3, below.
A pleasant side-effect of this is that when a build fails, you'll be able to narrow it down to a very specific area point and fix it all the faster.
3) Pre-populate the cookies
Every browser you get from Sauce Labs is in a clean state. This is great, and you really want this. Even if Fred says you don't want it, you want it. Tell Fred, "Fred, last night I spent an hour debugging tests, and found out nothing was wrong. You know what the problem was? Someone had installed the fuzzy_warm_critters_asciis browser extension, and it replaced every instance of my javascript with an ASCII cat. I want clean browsers every time." Do it. Fred will understand.
This poses a bit of a problem with shorter builds, because now each browser session has to go through a login process, and possibly a few more steps depending on your application logic, just to get to where you want to test. Previously, I simply recommended abstracting the login process into a separate method so that the tests looked cleaner, but when you're waiting for the build to finish, that's not good enough. You want them clean *and* fast. One way to get this is to set the cookie via Selenium after the browser starts up.
Log in to your user via server-side logic (e.g., generate a row in your sessions table, or however you have it setup locally), generate a cookie for that login session, and push it into the browser via Selenium. Bam. You're done logging in, and your test can continue on as though it had already spent the 5 seconds logging in and the 10 seconds adding items to the shopping cart, all so you can just test the "confirm purchase" button works on your site as expected in Edge.
This isn't to say that you shouldn't test out your login form or your "add fuzzy critter to cart" logic via browser tests. Because you should. But you shouldn't do it for every test. Test it once, test it good, and leave it at that.
This works on anything that relies on state to be built up. Logins, Shopping Carts, Playlists, etc. You get a clean browser for every tests, and your tests are speedy. And you know what? Even Fred will love it. And that - in and of itself - is worth it.
Got any tips on how you speed up your Selenium builds? Take us to school and share 'em. We're all ears! Oh, and for Fred's sake...
|\ _,,,---,,_ /,`.-'`' -. ;-;;,_ |,4- ) )-,_..;\ ( `'-' '---''(_/--' `-'\_)
SAUCE LABS - SELENIUM AND APPIUM TESTING IN THE CLOUD
The Sauce Labs cloud-based testing platform allows developers to automatically or interactively test mobile and web applications on hundreds of browsers and platforms including iOS, Android & Mac OS X. With built-in video recording of every test, debugging tools, and secure tunneling for local or firewalled testing, Sauce makes running, debugging and scaling test suites easier than ever. Sauce supports automated functional testing with Selenium and Javascript unit testing in all the popular programming languages, frameworks and CI systems.
