Automated browser tests are wonderful, almost as wonderful as fast-running build systems. Once we add the two together to create continuous integration, they seem to be at odd. We all go through it, even here at Sauce. So here are a few things we've found to really help speed up Selenium testing:
If all tests run in less than ten minutes individually, and it is possible to put each test on a different machine, then the total test run should be ten minutes or less … right?
Sadly, things don’t happen that way often enough. Generally, that is because of the design. hen writing your tests, always ask yourself, "Can I run this test by itself and have it pass?" If you think to yourself, "No... another test has to run *before* for this one to pass," then stop. Put the shovel down, and start climbing your way out of that hole. Tests that rely on previous tests cause intermittent issues, are hard to run when you want to reproduce those bugs, and are just slow. So keep your tests independent. It's how they were meant to be anyway.
When none of your tests are interdependent, that means they can all be run in parallel, with the entire test run taking as long as the longest test. Which brings us to our next point.
Write your tests to only test a single tiny bit of functionality. As a rule of thumb, keep the number of steps in your tests under 20. All the parallelism in the world won't save you from long-running tests, because a single test can't be parallelized. As another rule of thumb, e the best-written tests we've seen generally run in under 10 seconds.
Yes, that is short. If you're wondering how that's possible, check out step #3, below.
A pleasant side-effect of this is that when a build fails, you'll be able to narrow it down to a very specific area point and fix it all the faster.
This poses a bit of a problem with shorter builds, because now each browser session has to go through a login process, and possibly a few more steps depending on your application logic, just to get to where you want to test. Previously, I simply recommended abstracting the login process into a separate method so that the tests looked cleaner, but when you're waiting for the build to finish, that's not good enough. You want them clean *and* fast. One way to get this is to set the cookie via Selenium after the browser starts up.
Log in to your user via server-side logic (e.g., generate a row in your sessions table, or however you have it setup locally), generate a cookie for that login session, and push it into the browser via Selenium. Bam. You're done logging in, and your test can continue on as though it had already spent the 5 seconds logging in and the 10 seconds adding items to the shopping cart, all so you can just test the "confirm purchase" button works on your site as expected in Edge.
This isn't to say that you shouldn't test out your login form or your "add fuzzy critter to cart" logic via browser tests. Because you should. But you shouldn't do it for every test. Test it once, test it good, and leave it at that.
This works on anything that relies on state to be built up. Logins, Shopping Carts, Playlists, etc. You get a clean browser for every tests, and your tests are speedy. And you know what? Even Fred will love it. And that - in and of itself - is worth it.
Got any tips on how you speed up your Selenium builds? Take us to school and share 'em. We're all ears! Oh, and for Fred's sake...
|\ _,,,---,,_ /,`.-'`' -. ;-;;,_ |,4- ) )-,_..;\ ( `'-' '---''(_/--' `-'\_)