A common complaint that I hear is: “My tests run a lot faster locally than on Sauce Labs.”
Sauce Labs is in the cloud and not in your local network, so it makes sense that any given test is going to have some amount of slowdown. The advantage of Sauce Labs is that you can make up this difference by scaling up the number of tests you run at the same time. If you are seeing individual tests taking much more than twice as long to run remotely as they do locally, then there are definitely things you can do that will improve your performance.
However, let’s start with an unlikely disclaimer: test speed might not be what you should be prioritizing.
The goal of test automation is to get accurate results back in a consistent manner. If you are not getting accurate results, then the speed at which you get those results is not the important thing. I spoke with a team last year that wanted to decrease their run times from 9 hours to 6 hours. When I sat down to work with them on that, I asked what percent of their tests were failing. They told me 70% (!!?!??). When you have more failures than you can evaluate between test runs, your tests are effectively useless. This team had no business focusing on shaving off time on their test runs when all that will do is give them useless information faster. I’m not saying performance isn’t important, and there are some things that you can do to improve performance that will also improve reliability, but in general, focus on the most important thing first. Essentially, spend your time reducing “maintenance costs” rather than focusing on “execution costs.”
There are a number of improvements that might make a difference—like moving your testing servers closer to the Sauce Labs data center, and turning off video capture and screenshots—which aren’t worth the time savings. You might have errors that are entirely preventable and should never happen that are taking up time that could be spent running other tests. There are some powerful things that you might not know about that you can do with a proxy like Sauce Connect for reducing overall traffic and decreasing the amount of internet that some assets need to travel. Finally, many people don’t realize that their frameworks are likely optimized for local execution and are very inefficient when running against any remote server.
Over the next few weeks, I'll be walking you through some ways that you can speed up your tests, including parallel testing, eliminating time-wasting errors, and more! Stay tuned for future posts that will include code and video with screencasts.
