Firefox the Clear Winner for Automated Testing

As you probably know, Sauce Labs offers a tool for doing automated Selenium testing on arbitrary OS/browser combinations. Inspired by Arstechnica's breakdown of browser popularity among users, here's our data on browser popularity among testers:

Tests by browser Firefox has consistently been the most popular browser among testers. We used to think this was because it's our default offering, but we now know that's not true. Here's why:

IE by version Surprisingly, testers seem not to care about IE6, the most difficult browser to support. Possibly this is because it's also the most difficult browser to test; it's notorious for crashing during tests, creating false positive test failures. Other than that, IE version usage looks the same for testers as it does for users; the old versions never die, in contrast to recent events in Safari testing. Safari by version Safari 5 took off in April 2012. This doesn't reflect an overall increase in use; Safari popularity remained constant throughout this period. It simply cannibalized the use of Safari 4, in a more pronounced version of the Opera usage pattern. Opera by version Opera 12 is also available, but use didn't start until after these numbers were captured. Testers appear to have stopped caring about supporting multiple versions of Opera, and are generally content to test with the latest version only.

Notes

  • Google Chrome isn't broken down by version, because we support only one version at a time
  • Graphs based on 1,000,000 tests, randomly sampled out of 20,000,000 total
  • Firefox 3.6 still accounts for a disproportionate amount of FF tests.  Our getting-started guides all use Firefox 3.6, and that's the default browser-version combination. We thus can't tell the difference between tests that wanted FF3.6 and tests that didn't care what OS/browser they wanted, so you may still think it's unfair to count a FF3.6 job as evidence of Firefox's popularity. So here's what the Firefox and overall graphs look like without FF3.6. Firefox is still the winner in the end, but only passes IE in late 2011 as testers move away from 3.6:

Written by

The Sauce Labs Team

Topics

Cross browserAutomated testing