We run a lot of Selenium testing on Sauce. And with those tests, we see a lot of failures. Sometimes failures aren’t due to bad code, but to the actual browser crashing. Thankfully, when this happens while running a test, we have measures built into our system that reboot the browser and rerun your test for you. This happens without you even knowing, and helps to make your testing life easier.
But we still keep records of every time a browser crashes on our service. We've done this for all the tests ever run on Sauce, which gives us quite a lot of data about browser reliability when it interacts with Selenium. Back in 2011, we took a look at that data to find out which browsers had the highest error rates.
Two years later, with even more data (and more browser versions), we decided to take a look at the information again to find out: Which browser has the highest error rate?
Error rate by browser
We queried for all tests ever run on Sauce - 55 million and counting - and looked at the error rate by browser. The results were, well, not surprising.
Yes, it turns out Internet Explorer performs the worst among our users' tests, with a .25% error rate. But there’s more to the whole story here. This graph shows the error rate of all browser versions grouped together, but each browser has many different versions with varying error rates. So we broke down the relative error rates by browser version.
Error rate by browser version
Below, you can see the browser versions with the highest error rates that have at least 1,000 tests run on them in our service.
When we look at error rates by browser version, IE 6 & IE 7 are the clear winners (well - losers), with error rates of .31% and .29% respectively. Coming in behind IE 6 & 7 is Chrome 5 followed by Firefox 3.5. Only one later version of Chrome version appears in the graph, Chrome 19. Most versions of Chrome have error rates low enough that they didn’t show up on the graph. Neither do later versions of Firefox.
But all of the browser are improving with time, so we then decided to take a look at the error rates for the latest versions of all the browsers that we had data from at least 1,000 jobs on.
As you can see, for recent browser versions, Safari 6 has the highest error rate in our service at .12%, which is less than half of IE 6’s .31% error rate. Chrome 27 and Firefox 22 have virtually nonexistent error rates, with IE and Opera falling in the middle.
Opera has a surprisingly high error rate compared with other recent browser versions. And while Safari 5 has a .16% error rate, Safari 6 has fallen to a .12% error rate, which overall isn’t horrible.
Half of the browser versions we analyzed had error rates lower than .07%. That’s pretty low, and suggests that browsers are getting more reliable as more versions come out.
IE - Getting Better All the Time
Finally, we looked at a graph of error rates of different versions of Internet Explorer, and found an interesting trend.
Error rates for IE have been getting significantly lower with each new version. Microsoft has been so good at squashing their historically high error rates that IE 10 has an impressive 0.05% error rate.
Microsoft has really been pushing their new version of Internet Explorer as the most modern and high-performing version of IE. You’ve may have seen their nostalgia-tinged IE 10 commercial. The data we have on browser error rates suggests that their claims may have quite a bit of merit.
Though browser usage share is an imperfect science, indicators would suggest that Chrome is giving IE a run for its money. But will Internet Explorer’s changes and improved performance give them what it takes to climb back to the top? Guess we’ll have to wait and find out."
