Musings on Mobile Testing Automation

Posted Nov 1, 2012

We at Sauce Labs love automated testing. We believe test automation is a critical piece of the puzzle for pushing code from "GitHub to Heroku" as fast as possible, while building confidence that everything working yesterday is still working today. This is why I started Sauce Labs and why we invested in building out our "Selenium as a Service" cloud. We started with Selenium because it's the market leading testing tool for desktop web testing. However, for mobile it's a brand new game. It's not enough to just test web apps in mobile browsers. Looking at the success of the app marketplaces, (e.g. Apple App Store and Google Play), it's obvious that native and hybrid app development is popular with developers, too. And there is no consensus on the right tool for testing /anything/ on mobile -- native, hybrid, or web. While the mobile test automation tool market matures, many mobile developers have given up on using automated tests and gone back to manual testing. Developers will come back to test automation when consensus is reached, but consensus building doesn't just happen automagically. This is why we hosted the Mobile Testing Summit -- to get the world's leading open source test automation tool creators in one room -- and see what happens. At the summit, these creators are sharing their experiences of what's worked, what hasn't, and charting a course forward. As we chart that course, I think it's useful to reflect on the secrets of Selenium's success and how they can and do apply to mobile. 10) Be Good at One Thing (And Pick the Right Thing) Born in the age of AJAX and Web 2.0, Selenium was pretty good at testing JavaScript-heavy web applications. But it really excelled at driving Firefox on either OS X or Linux with Java client libraries. This was the "killer combo" that resonated with developers. Firefox was their favorite browser and MacBooks were their favorite development machine. Java probably wasn't their favorite programming language, but it's what most large scale web apps were written in. 9) Make Stone Soup (Start Small, Sell the Dream, and Ask for Help) The Selenium project's mission is to automate all browsers on all operating systems with any programming language. However, such a Big Hairy Audacious Goal is not something one person can do alone. Like the folk story where a village turned a pot and a simple stone into a delicious soup, Selenium needed lots of help from lots of people to get the job done. At the beginning, many browser/OS/language combinations were buggy or incomplete. For many years, Selenium really stunk at automating Safari on Mac. And if you used XPath element locators in Internet Explorer, Selenium was worse than an angry skunk. I'm sorry  to be blunt - but things did get better over time as more people got involved. At first, you don't need a detailed road map, but you do need to tell people what mountain you're climbing. If you start small enough, yet dream big enough, you just might be able to convince strangers to help you on your journey. 8) No Assembly Required Don't require developers to modify their apps in any way just to start using your test library. Developers should be able to test their app "as-is". Originally, Selenium violated this rule. Because of the same origin policy (long story), Selenium's JavaScript and HTML had to be served alongside the rest of a web application's public assets. This was a frequent complaint and impediment to adoption. Once Selenium Remote Control (RC) arrived, we were able to remove the app modification requirement. There are several reasons this is important -- some technical, others cultural. Developers want to keep the surface area of their apps as small as possible. Having extra code lying around in the app leads to an unnecessary increase in file size and another potential vector for bugs or security holes. And some teams charged with test automation are separated from the developers who compiled the app. They have to write tests, but they might not have any (or at least, an easy) way to modify the source just to make it automatable. Yes, that's sad and wrong, but a reality for many organizations. The least you can do is make their lives a little more comfortable. Make it easy to test an app as-is. 7) Be Newbie Friendly Newbie's want some help. Don't leave them hanging. Focus on the "First 15 minutes" experience. Although it's controversial in the testing world, you should consider having a Record & Playback tool for helping new users write their first test. And like a good mentor, encourage them not to abuse the tool. 6) Be Developer Friendly Pay attention to how developers prefer to work. Let developers use their favorite text editors or IDEs. They also like to write their tests in the same language as the app's source code, and check those tests into the same code repository. And don't invent a new programming language. Many of the commercial testing tools that Selenium supplanted broke all these rules, and explains why developers refuse to use them. 5) Scaling Happens Paul Hammant gets the credit here. It took me a long time to realize he was right all along. A few months into the project, Paul wrote the first versions of what later became Selenium RC. He used a technique, (now called "comet" or "long polling") to let any programming language talk to the Selenium automation engine running in the browser. Instead of inventing a new protocol to talk in this new way to a browser, he decided to re-use (some might say abuse) HTTP for the message transport. At the time, even though all Selenium tests were run locally and didn't need to talk over a network, using HTTP ended up having a secondary benefit -- you could drive Selenium tests remotely on different machines (with different OS and browser configurations) from anywhere on the network. Using HTTP also let testing scale up gracefully. It enabled projects like the Selenium Farm, Selenium Grid project, and even Sauce Labs to exist. Functional testing can be slow, but throwing more computers at the problem is a great way to speed things back up again. Through the magic of HTTP proxying -- you can distribute tests to huge farms of machines without having to push that complexity into the tests themselves. So, even though worrying about scaling is usually considered a premature optimization, it's not that premature when it comes to testing. Sometimes, you are going to need it. 4) Good Ideas Come from Everywhere Be welcoming of ideas from other projects. And occasionally, merge and join forces. When Simon Stewart and I spoke at the Google Test Automation Conference in 2007, we were both quite proud of our projects and happy doing our own thing. However, we quickly realized we shared many common goals and attitudes regarding the future of test automation. At the conference we hatched a plan to merge our projects. Simon's WebDriver project became Selenium 2. Selenium had solved 80% of the test automation problem. But WebDriver was really good for that last 20% the frequently tripped up Selenium users -- interacting with the host operating system for things such as keyboard and mouse events, handling pop-up dialog boxes, and security warnings. It's important to be humble and know your project's weaknesses. When someone comes along and fixes all your problems, and they inconveniently didn't solve them in your own codebase, don't get mad. Instead, get them drunk, and trick them into taking over your project. 3) Be Social It's not enough to simply create. You also have to promote your work. Speak at a meetup. Speak at conferences. Write blog posts. Create screencasts. But you don't have to do it alone. I've had the luck and pleasure of working in many environments where everyone was encouraged to publicly share their work as much as possible. I did my part in getting the word out about Selenium in the beginning, but friends and colleagues did a bulk of the work. I couldn't be in every city and every conference talking about Selenium. But my friends could. So, collect lots of friends. If you solve an important problem for them, they'll be more than happy to tell the world about it on your behalf. 2) Be Free (Open Source FTW) Everyone has their own story for why they believe in the power of open source. I became a big believer in open source years ago as a system administrator of a large proprietary HR system. Half way through the implementation, we unfortunately found out that some critical functionality just flat out didn't work. Without access to the source code, and being too small to get the attention of the vendor, we had no ability to fix the issue. We ended up writing our own replacement system -- this was the Time and Expense system that the Selenium project came from. Having access to the source code is especially important in testing -- as SDKs and standards evolve, testing tools are on a treadmill racing to keep up. It doesn't scale to expect just one project or one team to know how to automate every crazy cool new HTML5 or iOS feature at all times. With open source, when your users run into a feature that your testing tool can't handle, they're not stuck. They have the power to fix things and move on. 1) Be Foolish Do the things that others think are crazy. Within reason, of course. If you believe passionately in a particular idea, do it anyway, despite what people say. In 2003, using JavaScript was considered a very unprofessional thing. It was buggy and inconsistent in every browser. But using JavaScript let us create some really cool and fast features that we couldn't do any other way. Then 6 months later Gmail and Google Maps came along and using JavaScript was no longer crazy. Foolish today, genius tomorrow. Paradoxically, if you want people to keep thinking well of you, you have to keep doing things they think are quite foolish. So, stay foolish. To be continued....

Written by

Jason Huggins


Automated testingMobile testing