Today's guest post comes from Matthew Edwards, who wrote a bit about how he uses Sauce and Appium to run his mobile tests in the cloud. Matthew leads the mobile automation team at Aquent. If you're interested in automated testing, they're hiring.
Google presented at GTAC 2013 how they're "catching 99% of all bugs" using rooted x86 emulator images with hardware acceleration. The emulators are run in a datacenter and they've completed "82 million Android tests in March." A person in the audience asked, how can I use this? The response was if you're at Google then you can use it. If you're not "well, that sucks."
I wanted a solution I could use that didn't involve working for Google. Almost all the Google testing tools are open source with the exception of Espresso. In addition, I test on iOS and I'd like to apply the same methodology of testing in data centers to that platform. Each test can run on both platforms by using Appium due to the standardised WebDriver protocol. Jonathan demonstrated this in his GTAC 2013 presentation using the app I work on (Woven). There are three key areas of the automation strategy.
First, having a properly configured emulator and test runner. This is important for capturing all logging information and measuring flakiness. Google was able to achieve 0.15% flaky tests. In my testing with Calabash Android, I encountered the same flakiness issue Google mentioned in the presentation about Robotium. Fortunately Appium allows using uiautomator which is Google's newest testing technology and it's worked well.
Second is running in the cloud at scale. I've maintained a small internal physical device cloud and emulator cloud. It's not fun. I'd much rather pay a service provider to take care of scaling for me. Sauce Labs offers Android emulators and iOS simulators as a service. I have been running Woven iOS and Android tests on Sauce since it was announced. It's the most effective service I've used by far. Most providers focus exclusively on physical devices because they can charge outrageous amounts of money. It's also the worst way to test. As Google says "We're not saying there is no place for device-specific testing at all. There is still a place for it. But, you know, you need to do it when you have done everything else."
The third is testing technology. If Google releases Espresso as open source, then I'll be adding support to Appium. The existing instrumentation based testing technologies have significant issues with flakiness as mentioned in the presentation. Standardising on the WebDriver protocol enables swapping out the underlying testing technology without having to rewrite tests. This enables Appium to use best of breed testing technology from Apple (UI Automation) and Google (UI Automator) while retaining the flexibility to add new backends in the future. For testing Woven, I use Ruby on OS X with the gems appium_lib and appium_console. The appium_lib tests for iOS and Android are very similar to the production tests I write for Woven. First, I’ll open up a console using arc to interactively write a test. Then I’ll run the test locally using Rake. Next, I’ll commit the change to GitHub. Finally, Jenkins will build the latest version of the application from source, run the tests on Sauce Labs, and email me if there are any problems.
I hope you’ll watch the GTAC presentations, try out Appium, and start using emulators and simulators in your mobile testing strategy.
