It's been a while since we’ve posted in detail about our automation and testing process here on the Mobile Team at Gilt. Much has changed in the past few months, and the past couple of weeks in particular have been pretty exciting on the testing front. First off, we had the opportunity to give a talk about how we approach native mobile testing, to a packed house at the Brooklyn iOS Meetup. Second, Appium 1.0 - now already 1.1 - was officially released! Cue the champagne and fireworks! In seriousness though, the Appium milestone release is a pretty big deal for us.
Some History
We’ve written briefly about Appium before, and how it fits in with our overall process. About a year ago, Appium popped up on our radar here at Gilt after one of our engineers saw a talk about it at SeConf. We were intrigued and did a bit of experimenting, and after discovering that we’d be able to leverage some of our existing Selenium infrastructure, as well as our in house Selenium talent, we deciding to adopt Appium as our testing framework of choice.
Appium had been in development for some time by this point, but it was still early days for the project, and it had some rough edges. Tag names were inconsistent, and the XPath locator strategy was pretty sad. We’ll get into that later...
Appium's pace of development was really fast, and things changed regularly. Often the changes would be drastic and would break our tests. Stability also seemed to fluctuate. On some versions, things were terrific, and our tests would run smoothly. Other times, tests would fail or crash for no apparent reason. The failures weren’t always Appium’s fault, but they were still pretty frustrating. Once or twice we wondered why we were even bothering and considered switching back to KIF. We persisted though, and we’re really happy we did, because Appium has grown up!
So, What’s All The Fuss About?
Appium now implements the defined part of the Mobile JSON Wire Protocol. This is basically an extension to the selenium wire protocol, that provides support for mobile specific actions. Desired capability keys seem to have been finalized by this spec, which is great, since these have changed a couple times over the past year, causing a couple of headaches. It’s also good to see some bigger companies getting involved here.
For us though, the toughest issue has been stability. Countless hairs have been turned grey due to tests failing or crashing for no apparent reason. While we've jumped through a variety of other hoops in our quest to solve the overall flakiness issue (page objects, judicious waiting), Appium has made huge strides in this area. This is no small achievement, given that Appium is really proxying commands to UIAutomation, which itself is pretty fragile and prone to random crashing.
We're also super happy to see that force-quit-instruments no longer shuts test sessions down during long waits or element searches. Most surprisingly, page source is now XML based. XML? It’s like it’s 1999 all over again! Actually this change has some pretty amazing benefits. The biggest of which is that we can now perform "real" XPath queries on our page source, and Appium is now using a compliant XPath parser. Hooray! We complained before about Appium's XPath engine being pretty lousy. In olden times, the XPath engine was regex based and would search for the desired element(s) in a JSON representation of the page source. Dreadful stuff... Needless to say, this was inefficient and only worked correctly with "just right” XPath queries. This basically left Appium users with only two legitimate locator strategies. We’re super happy to see that this has been resolved.
Too Good To Be True?
We’re generally really happy with the way Appium works, but there are a handful of issues that still cause some pain for us. Error reporting is not Appium’s strong suit. While overall reliability is orders of magnitude better in the latest release, it’s still incredibly hard to identify the exact problem when failures occur. Integrating Appium with your Xcode workflow is left up to you, and while this isn’t terribly hard, it would be nice to see tighter Xcode integration. I still stand by our previous advice of steering clear of any non-standard UI tricks and hacks. Minimize the use of overlays, and do whatever you can to avoid passthrough views. If anything, Appium will push you to strive for UI simplicity and to get serious about accessibility. The proposed touch gestures API might provide some flexibility here, but we’ll have to wait and see.
Coming Soon
We’re still busy getting to grips with the changes brought by this milestone release. The increased interest in the the Mobile JSON Wire Protocol by some bigger companies has pushed our confidence in Appium up a couple notches. While the spec is not yet complete, it does propose some really exciting features down the line. Notably, we’re really excited by the prospect of server-side waits. The way we manage waiting for elements has had a huge impact on the reliability and speed of our tests, and so it’s really exciting to see this part of the process getting some love.
The Bottom Line
Almost all reliability issues that plagued us in earlier versions appear to have been resolved. We’re now able to spend more time working on tests, and much less time wrestling with Appium and Instruments. Prior to this release, our tests were totally untrustworthy. The changes made leading up to the 1.0 release, combined with better test practices on our part, have dramatically boosted our confidence in our automated tests. Test results are now taken seriously and are becoming an invaluable aid when performing QA work on our apps. We’re really excited to see how the Mobile JSON Wire Protocol evolves, and what comes next for Appium.
By Matt Isaccs of Gilt Groupe for Sauce Labs
