Last month, we hosted a webinar about Automated Mobile Testing at Gilt. Matt Isaacs, software developer at Gilt, told us all about how Gilt Groupe uses Appium to automated their mobile testing. He discussed previous mobile testing solutions, issues his team encountered and solutions they found. You can find the recording of the webinar here, and the slides below. We also have a Q&A of questions from the webinar below. http://www.slideshare.net/saucelabs/abp-mi
Q&A
Q: What are some of your bad experiences/bugs that drove you to automated testing?
Nothing really stands out over the time that I’ve been with Gilt. There are two big issues that are motivating our push for automation: First off, our Release + QA process has some serious flaws which we’re trying to deal with before they result in the kind of nightmare experiences that you’re asking about. Second, the bugs that cause the most pain for us are of the elusive and hard to reproduce variety. These are the sorts of issues that are most effectively served by manual QA, but don’t get enough attention because too much of our manual QA time is spent on sanity tests and fix verifications.
Q: Do you have thoughts on a working compromise between automating (and maintaining) UI tests and performing them manually?
The obvious candidate for automation will be your daily, repetitive QA tasks. For us, it’s a set of sanity tests that ensure that users are able to log in or register a new account, browse items, add them to the cart or waitlist, and check out successfully. These need to be done whenever we test a build to ensure that development work hasn’t broken any existing, crucial functionality. Automating the Steps To Reproduce for persistent/recurrent bugs can be worthwhile too, but the emphasis here is on recurrent. I wouldn’t really advise going too far beyond this just yet. Some may disagree here, but following a standard TDD approach of writing UI tests before implementing UI features is probably going to turn into a gigantic time investment. It really depends on what kind of user interactions are involved in navigating your app, as well as the app's state of accessibility. This isn’t about 100% automated test coverage, but rather about maximizing the effectiveness of our manual QA time. Q: does your team work together with the QA team in order to assure that all elements can be easily accessible with Appium? For now, no. The reason being that our QA team is not yet responsible for writing automated tests.
Q: Matt, how do you deal with "waits" to transfer between screens?
We’re still experimenting with ways to handle waits reliably. The Page Objects pattern has really helped us out here. During Page Object creation, you’ll typically do checks to confirm that the desired page is loaded and visible. I think its best to handle waits here. We do something similar to to ScalaTest’s Eventually (http://doc.scalatest.org/2.0/#org.scalatest.concurrent.Eventually), but you could also increase your driver’s implicit wait timeout at this point, perform your page checks, and then set it back to its original value.
Thanks to Matt & Gilt Groupe for lots of great info. Stay tuned for more great webinars.
