This post was first published on February 5, 2018 and has been updated.
Appium, XCUITest, Espresso, and more
A huge number of mobile testing tools have been developed in recent years to support mobile development. As more companies are developing mobile products and the marketplace is seeing more devices, platforms, and versions, testing your mobile apps is vital. When it comes to choosing what mobile testing tool is right for you, there is a huge array of options, each with different strengths and weaknesses.
As a big supporter of open source technology, Sauce is committed to furthering the open source cause, and we believe there are a number of great open source mobile testing tool that should be considered when devising a mobile testing methodology. There are also the official offerings from the mobile vendors, which are not necessarily open source but have their own advantages. All these tool have different ways of tackling mobile test automation, and the right solution for your project will ultimately come down to which framework fits your needs best. We’ll discuss some aspects of some of the different mobile automation tools (open source or not), as well as why we chose Appium as the main mobile automation framework supported on Sauce.
Why Should You Do Mobile Automation in the First Place?
It’s clear that anyone making apps should test them. The need for digital confidence, to know your app looks, functions, and performs as intended every time, is especially important now more than ever. Failure to catch bugs or regressions can cost companies thousands of dollars a day, and releasing broken apps can frustrate and alienate your users. The cost of a bug in a mobile app can be much higher than a similar bug on a web app. Not only could a bug immediately lead to poor reviews and thus a lowered install rate, but on many mobile app marketplaces, getting a fixed version out could take days or even weeks. But why should you automate your mobile app testing? The reasons for doing automated mobile testing are arguably the same as for traditional web development.
Speed of Testing
While manual testing can be useful, it is a slow and human resource-heavy process. By automating your testing, you can run a suite of tests that would take a manual tester hours to complete in minutes or seconds. Speeding up your testing can allow you to expand your test coverage so you can be more confident that you’re releasing bug-free code.
Scope of Testing
Not only is manual testing slow, but there is a limit to how many user flows or scenarios can be tested realistically. Part of the insight of the Agile software development methodology is that testing should be done as close to code changes as possible. Ideally, this means that your app should be tested on every single commit, not least because a failing test can be tied to a small unit of code change, rather than potentially being related to dozens or hundreds of changesets that went out with a release. Also ideally, all tests should be run, including those for parts of the app which haven’t been touched by the code change. This is because in complex applications it is often not possible to know beforehand what the interaction effects of a code change will be, and the best way to be sure there is no extraneous bug introduced is to run through the entire suite of tests.
Needless to say, manually testing every possible case on every single commit is impossible, especially when you consider the number of platforms to be tested. Leveraging a mobile automation framework is at least part of the answer to all of these problems. Even with automation, however, scaling up your testing can be difficult. End-to-end tests are expensive and, to ensure the test-suite completes in a reasonable amount of time, you’ll need to run them at a high concurrency. If you choose a cloud testing platform like Sauce Labs, you can scale instantly without needing to buy or install more devices or purchase more VM licenses.
Cost of Testing
Automated testing can save you time and money, since you can spend fewer resources on manual testing. Add to that the cost savings of using Sauce Labs Continuous Testing Cloud instead of maintaining in-house infrastructure, and automated mobile testing can be a cost-effective solution for ensuring you’re releasing bug-free apps.
Mobile Testing Tools Compared
The majority of Sauce Labs users choose Appium, an open source mobile automation tool for testing native, hybrid, and web apps. However, there are many different options for mobile testing, all with different philosophies and approaches. We’ll discuss Appium’s philosophy and why we choose to support it, as well as explore some of the other mobile testing tools available and the differences.
Appium
Appium’s Philosophy
We believe that Appium’s approach to mobile automation makes it a strong mobile testing tool that will continue to improve. With the mobile marketplace fragmented between OS platforms and app types, there is a need for a truly cross-platform mobile automation tool, and Appium aims to be that tool.
Appium’s Philosophy Follows Four Principles:
Test the same app you submit to the marketplace. You shouldn’t need to compile third party apps or frameworks into your app’s code that needs to be stripped from the app before submitting it to the app store. The reason for this is simple - if you test one version of your app and submit another version, there is a risk for bugs you didn’t foresee being released into the marketplace. Therefore, to minimize this risk, we believe it is prudent to test the same app you release.
Write your tests in any framework, using any language. People choose which language to write in for many different reasons, and many even write in multiple languages in a given day. This means that any tool that limits what languages and frameworks it supports will be limiting to those using it. Appium strives to support as many languages and frameworks as possible, to give developers the most flexibility.
Use a standard automation specification and API. Developing standards for specifications and API benefits everyone in the open source ecosystem. There are already existing specifications and standards for web testing tools, and Appium’s philosophy is that mobile automation tools should reuse and adapt these existing standards to determine what the standards for mobile testing tools should be.
Build a large and thriving open-source community effort. For both the health of the tool and the benefit of anyone wanting to test, it’s a good idea to encourage a robust community to support and improve the project. Appium strives to build a vibrant community on the belief that theat community will further the technology for everyone.
It’s because of this approach that Sauce Labs has made the effort to support Appium as a mobile automation tool. We believe that this philosophy translates to a well-thought out vision that drives the tool forward. We also believe that this philosophy sets Appium apart from other open source testing frameworks.
Benefits of Appium
Appium is one of the only options for cross-platform testing. In other words, if you have the same app running on both iOS and Android, you can use Appium to run your tests on both platforms with minimal or no code changes. This allows significant reuse of code.
As we mentioned above, Appium allows you to use any programming language for your tests, and can be mixed into any custom test framework you like. Appium gives you more freedom in how to set up your testing than any alternative project.
Appium provides a stable API on top of a crazy automation space. The underlying vendor-provided automation tools change their API on what seems like an annual basis. Appium covers these differences and gives you the same WebDriver-based interface as the years go on, making maintaining your tests easy as Appium upgrades the underlying technology.
If you care about hybrid or web testing, Appium provides the most natural vocabulary for switching between native app content and web content, since it leverages the WebDriver protocol for both.
Sauce Labs is committed to the health of the open source ecosystem, and we believe Appium is a great option for those looking for a mobile testing tool. Sign up for a free trial to start testing.
XCUITest
XCUITest is the official UI testing tool from Apple. XCUITests are bundled with your application and run via Xcode. XCUITest is only for iOS testing, and tests must be written in Swift or Objective-C. Some benefits are:
Stay within the Xcode environment
Use Swift or Objective-C alongside your app code
Take advantage of Apple’s services around XCUITest
Can be used on simulators and real devices
Espresso
Espresso is the official UI testing tool from Google (superseding both UiAutomator and UiAutomator2). Espresso has a completely different architecture, where UI tests live alongside the app code, and can even have access to shared code. Espresso is only for testing Android apps, and Espresso tests must be written in Java or Kotlin. Some of the benefits of Espresso are:
Speed: Espresso was designed from the ground up for speed of automation
Reliability: Espresso has built-in support for ensuring automation does not continue until a view is in an idle state. This avoids a common class of automation bugs where the test tries to continue to the next step but the application is not yet ready
Stay within the Android Studio environment
Take advantage of Google’s services around Espresso
It can be used on emulators and real devices
Earl Grey
Earl Grey is Google’s answer to XCUITest for testing iOS apps, and is an iOS UI Automation framework which also happens to be open source! (However, using Earl Grey does report anonymous statistics back to Google). Earl Grey is for iOS only, and tests must be written in Objective-C or Swift. The main benefit of Earl Grey is that it brings Espresso’s synchronization features to iOS testing, to once again ensure that the automation is not trying to do something in the app while the app is busy.
Detox
Detox is a JavaScript-based test framework for React Native Applications only which, like Appium, aims to be cross-platform. It is meant to be used as a Node.js library in your test code, so tests must be written in JavaScript. Detox is also fully open source. Some of its benefits are:
Write cross-platform tests
Like Espresso and Earl Grey, Detox tries to ensure your app is in an idle state before allowing automation to continue
Underlying technology is precisely Espresso and Earl Grey, so it builds off of Google’s good work
Designed to work with emulators and simulators and does not support running tests on real iOS devices and partially supports running tests on Android real devices
Choosing a Mobile Testing Tool
Given the above list of options, it can be overwhelming to know where to start. At the end of the day, all of these technologies allow the work of automated testing to proceed. There are too many specific pros and cons of each tool to list them here. In general, a good choice for introducing a new mobile testing tool involves asking questions about how a team functions and who writes the tests, more than simply looking at a technical feature list. For example, if your testing is organized by a QA team that likes to write Ruby code, then Appium will be the best choice. Similarly, if your needs are for testing a cross-platform hybrid app, Appium is the obvious winner. If, on the other hand, you are testing a single-platform native app on Android, and your developers are writing the code, it may be wise to explore Espresso, because of the tight integration made possible between the test and app code. At Sauce Labs, we support Appium’s vision of one automation protocol for all platforms, but recognize that in the real world, different tools are appropriate for different scenarios, that is why our Continuous Testing Cloud supports Appium, Espresso and XCUITest.
Sauce Labs - Scalable Mobile Testing
Sauce Labs makes automated mobile testing awesome. Our cloud-based testing platform helps developers test native, hybrid and mobile web applications across the latest and most relevant browser / OS platforms, including iOS, Android & Mac OS. Sauce supports Selenium, Appium and popular JavaScript testing frameworks, and integrates with all of the top programming languages, test frameworks and CI systems. With built-in video recording and screenshots of every test case, debugging tools, and secure tunneling for local or firewalled testing, Sauce makes running, debugging and scaling test suites quick and easy.
