Android Testing

Posted Feb 7, 2018

Making sense of Android app testing - automated/manual, testing in-house/on the cloud, Android Selenium, unit testing, CI integration.

Breaking Down Testing Apps on Android Devices

Android testing is a field that has seen a lot of change over the years. Mobile testing in general is still new, and many users are exploring their options and the available tools, so they can set themselves on the right path for their testing efforts. On this page we’ll help you make sense of the Android app testing in general, and explain the Sauce Labs offering within this bigger picture. (Note - we do not cover beta testing or crowd testing of Android apps, only testing that is initiated by you, the app creator or QA specialist, whether manually or in an automated fashion)

Just so you know what to expect, our offering includes Espresso and Appium, an open source mobile test automation framework - as well as cloud-based services for running tests on a larger scale. There’s a limited free plan, and beyond that, you pay a subscription price based on the number of minutes of tests you intend to run each month or the number of simultaneous test sessions you utilize.

If you have not yet heard of Sauce Labs, a few words about us: Sauce’s cloud testing platform has to date run more than 1 billion tests for a variety of organizations from the finance, banking, retail and media verticals and thousands more -- organizations that are serious about testing, which are working with us to make it awesome. This same platform supports not only web browser testing but also supports Android for both automated and manual testing.

What Do You Mean When You Say “Android Testing”?

There are a few important general concepts in the testing world which apply to Android testing (technically Android mobile app testing) as much as any other kind of app testing.

Let’s first take a look at these so that we can see how Android app testing fits into the overall scheme.

  • Manual testing - this is testing done by hand, by a human being. In other words, a person fires up an application and walks through its various flows and experiences, looking out for bugs or other issues along the way. Learn more about live or manual testing.
  • Automated testing - at a certain point, manual testing becomes overly slow, expensive, and tedious. The pressure in today’s app marketplace is very strong to ship app updates frequently, and at the same time to ensure they are high quality. The problem is bigger than just making sure one particular change is OK before shipping it. For apps which might have hundreds or even thousands of developers making code changes every day, how do you ensure that none of these changes inadvertently interact in a negative way, creating a bug? The only way to keep quality at the same pace as development is to automate testing of the app. Automated tests are simply computer programs whose sole purpose is to test the actual app you’re shipping to your users. There are several general kinds of automated tests, and a host of implementation frameworks and options for getting going with automation. But first, let’s look at the two main kinds of automated test:
    • End-to-end / functional tests: these are tests which mirror the way manual tests work, by taking the perspective of a user and robotically negotiating the app’s various user pathways. In other words, they operate on the UI of the application. This means they exercise the full stack of the application, from the UI to the front-end business logic to any kind of server interaction. For this reason they are very powerful. They are also, relatively speaking, a bit slow. Like lightning compared to human testers, but still slow when you’re talking about running hundreds or thousands of tests on every application code change. They’re best reserved for ensuring the most crucial experiences for your users are solidly tested.
    • Unit tests: these are tests which exercise small, discrete bits of application logic. They operate on the level of app functions, not user experiences. For example, if an internal app function is responsible for parsing a date string into a number, then you would have a set of unit tests whose job is to ensure that this function provides the right output given the right variety of inputs, especially including edge cases. Because unit tests have access to app code and can directly target the smallest bits of logic in your application, they are blazingly fast. Hundreds or thousands of unit tests can be run in the blink of an eye. Unit tests do not test interactions between app components, however, and thus they should not be relied on to the exclusion of end-to-end tests. Details about automated Android app testing.
  • Testing frameworks: whether you’re writing unit tests or end-to-end tests, you won’t want to start from scratch. All kinds of test frameworks exist that enable the setting up of test environments or the elimination of boilerplate code. This is where it’s important to research which frameworks are popular and supported, so you can be confident in your choice of testing tools. Sauce Labs supports Appium, Espresso and XCUITest for Android app testing.
  • Continuous integration: one of the huge benefits of automated testing is that it enables automation of the entire application delivery process. Continuous Integration (CI) is the idea that new code should be constantly tested and integrated into the application when it is known to have passed the tests. Automated testing means that new code can be validated extremely quickly, and little bits of functionality or small bug fixes can make their way to users on their own, without necessarily having to wait for a huge release. In general, this is a good practice because if something goes wrong, it’s easier to undo a small change, not to mention easier to track down what specifically went wrong in the first place. There are a variety of CI systems like Jenkins or Travis, which facilitate the pulling of new code, building it, and exercising it with the various types of tests we discussed. More information on Sauce Labs integrations with CI servers.

Now let’s take a look at how these genres of testing apply specifically in the Android world:

  • Manual testing - this is done by running the app on an Android emulator on your own computer, or installing the app on a real Android device connected via USB. There are service providers, including Sauce Labs, who can help you do manual testing on many different Android configurations without having to set up a device lab yourself. This is especially convenient when the desired test platform or device is not one you have available locally.
  • Test automation - you can write both unit and end-to-end tests for Android applications.
    • End-to-end / system tests - again, this is a full system test that simulates a real user, and checks how all the system’s components work together. This involves launching the app on an emulator or real device, automating the UI to perform certain user actions, and checking the result. For example, a test can be run that logs into an app and tries to upgrade a user account from “free" to “paid”: a highly sensitive operation that needs to be tested end-to-end! Another aspect of end-to-end tests for Android is that tests can run differently on different Android configurations (e.g. on a tablet vs. an Android phone), so you need to run each test several times on each configuration you want to support. This is of course to ensure that your app itself works great across all the configurations you care about.
      • Running end-to-end tests yourself - you can use Android emulators or your own real devices to run scripted end-to-end tests. However, the number of tests you can run, and the number of Android configurations you can test, will be limited by the number of computers and Android devices you physically own. You need to install and manage the infrastructure yourself.
      • Running end-to-end tests in the cloud - there are several providers, including Sauce Labs, that allow you to supply your app and a test script, and run end-to-end tests on the provider’s servers in the cloud. The cloud provider has many more servers than any one tester can set up and has hundreds of combinations of Android emulators and real mobile devices covering different browsers, OS versions and configurations, so you can improve your coverage and run more tests more quickly. But all this fun comes at a price - you’ll need to pay the provider, usually based on the number and complexity of the tests you perform and the type of device you test on. Typically, Android emulators cost less to use than real mobile devices because mobile devices are more expensive to configure and maintain in a cloud.
    • Unit testing - Android has two ways of running unit tests, so-called “local” tests and so-called “instrumented” tests. Local tests run on your local system against a mocked version of the Android OS, so that any calls to the Android system do not actually do anything (otherwise your unit test would require a real phone or emulator and be much slower). Instrumented tests, on the other hand, have access to a richer environment of Android functionality, which might be necessary to test some of your app methods. So you should be able to run a large number of tests on your local machine. To get started with Android unit testing, see the official Android tutorials on creating unit tests and running tests on an emulator.
  • Testing frameworks - to run automated tests, whether unit tests or full-blown system tests, you will need a testing framework. For unit tests, Android comes with all the batteries included, and nothing much needs to be said. For functional tests, the landscape is much more interesting. Google itself provides a number of different UI testing frameworks: UiAutomator, UiAutomator2, and Espresso. The most popular and modern tool from Google is certainly Espresso, which takes advantage of tests being bundled with app code to find ways to make the tests robust and fast. There are a number of other options, however, each with its own benefit. The most popular alternative is certainly Appium, a completely open source project backed by Sauce Labs and others, with the goal of bringing WebDriver compatibility to mobile automation. If you want to write tests in a language other than Java, or have experience with Selenium, or have need to test apps across both the Android and iOS platforms, then Appium is going to be the strongest candidate.
  • Continuous integration - Some of the Android testing frameworks, including Appium, allow you to hook up to a CI system and run automated Android tests as part of your agile development process. But you should be aware that mobile CI is still in its early days and does not always work as smoothly or as quickly as CI for web applications.

Sample Java Test Script for Android App Testing

To get you started, here is an example of a sample Java test script for Android testing using Appium. Complete details and additional scripts can be found here.

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.remote.DesiredCapabilities;
public class SampleSauceCheckBoxTest {
  public static final String USERNAME = "YOUR_USERNAME";
  public static final String ACCESS_KEY = "YOUR_ACCESS_KEY";
  public static final String URL = "https://" + USERNAME + ":" + ACCESS_KEY + "";
  public static void main(String[] args) throws Exception {
    DesiredCapabilities capabilities = new DesiredCapabilities();
    capabilities.setCapability("deviceName""Samsung Galaxy S4 Emulator");
    WebDriver driver = new AndroidDriver<>(new URL(URL), capabilities);
     * Test Actions here...

Interested in diving in and trying Sauce Labs for Android app testing? Sign up for our Free Trial!

Written by

Sauce Labs


Mobile testingApp testingCI/CDFrameworksManual/Live testingAutomated testing