From 2009 to March 2021, Android grew from a market share on mobile devices of less than 1% to 71%. That means more than two out of every three smartphones is an Android. Unless your company makes software for a very particular market, Android is likely to be the most popular operating system for your customers. This means making native Android applications and web pages that will be viewed on Android, which means testing for those devices. Tablet market share for Android is around 45%; since 2012 Apple tablet share dropped from 85% to 53%.
Source: statcounter
While the phone products are mature, the testing practices are certainly not, with a wide variety of frameworks, languages, tools, and methods to perform development and especially testing. On this page we'll provide a sense of the bigger picture of Android testing, the risks, and challenges that the testing addresses, along with how Sauce Labs offerings help resolve those challenges. Our focus is the testing that is created by the app creator or QA specialist, so the article does not cover beta testing and crowdsourced testing.
To do that, we need to answer two questions: What are the variety of techniques for software testing, and then what do they mean for android?
What does "Android testing" mean?
Manual Testing. The most well-known kind of testing, and often highly valuable, is putting a new build of the application on a physical device and running the software, end-to-end, against real servers. Most of us are familiar with this work, which might be done by programmers, testers, or some form of customer-proxy as User Acceptance Testing, or UAT. Some small organizations, especially developing internal company software to use on the single-model, company-supported Android device, might find that manual testing covers most of their needs. Sadly, for larger projects, teams, and customer bases, manual testing becomes insufficient. Where Apple devices are made by just one company, Google partners with many manufacturers. As a result, Android in particular is a highly fragmented product family, which compatibility testing addresses.
Compatibility testing. Android has a broader variety of operating systems, screen sizes, hardware specifications and backwards compatibility. At this point there are tens of thousands of combinations of Android devices in the wild today. Compatibility testing is the process of taking an application that is done testing on one configuration and testing for correctness on another, slightly different machine. Even if an organization had a large enough device library to sustain such testing, the time involved in performing the testing would be prohibitive. Dan Nosowitz's fragmentation image gives some idea of the marketspace for Android devices -- even the top 12 only cover about 25% of the landscape, and this research is now out of date.
Source: Dan Nosowitz, popsci.com
Add to that that a great deal of testing is done remotely due to COVID-19, and purchasing physical devices by location becomes impossible. One way to perform effective compatibility testing is to use simulators, emulators, or real devices in the cloud, a service that Sauce Labs can provide. Another is to rotate through cloud-based devices while performing test automation.
Automated Testing. Selenium (for mobile web) and Appium (for native Android) and Espresso are three code libraries that treat an application like an object, allowing the programmer to click on objects, get the text on the screen, and so on. Sauce labs can provide the virtual devices to run tests on these devices, no device lab required. These tests typically come in two flavors:
User Interface (end-to-end) tests. These are tests that mirror the way manual tests work, by taking the perspective of a user and robotically negotiating the app’s various user paths. These operate at the Graphic User Interface, GUI or UI level. This means they exercise the full stack of the application, from the UI to the front-end business logic to any kind of server interaction. That means they can find errors at any level of the technology stack. It also means they are relatively slow. A full suite of Android tests might involve two thousand user interface actions. At two seconds per click, that will take over an hour on one device. Sauce labs provides a grid product; an hour across sixteen devices running in parallel is under four minutes. Due to the delays and maintenance effort, the end to end tests are usually reserved for testing the most crucial parts of the user experience. Selenium and Appium run in a dozen different program languages; Espresso is limited to Kotlin or Java.
Unit Tests. Generally written in the same programming language as the production code, these isolate the individual methods of the code to exercise small, discrete bits of application logic. For example, if an internal app function is responsible for converting something the user enters (a string) into an internal-programming-language date string into a number, unit tests would provide examples of sample inputs and the expected result, including the edge case where the string is blank, all spaces, nonsense, and so on. Because unit tests have access to app code and can directly target the smallest bits of logic in your application, they are blazingly fast. Hundreds or thousands of unit tests can run in seconds. Unit tests do not test interactions between app components, however, and thus they should not be relied on to the exclusion of end to end tests. Here are details about automated Android app testing.
Testing Framework. A Selenium, Appium, or Espresso test, or "test case" is essentially a computer program. These individual tests might cover a feature or a specific flow; the collection of these tests creates the "test suite." A framework collects test cases, runs them in order (perhaps in parallel) and creates reports of test results that trace errors back to a specific test and test step. Frameworks provide an important piece of the architecture because they integrate the build system to check the newest build, and also abstract the results. A framework might support a single view of an application's test results for Web and Mobile, for example. Think of frameworks a bit like the old Soviet Russia Joke. Usually, when testing, you call the test case. When using a framework, framework calls you!
Continuous Integration. Once tests are automated they can run on every build immediately after the build, finding problems introduced by that change. Continuous Integration (CI) systems can run on every change, with an awareness of who made the change and what it was, making debugging and fixing (or maintaining the tests) a snap. This means small, isolated changes can be verified against the entire regression suite quickly and rolled out without a large human regression test effort. Then if something does go wrong, it will be easier to undo that small change. There are a variety of CI systems like Jenkins or Travis, which facilitate the pulling of new code, building it, and exercising it with the various types of tests we discussed. More information on Sauce Labs integrations with CI servers.
Performance Testing. Testing with one user is unlikely to simulate the load that may happen with many multiple users in production. Performance testing is testing for performance and includes load testing, which is testing for multiple users. Both of these involve testing the server, but page rendering can also be important for the mobile device. Soak testing involves running the system for an extended period of time to see if there are memory leaks that happen, and can involve client or server.
Security Testing. The two primary methods to perform security testing are automated scanning of source code and penetration testing. Systems hardening and social engineering (phishing) are worth mentioning. Mobile applications are production code, and deserve the same attention.
Sample Java test script for Android app testing
To get you started, here is an example of a sample Java test script for Android testing using Appium. Complete details and additional scripts can be found here.
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.remote.DesiredCapabilities;
import io.appium.java_client.android.AndroidDriver;
import java.net.URL;
public class SampleSauceCheckBoxTest {
public static final String USERNAME = "YOUR_USERNAME";
public static final String ACCESS_KEY = "YOUR_ACCESS_KEY";
public static final String URL = "https://" + USERNAME + ":" + ACCESS_KEY + "@ondemand.saucelabs.com:443/wd/hub";
public static void main(String[] args) throws Exception {
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability("platformName", "Android");
capabilities.setCapability("deviceName", "Samsung Galaxy S4 Emulator");
capabilities.setCapability("platformVersion", "4.4");
capabilities.setCapability("app", "http://saucelabs.com/example_files/ContactManager.apk");
capabilities.setCapability("browserName", "");
capabilities.setCapability("deviceOrientation", "portrait");
capabilities.setCapability("appiumVersion", "1.5.3");
WebDriver driver = new AndroidDriver<>(new URL(URL), capabilities);
/**
* Test Actions here...
*/
driver.quit();
}
}
About Sauce Labs
If you have not yet heard of Sauce Labs, a few words about us: Sauce’s cloud testing platform has to date run more than 4 billion tests for a variety of organizations from the finance, banking, retail and media verticals and thousands more -- organizations that are serious about testing, which are working with us to make it awesome. This same platform supports not only web browser testing but also supports Android for both automated and manual testing.
Interested in diving in and trying Sauce Labs for Android app testing? Sign up for our Free Trial!
