What is Mobile App Testing?
Mobile application testing (MAT) is the process of testing mobile apps to ensure that mobile apps work as they should on any device or browser. Mobile testing can be done manually on real devices and emulators/simulators or with automation.
Manual Testing vs. Automated Testing
As you probably know, mobile app testing can be done in two different ways: manual testing and automated testing.
Manual testing is the most common form of mobile testing. It involves a multi-sensory approach that focuses on the user experience of your app. Humans use your app to test how the user interface (UI) looks, how the interactive components “feel,” etc.
Automated testing is a single-sense approach where a mobile testing framework like Selenium, Appium, XCUITest, or Espresso runs test scripts that look for specific functional issues. Test automation is useful in situations like running complicated, repetitive tests that are prone to error.
There are benefits to each testing method. While manual testing may be more time-consuming than automated testing, you can get more specific, detailed feedback on user experience-related issues like performance, battery life, and usability. Humans are also better than computers at finding certain issues, such as whether buttons on the screen overlap or are in the wrong place. On the other hand, test automation has major benefits when it comes to efficiency and cost-effectiveness. You can run more tests more quickly and release new versions of your app more frequently.
Given the benefits and capabilities of each testing method, we recommend using an integrated testing approach that incorporates both manual and automated tests. With this approach, a best practice is to automate 80% of your test cases while performing 20% manually.
Best Practices for Choosing Mobile Devices for Testing
With so many mobile device models and manufacturers in the market, it’s no longer a safe bet to test only on Apple and Samsung devices. While iPhones are still the most popular mobile phones in every region, manufacturers like Xiaomi, Huawei, and OPPO are disrupting markets in densely populated regions like China and India. Given all the possibilities, it can feel overwhelming or downright impossible to know what devices you should test on.
Luckily, we’ve done some of the heavy lifting for you. Here are our recommended best practices for selecting mobile devices for testing.
#1: Consider Your Target Users When Selecting Mobile Devices to Test
To ensure your testing coverage is as broad as possible, we recommend selecting your devices based on the most important factors to your target users/region. QA and software development teams can collaborate with Marketing to better understand the user personas and usage patterns of your target audience. If you don't yet have enough user data, you can begin by looking at external data sources to assess which devices are popular in your intended region/market. Helpful data points to consider:
And if this isn’t your first rodeo, you can use the customer analytics/data from your app to inform your testing strategy and test coverage.
#2: Select the Right Amount and Variety of Devices
The number of devices to test your app on depends on the type of tests you’re running. Since manual tests are time-consuming, a smaller selection (5-10) of the most popular devices for your target market/audience is recommended. With automated testing, you can afford to have a larger and broader range of devices in your arsenal. For public apps, we recommend running your automated tests on 25-30 different devices.
Depending on your target users, a testing strategy that incorporates both Android and iOS devices is your best bet for comprehensive testing coverage. Most midsize and larger companies we work with have a test lab of around 10-20 devices.
Choosing devices for manual vs. automated testing
When selecting devices for manual testing, a general best practice is to include a high-end model, a low-end model, and a tablet (depending on your app) in your device mix. For iOS, this could look like testing on the following: one high-end iPhone (e.g., iPhone 13), one low-end iPhone (e.g., iPhone SE 2022/iPhone 8), and an iPad. For Android, like with iOS, a good practice is to focus on a popular device brand like Samsung and start with a high-end and low-end device. To get better brand coverage, you could add other top brands like Huawei and Xiaomi to your device selection. Similarly, including Google Pixel devices can give you a head start with testing the latest Android beta versions.
For automated testing, you can use the same device strategy (low/high-end devices), but add a larger mix of OS versions to get optimal coverage and ensure faster test execution.
Test speed v/s granular device selection
Based on your need for speed, it could also serve your teams better to focus on choosing the right device versus the granular elements of screen size, OS, and others. But in this case, to ensure sufficient coverage you should select the Android and iOS devices that are used the most by your end-users. For example: Using Android 11 on Google Pixel or the latest from Samsung, and on the iOS side, going with iOS 15 on the latest models from iPhone12 or 13.
Recommended mobile devices by company size
Using the factors and considerations above, as well as other sources, here are some recommended mobile devices to choose from based on your organization size. Useful resources for researching and comparing mobile device models, specifications, and analyzing device traffic include GSMArena, PhoneArena, Statista.com, screensiz.es, gs.statcounter.com, and DeviceAtlas.
Startup
Best Practice: Select a small mix of high-end and low-end iOS and Android devices, including smartphones and tablets. Add the latest Google Pixel model to test Android beta versions.
|
SMB
Best Practice: Broaden your device coverage by adding additional high-end and low-end iOS and Android models, plus other brands depending on popularity/market share and your target market/audience.
|
Enterprise
Best Practice: Choose a broad mix of high-end/low-end iOS and Android models and devices, the latest Google Pixel model, and other brands depending on popularity/market share and your target market/region.
|
iOS:
- iPhone 12 (iOS 15.x)
- iPhone X/XS/11 (iOS 14.x)
- iPhone 6(S)/7(S)/8 (iOS 13.x)
- iPad (latest with iOS 15.x)
Android:
- Samsung Galaxy S20/21 (Android 12)
- Samsung Galaxy A50/51/52 (Android 11)
- Samsung Galaxy A40/41/42 (Android 10)
- Samsung Galaxy A11/12/13 (Android 9)
- Samsung Galaxy Tab A8 (latest with latest supported Android on it)
Beta:
- Google Pixel 5/6 (exclusively for Beta versions)
- iPhone 12/13 (exclusively for Beta versions)
| To the Startup list, add additional iOS and Android devices from the below list, choosing equally between the two operating systems unless mentioned explicitly.
iOS:
- iPhone XR or XS/11/12/13 Max (pro not needed)
- iPhone 7S/8 Plus (models with larger screens)
- iPhone SE 2020 or 2022
- iPad Mini/iPad Pro
Android:
- Samsung Galaxy S20/21S22 (Android 12)
- Samsung Galaxy S7/8/9 (Android 8)
- Xiaomi 11T (high end)
- Xiaomi 11 Lite (low end)
- Samsung Galaxy Tab S6/S7 - light (with latest supported Android OS)
- Samsung Galaxy Tab S6/S7/S8 (with latest supported Android OS)
| To the Startup + SMB list, add additional Android devices from the below list.
Android:
Add at least one high-end and one low-end model from the following brands:
|
#3: Periodically reassess which devices you test on
The mobile device market is constantly changing - new players will enter the market, established players like Apple will release their new devices annually, and others’ popularity may ebb and flow.
When planning your testing coverage, it’s important to consider current market conditions as well as upcoming releases. You should reassess your testing plan quarterly or biannually to incorporate new devices that become popular while reducing the priority of (or removing altogether) devices that are waning in popularity.
When selecting device versions, it’s helpful to look at the technical specifications of similar models. For example, if two versions are fairly similar (such as Samsung Galaxy S9 and S10), you can pick the latest version. Useful resources for researching and comparing mobile device models, specifications, and analyzing device traffic include GSMArena, PhoneArena, statista.com, screensiz.es, gs.statcounter.com, and DeviceAtlas.
For Android devices, Google Pixel phones are always the first to support the latest version of Android OS. If you want to test your app on Android beta versions, Pixels are a good place to start.
Global best-selling smartphones 2022
Apple iPhone 13
Apple iPhone 13 Pro Max
Apple iPhone 13 Pro
Apple iPhone 12
Apple iPhone 12 Pro Max
Apple iPhone 13
Apple iPhone 12 Pro
Apple iPhone 11
Samsung Galaxy A32 5G
Samsung Galaxy A12
Xiaomi Redmi 9A
Apple iPhone SE 2020
Apple iPhone 13 Pro Max
Xiaomi Redmi 9
2022 smartphone and mobile device releases
Apple iPad Pro 2022 – (rumored Fall 2022)
Apple iPhone 14 range - September 2022
Google Pixel 6a – Summer 2022 (TBC)
Google Pixel 7 range – Late 2022
Google Pixel Fold – Late 2022
Huawei nova 9 SE (China markets only)
Samsung Galaxy Z Flip 4 - Summer 2022
Samsung Galaxy Z Fold 4 – August or September 2022
Recommended mobile devices by region
Given our recommended best practices, here are some recommended mobile devices to choose from and add to your list.
North America
| APAC
| EMEA
|
iOS
- iPhone 12 (iOS 15.x)
- iPhone 7S/8 Plus (models with larger screens)
- iPhone 6(S)/7(S)/8 (iOS 13.x)
- iPhone SE 2020 or 2022
- iPhone XR or XS/11/12/13 Max (pro not needed)
- iPad (latest with iOS 15.x)
- iPad Mini/iPad Pro
Android
- Samsung Galaxy S20/21 (Android 12)
- Samsung Galaxy A50/51/52 (Android 11)
- Samsung Galaxy A40/41/42 (Android 10)
- Samsung Galaxy A11/12/13 (Android 9)
- Samsung Galaxy Tab A8 (with latest supported Android OS)
- Samsung Galaxy S7/8/9 (Android 8)
- Samsung Galaxy Tab S6/S7 - light (with latest supported Android OS)
- Samsung Galaxy Tab S6/S7/S8 (with latest supported Android OS)
Beta
- Google Pixel 5/6 (exclusively for Beta versions)
- iPhone 12/13 (exclusively for Beta versions)
| iOS
- iPhone 12 (iOS 15.x)
- iPhone 7S/8 Plus (models with larger screens)
- iPhone 6(S)/7(S)/8 (iOS 13.x)
- iPhone SE 2020 or 2022
- iPhone XR or XS/11/12/13 Max (pro not needed)
- iPad (latest with iOS 15.x)
- iPad Mini/iPad Pro
Android
- Samsung Galaxy S20/21 (Android 12)
- Samsung Galaxy A50/51/52 (Android 11)
- Samsung Galaxy A40/41/42 (Android 10)
- Samsung Galaxy A11/12/13 (Android 9)
- Samsung Galaxy Tab A8 (with latest supported Android OS)
- Samsung Galaxy S7/8/9 (Android 8)
- Samsung Galaxy Tab S6/S7 - light (with latest supported Android OS)
- Samsung Galaxy Tab S6/S7/S8 (with latest supported Android OS)
Beta
- Google Pixel 5/6 (exclusively for Beta versions)
- iPhone 12/13 (exclusively for Beta versions)
| iOS
- iPhone 12 (iOS 15.x)
- iPhone 7S/8 Plus (models with larger screens)
- iPhone 6(S)/7(S)/8 (iOS 13.x)
- iPhone SE 2020 or 2022
- iPhone XR or XS/11/12/13 Max (pro not needed)
- iPad (latest with iOS 15.x)
- iPad Mini/iPad Pro
Android
- Samsung Galaxy S20/21 (Android 12)
- Samsung Galaxy A50/51/52 (Android 11)
- Samsung Galaxy A40/41/42 (Android 10)
- Samsung Galaxy A11/12/13 (Android 9)
- Samsung Galaxy Tab A8 (with latest supported Android OS)
- Samsung Galaxy S7/8/9 (Android 8)
- Xiaomi 11T (high end)
- Xiaomi 11 Lite (low end)
- Samsung Galaxy Tab S6/S7 - light (with latest supported Android OS)
- Samsung Galaxy Tab S6/S7/S8 (with latest supported Android OS)
- Huawei (choose one high-end and one low-end device)
- Oppo (choose one high-end and one low-end device)
- Vivo (choose one high-end and one low-end device)
Beta
- Google Pixel 5/6 (exclusively for Beta versions)
- iPhone 12/13 (exclusively for Beta versions)
|
Choose the Right Mobile Devices to Ensure Optimal Coverage
A mobile app testing plan that integrates manual and automated testing and includes both the right number and type of mobile devices will ensure your test coverage is comprehensive, efficient, and cost-effective.
Sign up for a free Sauce Labs trial and experience the benefits of mobile testing in the Sauce Labs cloud. Test your mobile apps across the most comprehensive range of real devices (iPhone, iPad, Samsung Galaxy, Nexus, and more) to increase coverage and release with confidence.
Sources: