How to Choose Mobile Devices for Testing

What is Mobile App Testing?

Mobile application testing (MAT) is the process of testing mobile apps to ensure that mobile apps work as they should on any device or browser. Mobile testing can be done manually on real devices and emulators/simulators or with automation.

Manual Testing vs. Automated Testing

As you probably know, mobile app testing can be done in two different ways: manual testing and automated testing.

Manual testing is the most common form of mobile testing. It involves a multi-sensory approach that focuses on the user experience of your app. Humans use your app to test how the user interface (UI) looks, how the interactive components “feel,” etc.

Automated testing is a single-sense approach where a mobile testing framework like Selenium, Appium,  XCUITest, or Espresso runs test scripts that look for specific functional issues. Test automation is useful in situations like running complicated, repetitive tests that are prone to error. 

There are benefits to each testing method. While manual testing may be more time-consuming than automated testing, you can get more specific, detailed feedback on user experience-related issues like performance, battery life, and usability. Humans are also better than computers at finding certain issues, such as whether buttons on the screen overlap or are in the wrong place. On the other hand, test automation has major benefits when it comes to efficiency and cost-effectiveness. You can run more tests more quickly and release new versions of your app more frequently.

Given the benefits and capabilities of each testing method, we recommend using an integrated testing approach that incorporates both manual and automated tests. With this approach, a best practice is to automate 80% of your test cases while performing 20% manually.

Best Practices for Choosing Mobile Devices for Testing

With so many mobile device models and manufacturers in the market, it’s no longer a safe bet to test only on Apple and Samsung devices. While iPhones are still the most popular mobile phones in every region, manufacturers like Xiaomi, Huawei, and OPPO are disrupting markets in densely populated regions like China and India. Given all the possibilities, it can feel overwhelming or downright impossible to know what devices you should test on.

Luckily, we’ve done some of the heavy lifting for you. Here are our recommended best practices for selecting mobile devices for testing.

#1: Consider Your Target Users When Selecting Mobile Devices to Test

To ensure your testing coverage is as broad as possible, we recommend selecting your devices based on the most important factors to your target users/region. QA and software development teams can collaborate with Marketing to better understand the user personas and usage patterns of your target audience. If you don't yet have enough user data, you can begin by looking at external data sources to assess which devices are popular in your intended region/market. Helpful data points to consider:

• Traffic volume

• Market share

• Recommended OS/OS version

• Screen sizes

• Popularity

• Region

• Industry

• Company size

And if this isn’t your first rodeo, you can use the customer analytics/data from your app to inform your testing strategy and test coverage.

#2: Select the Right Amount and Variety of Devices

The number of devices to test your app on depends on the type of tests you’re running. Since manual tests are time-consuming, a smaller selection (5-10) of the most popular devices for your target market/audience is recommended. With automated testing, you can afford to have a larger and broader range of devices in your arsenal. For public apps, we recommend running your automated tests on 25-30 different devices.

Depending on your target users, a testing strategy that incorporates both Android and iOS devices is your best bet for comprehensive testing coverage. Most midsize and larger companies we work with have a test lab of around 10-20 devices.

Choosing devices for manual vs. automated testing

When selecting devices for manual testing, a general best practice is to include a high-end model, a low-end model, and a tablet (depending on your app) in your device mix. For iOS, this could look like testing on the following: one high-end iPhone (e.g., iPhone 13), one low-end iPhone (e.g., iPhone SE 2022/iPhone 8), and an iPad. For Android, like with iOS, a good practice is to focus on a popular device brand like Samsung and start with a high-end and low-end device. To get better brand coverage, you could add other top brands like Huawei and Xiaomi to your device selection. Similarly, including Google Pixel devices can give you a head start with testing the latest Android beta versions.

For automated testing, you can use the same device strategy (low/high-end devices), but add a larger mix of OS versions to get optimal coverage and ensure faster test execution. 

Test speed v/s granular device selection

Based on your need for speed,  it could also serve your teams better to focus on choosing the right device versus the granular elements of screen size, OS, and others. But in this case, to ensure sufficient coverage you should select the Android and iOS devices that are used the most by your end-users. For example: Using Android 11 on Google Pixel or the latest from Samsung, and on the iOS side, going with iOS 15 on the latest models from iPhone12 or 13.

Recommended mobile devices by company size

Using the factors and considerations above, as well as other sources, here are some recommended mobile devices to choose from based on your organization size. Useful resources for researching and comparing mobile device models, specifications, and analyzing device traffic include GSMArena, PhoneArena, Statista.com, screensiz.es, gs.statcounter.com, and DeviceAtlas.

#3: Periodically reassess which devices you test on

The mobile device market is constantly changing - new players will enter the market, established players like Apple will release their new devices annually, and others’ popularity may ebb and flow.

When planning your testing coverage, it’s important to consider current market conditions as well as upcoming releases. You should reassess your testing plan quarterly or biannually to incorporate new devices that become popular while reducing the priority of (or removing altogether) devices that are waning in popularity.

When selecting device versions, it’s helpful to look at the technical specifications of similar models. For example, if two versions are fairly similar (such as Samsung Galaxy S9 and S10), you can pick the latest version. Useful resources for researching and comparing mobile device models, specifications, and analyzing device traffic include GSMArena, PhoneArena, statista.com,  screensiz.es, gs.statcounter.com, and DeviceAtlas.

For Android devices, Google Pixel phones are always the first to support the latest version of Android OS. If you want to test your app on Android beta versions, Pixels are a good place to start.

Global best-selling smartphones 2022

1. Apple iPhone 13

2. Apple iPhone 13 Pro Max

3. Apple iPhone 13 Pro

4. Apple iPhone 12

5. Apple iPhone 12 Pro Max

6. Apple iPhone 13

7. Apple iPhone 12 Pro

8. Apple iPhone 11

9. Samsung Galaxy A32 5G

10. Samsung Galaxy A12

11. Xiaomi Redmi 9A

12. Apple iPhone SE 2020

13. Apple iPhone 13 Pro Max

14. Xiaomi Redmi 9

2022 smartphone and mobile device releases

• Apple iPad Pro 2022 – (rumored Fall 2022)

• Apple iPhone 14 range - September 2022

• Google Pixel 6a – Summer 2022 (TBC)

• Google Pixel 7 range – Late 2022

• Google Pixel Fold – Late 2022

• Huawei nova 9 SE (China markets only)

• Samsung Galaxy Z Flip 4 - Summer 2022

• Samsung Galaxy Z Fold 4 – August or September 2022

Recommended mobile devices by region

Given our recommended best practices, here are some recommended mobile devices to choose from and add to your list.

Choose the Right Mobile Devices to Ensure Optimal Coverage

A mobile app testing plan that integrates manual and automated testing and includes both the right number and type of mobile devices will ensure your test coverage is comprehensive, efficient, and cost-effective.

Sign up for a free Sauce Labs trial and experience the benefits of mobile testing in the Sauce Labs cloud. Test your mobile apps across the most comprehensive range of real devices (iPhone, iPad, Samsung Galaxy, Nexus, and more) to increase coverage and release with confidence.

Sources:

• GSMArena

• https://www.appbrain.com/stats/top-android-phones

• https://www.counterpointresearch.com/global-top-10-smartphones-2021/

• https://www.statista.com/statistics/755671/united-states-smartphone-market-share-by-model/

• https://www.counterpointresearch.com/global-top-10-smartphones-2021/

• https://gs.statcounter.com/vendor-market-share/mobile

• https://gs.statcounter.com/android-version-market-share/mobile-tablet/worldwide

• https://www.statista.com/statistics/755671/united-states-smartphone-market-share-by-model/