How to Choose Mobile Devices for Testing 2023

With the rapidly increasing popularity and complexity of mobile apps, you need to make sure your app works as intended across various device models, operating systems, and browsers. But with tens of thousands of mobile device models on the market and more being released every year, how do you choose which devices to test on, and how many do you actually need to test on for comprehensive testing coverage?

In this article, we’ll provide best practices for selecting mobile devices to test, as well as our recommended devices to test on by region and business size.

What is Mobile App Testing?

Mobile application testing (MAT) is the process of testing mobile apps to ensure that mobile apps work as intended on any device or browser. Mobile testing can be done manually on real devices and emulators/simulators or with automation.

Manual Testing vs. Automated Testing

Deciding between manual testing and automated testing is the first step in choosing the right devices for testing. Manual testing is the most common form of mobile testing. It involves a multi-sensory approach that focuses on the user experience of your app. Humans use your app to test both how the user interface (UI) looks and how the interactive components “feel.”

Automated testing is a single-sense approach where a mobile testing framework like Selenium, Appium,  XCUITest, or Espresso runs test scripts that look for specific functional issues. Test automation is useful when you are running complicated, repetitive tests that are prone to error. 

There are benefits to each testing method. While manual testing may be more time-consuming than automated testing, you can get more specific, detailed feedback on user experience-related issues like performance, battery life, and usability. Humans are also better than computers at  identifying certain issues, such as whether buttons on the screen overlap or are in the wrong place. However, if maximizing efficiency and cost-effectiveness is important to you, we recommend including test automation in your strategy. It will enable you to run tests more quickly and release new versions of your app more frequently. 

Given the benefits and capabilities of each testing method, we suggest you use an integrated testing approach that incorporates both manual and automated tests. For the best results, automate 80% of your test cases while performing 20% manually.

Best Practices for Choosing Mobile Devices for Testing

With so many mobile device models and manufacturers in the market, it’s no longer a safe bet to test only on Apple and Samsung devices. While iPhones are still the most popular mobile phones in every region, manufacturers like Xiaomi, Huawei, and OPPO are disrupting markets in densely populated regions like China and India. Given the magnitude of possibilities, deciding which devices to test on can be overwhelming.

Luckily, we’ve done some of the heavy lifting for you. Below are the best practices to keep in mind as you optimize your mobile device testing strategy.

Best Practice #1: Consider Your Target Users When Selecting Mobile Devices to Test

To ensure your testing coverage is as broad as possible, we recommend selecting your devices based on the most important factors to your target users/region. QA and software development teams can collaborate with Marketing to better understand the user personas and usage patterns of your target audience. If you don't yet have enough user data, you can begin by looking at external data sources to assess which devices are popular in your intended region/market. Helpful data points to consider:

• Traffic volume

• Market share

• Recommended OS/OS version

• Screen sizes

• Popularity

• Region

• Industry

• Company size

And if this isn’t your first rodeo, you can use the customer analytics/data from your app to inform your testing strategy and test coverage.

Best Practice #2: Select the Right Amount and Variety of Devices

To decide how many devices to test on, you first have to consider the type of tests you’ll be running. Since manual tests are more time-consuming, we suggest using a smaller selection (5-10) of the most popular devices for your target market/audience. For manual testing, it is also important to assess the risk of not testing on all devices, and think about a strategic mix of device models that will ensure optimal coverage. [ For example, imagine if the requirement is to validate the functionality and layout of your app for the iPhone X/ XS/ 11 Pro/ 12 Mini/ 13 Mini. You could limit your testing to just one of these device models, instead of testing on all five, as they all have the same screen dimensions. So for the scope of your manual tests, there would be no difference if you choose one or the other.]

With automated testing, you can afford to have a larger and broader range of devices in your arsenal. For public apps, we recommend running your automated tests on 25-30 different devices.

Depending on your target users, a testing strategy that incorporates both Android and iOS devices is your best bet for comprehensive testing coverage. Most midsize and larger companies we work with have a test lab of around 10-20 devices. Companies should consider the optimal device number based on the number/ size of teams involved in testing applications. For instance, the 10-20 device number generally tends to serve customers well when they have one or two teams testing the application. If there are more teams involved in testing other apps/ features, we recommend taking device availability into account which is influenced by the number of builds, PRs and more.

Choosing devices for manual vs. automated testing

When selecting devices for manual testing, a general best practice is to include a high-end model, a low-end model, and a tablet (depending on your app) in your device mix. For iOS, this could look like testing on the following: one high-end iPhone (e.g., iPhone 13), one low-end iPhone (e.g., iPhone SE 2022/iPhone 8), and an iPad. For Android, like with iOS, a good practice is to focus on a popular device brand like Samsung and start with a high-end and low-end device. To get better brand coverage, you could add other top brands like Huawei and Xiaomi to your device selection. Similarly, including Google Pixel devices can give you a head start with testing the latest Android beta versions.

For automated testing, you can use the same device strategy (low/high-end devices), but add a larger mix of OS versions to get optimal coverage and ensure faster test execution. 

Test speed vs. granular device selection

Based on your need for speed, it can also serve your teams better to focus on choosing the right device rather than the granular elements of screen size, OS, and others. However, to ensure sufficient coverage you should select the Android and iOS devices that are used the most by your end-users. For example: Using Android 11/12 on Google Pixel or the latest from Samsung, and on the iOS side, going with iOS 15/16 on the latest models from iPhone 13 or the new14.

While static allocation allows you more fine-grained control over the device used in your tests, it can also cause delays in your test execution if that device isn't available when you run your tests. Sauce Labs offers dynamic device allocation, to be used for all automated mobile application testing in CI/CD to allow you to move faster with your test execution. It allows you to specify the basic parameters for the platform, operating system, and/or type of device you want to use in your tests using regular expressions (regex) to dynamically allocate a device with your specifications from our real device pool.

Recommended mobile devices by company size

Using the factors and considerations above, as well as other sources, here are some recommended mobile devices to choose from based on your organization size. Useful resources for researching and comparing mobile device models, specifications, and analyzing device traffic include GSMArena, PhoneArena, Statista.com, screensiz.es, gs.statcounter.com, and DeviceAtlas.

Best Practice #3: Periodically Reassess Which Devices You Test On

The mobile device market is constantly changing – new players will enter the market, established players like Apple will release their new devices annually, and others’ popularity may ebb and flow.

When planning your testing coverage, it’s important to consider current market conditions as well as upcoming releases. You should reassess your testing plan quarterly or biannually to incorporate new devices that become popular while reducing the priority of (or removing altogether) devices that are waning in popularity.

When selecting device versions, it’s helpful to look at the technical specifications of similar models. For example, if two versions are fairly similar (such as Samsung Galaxy S9 and S10), you can pick the latest version. Useful resources for researching and comparing mobile device models, specifications, and analyzing device traffic include GSMArena, PhoneArena, statista.com,  screensiz.es, gs.statcounter.com, and DeviceAtlas.

For Android devices, Google Pixel phones are always the first to support the latest version of Android OS. If you want to test your app on Android beta versions, Pixels are a good place to start.

Global Best-Selling Smartphones 2022

• Apple iPhone 13

• Samsung Galaxy S22 Plus 

• Apple iPhone 13 Pro and Pro Max

• Apple iPhone 13 Mini

• Samsung Galaxy S22 ultra

• Google Pixel 6 and Pixel 6 Pro

• Samsung Galaxy Z Flip 3 and Z Fold 3

Recommended Mobile Devices by Region

Given the best practices we provided, here are some recommended mobile devices to choose from and add to your list.

Choose the Right Mobile Devices to Ensure Optimal Coverage

A mobile app testing plan that integrates manual and automated testing and includes both the right number and type of mobile devices will ensure your test coverage is comprehensive, efficient, and cost-effective.

Sign up for a free Sauce Labs trial and experience the benefits of mobile testing in the Sauce Labs cloud. Test your mobile apps across the most comprehensive range of real devices (iPhone, iPad, Samsung Galaxy, Google Pixel, Xiaomi, and more) to increase coverage and release with confidence.

Sources:

• GSMArena

• https://www.appbrain.com/stats/top-android-phones

• https://www.counterpointresearch.com/global-top-10-smartphones-2021/

• https://www.statista.com/statistics/755671/united-states-smartphone-market-share-by-model/

• https://www.counterpointresearch.com/global-top-10-smartphones-2021/

• https://gs.statcounter.com/vendor-market-share/mobile

• https://gs.statcounter.com/android-version-market-share/mobile-tablet/worldwide

• https://www.statista.com/statistics/757441/worldwide-htc-smartphone-unit-shipments/

• https://www.statista.com/statistics/271490/quarterly-global-smartphone-shipments-by-vendor/