This article offers advice and best practices for how to select mobile devices for testing, depending on region and device popularity.
How to Choose Mobile Devices for Testing
Posted Oct 6th, 2021
With the rapidly increasing popularity and complexity of mobile apps, you need to make sure your app works as intended across various device models, operating systems, and browsers. But with tens of thousands of mobile device models on the market and more being released every year, how do you choose which devices to test on, and how many do you actually need to test on for comprehensive testing coverage?
In this article, we’ll provide best practices for selecting mobile devices to test, as well as provide our recommended devices to test on by region.
Manual testing vs. automated testing
As you probably know, mobile app testing can be done in two different ways: manual testing and automated testing.
Manual testing is the most common form of mobile testing. It involves a multi-sensory approach that focuses on the user experience of your app. Humans use your app to test how the user interface (UI) looks, how the interactive components “feel,” etc.
Automated testing is a single-sense approach where a computer application like Selenium or Appium runs test scripts that look for specific functional issues. Test automation is useful in situations like running complicated, repetitive tests that are prone to error.
There are benefits to each testing method. While manual testing may be more time-consuming than automated testing, you can get more specific, detailed feedback on user experience-related issues like performance, battery life, and usability. Humans are also better than computers at finding certain issues, such as whether buttons on the screen overlap or are in the wrong place. On the other hand, test automation has major benefits when it comes to efficiency and cost-effectiveness. You can run more tests more quickly and release new versions of your app more frequently.
Given the benefits and capabilities of each testing method, we recommend using an integrated testing approach that incorporates both manual and automated tests. With this approach, a best practice is to automate 80% of your test cases while performing 20% manually.
Best practices for choosing mobile devices for testing
With so many mobile device models and manufacturers in the market, it’s no longer a safe bet to test only on Apple and Samsung devices. While iPhones are still the most popular mobile phones in every region, manufacturers like Xiaomi, Huawei, and OPPO are disrupting markets in densely populated regions like China and India. Given all the possibilities, it can feel overwhelming or downright impossible to know what devices you should test on.
Luckily, we’ve done some of the heavy lifting for you. Here are our recommended best practices for selecting mobile devices for testing.
Best practice #1: Consider your target users when selecting mobile devices to test
To ensure your testing coverage is as broad as possible, we recommend selecting your devices based on the most important factors to your target users/region. QA and software development teams can collaborate with Marketing to better understand the user personas and usage patterns of your target audience. If you don't yet have enough user data, you can begin by looking at external data sources to assess which devices are popular in your intended region/market. Helpful data points to consider:
Traffic volume
Market share
Recommended OS/OS version
Screen sizes
Popularity
Region
Industry
Company size
And if this isn’t your first rodeo, you can use the customer analytics/data from your app to inform your testing strategy and test coverage.
Best practice #2: Select the right number and type of devices
The number of devices to test your app on depends on the type of tests you’re running. Since manual tests are time-consuming, a smaller selection (5-10) of the most popular devices for your target market/audience is recommended. With automated testing, you can afford to have a larger and broader range of devices in your arsenal. For public apps, we recommend running your automated tests on 25-30 different devices.
Depending on your target users, a testing strategy that incorporates both Android and iOS devices is your best bet for comprehensive testing coverage. Most midsize and larger companies we work with end up with a test lab of 10-20 devices.
When selecting devices for manual testing, a general best practice is to include a high-end model, a low end model, and a tablet (depending on your app) in your device mix. For iOS, this could look like testing on the following: one high-end iPhone (eg. iPhone 12/13), one low-end iPhone (eg. iPhone SE 2020/iPhone 8), and an iPad. For Android, like with iOS, a good practice would be to focus on a popular device brand like Samsung and start with a high-end and low-end device. In order to get better brand coverage you could add other top brands like Huawei and Xiaomi to your device selection. Similarly, including Pixel devices can give you a head start with testing the latest Android beta versions.
For automated testing, you can use the same device strategy (low/ high-end devices), but add a better diversity in OS versions to get optimal coverage and ensure faster test execution.
Best practice #3: Periodically reassess which devices you test on
The mobile device market is constantly changing - new players will enter the market, established players like Apple will release their new devices annually, and others’ popularity may ebb and flow.
When planning your testing coverage, it’s important to consider current market conditions as well as upcoming releases. You should reassess your testing plan quarterly or biannually to incorporate new devices that become popular while reducing the priority of (or removing altogether) devices that are waning in popularity.
When selecting device versions, it’s helpful to look at the technical specifications of similar models. For example, if two versions are fairly similar (such as Samsung Galaxy S9 and S10), you can pick the latest version. A useful resource for researching and comparing mobile devices is GSMArena.
For Android devices, Google Pixel phones are always the first to support the latest version of Android OS. If you want to test your app on Android beta versions, Pixels are a good place to start.
Top 10 most popular smartphones globally as of 2021 Q1:
Apple iPhone 12
Apple iPhone 12 Pro Max
Apple iPhone 12 Pro
Apple iPhone 11
Xiaomi Redmi 9A (India)
Xiaomi Redmi 9 (India)
Samsung Galaxy A12
Xiaomi Redmi Note 9 (India)
Samsung Galaxy A21s
Samsung Galaxy A31
Major mobile device releases:
Apple iPhone 13 series - September 2021
Xiaomi 11T series - September 2021
Google Pixel 6 series - Late 2021
OnePlus 9T - Fall 2021
Huawei P50 series - August 2021 in China, worldwide TBD
iPhone SE 3 - Late 2021 (rumored)
Samsung Galaxy S22 series - Early 2022
Given the above factors and considerations, here are some recommended mobile devices to choose from.
Conclusion
A mobile app testing plan that integrates manual and automated testing and includes both the right number and type of mobile devices will ensure your test coverage is comprehensive, efficient, and cost-effective.
Sign up for a free trial and experience the benefits of mobile testing in the Sauce Labs cloud. Test your mobile apps across the most comprehensive range of real devices (iPhone, iPad, Samsung Galaxy, Nexus, and more) to increase coverage and release with confidence.
