To put any idea into a practical action plan, you need a strategy. Same goes for mobile testing: if you don’t have a strategy, you cannot test effectively and efficiently. In this article, we’ll explain shortly the factors you need to take into consideration when creating a winning mobile testing strategy.
Mobile testing challenges
To create a strategy you have to know what challenges will stand in your way. After you identify those, you can start to explore the different options you have, to create an action plan.
One of the very first things to figure out is to decide which will be your target market, where you want your app to be available. You can then start to face the challenges using the information you have on your target market.
The biggest challenge to face in mobile testing is definitely a huge, growing number of Android mobile devices and therefore screen-sizes and OSs fragmentation. Apple has less mobile devices but there is still some fragmentation there. Localization, user’s mobility, and different mobile networks are other main mobile testing challenges.
8 steps for a mobile testing strategy
1 – Device selection
The device selection is strictly related to your target market. You will want to test on those devices that have high coverage in the market you want to distribute your app in. You can use different criteria to find the most suitable devices to test on. You can analyze the most sold devices in that particular market or the most used screen sizes and Android Os versions.
When you have collected all those information, you can start by choosing 5 – 10 devices that are most relevant. You can always expand the testing devices later on. This mobile testing cheat-sheet will help you get started.
2 – Real device vs. device emulator
After having analyzed the market and chosen the devices you want to test on, you will have to choose if you want to use real devices or mobile device emulators for testing. Both have benefits and drawbacks.
- Give you more reliable results
- Testing will be performed on the same devices your users will use, manufacturer OS tweaks included
- You can test for incoming interrupts like calls or SMS
- No false positives like it happens on virtual solutions
- Real devices need maintenance
- Difficult to get devices that are only available in certain countries
Mobile device emulators:
- Simulates both hardware and software
- Many are free and open source
- Helps you find unexpected behavior
- Can be connected to an IDE for testing in the early stages of development
- Emulators are usually slower than a real device
- It won’t take into consideration factors like battery drainage/overheating or conflicts with other preinstalled apps
- They may not support all the OS versions
- Setting up or building a good emulator might take a lot of time and can be expensive
If you’re interested in reading more about this topic, you will find this article about mobile device emulators very informative.
3 – In the cloud vs. in-house
There is a lot of cloud-talk lately, everything seems to be in the cloud but is it the best option for testing? We think yes, but there are in-house solutions if you have special requirements.
Testing in the cloud can overcome obstacles like acquiring and maintaining many devices, as well as accessing the device pool from different locations. The cloud is a useful tool to meet today’s need of mobility and teams located all over the world.
An in-house solution is more adaptable to individual needs – making it a perfect solution for companies that need an extra security layer and their data cannot leave the office. Unfortunately due to the storage space the device choice will be limited.
4 – Network connectivity
Many apps need a network connection to work properly, but it’s impossible to test the app on all networks. Moreover, you have to take into consideration the roaming in foreign countries and how will that affect your app. To test networks efficiently you would have to travel everywhere using SIM cards from every provider. Cloud testing would be an ideal solution for this purpose.
5 – Manual vs. automated testing
Many people see manual testing and automated testing as two opposing testing methods, while in fact, they are complementary.
They should both be used to test apps efficiently and comprehensively.
Automated testing should be used to test repetitive use cases, use cases that need to be run often because they test basic functionalities, tests that have predictable and verifiable results. You should also test with automation all those test cases that are impossible to test manually.
Manual testing is used to test the look and feel of an app, how it is to use it, if there is something missing, if it is easy to navigate etc. Manual testing is also very useful to reproduce bugs. It should also be used to test complicated use cases.
One way to connect the two testing methods would be by using integrated testing. This option combines manual and automated testing in a 20/80 ratio to test comprehensively and save ultimately both time and money.
6 – Performance testing
Performance testing is very important because mobile devices don’t have unlimited storage space. Storage can be now bought at very low prices, but not every mobile device can expand storage and between users content and apps cached files, your app performance can be threatened.
If your app generates content files (e.g. pictures, videos…), the storage space used by it on your device will grow over its lifetime, depriving users of precious space, without them knowing. If your app works on servers, you should also consider testing what happens with the growing number of users. Will your server crash??
There are different ways to test your app for its performance. Typical test you can run are:
- Load testing
- Stress testing
- Endurance testing
- Spike testing
7 – Security testing
Many people are growing more and more worried about their data and credentials being exposed. So it is crucial to make sure if logins credentials are correctly encrypted when authenticating over a network. The connection might not always be a secure one.
Sensitive data, e.g. mobile banking information should also not be stored in mobile devices.
The easiest way to test this is to:
- Open the application/open web app
- Log in
- Close the application/click “back” in the browser
- Open the application again/navigate to web app again
If you are asked to log in again, then your app/web app is secure; if it stays logged in, then you have a problem.
Sometimes checking for security issues is more difficult than this and needs out-of-the-box thinking, but sometimes a simple test is enough.
8 – UI testing
The UI needs testing as well: it is what will tell you if your app works, is easy and intuitive to use, and if it meets users expectations. It is a testing method to identify bugs and flaws using the User Interface.
In the end, to win at mobile testing you have to be aware of the challenges and build a strong mobile testing strategy. To create a bulletproof strategy you have to keep these 8 points in mind:
- Device selection: selecting the right mobile devices to test on is important to ensure the success in your target market.
- Real devices vs. emulators: choosing to use one or the other is your choice, but be aware of pros and cons of each.
- In the cloud vs. in-house: this choice depends on your requirements and what you want to achieve with testing.
- Network connectivity: apps can behave differently under different networks.
- Manual vs. automated testing: two different but complementary testing methods
- Performance testing: will that app take up too much of the user’s storage space? Will it crash?
- Security testing: credential must be encrypted, always.
- UI testing: ultimately look and usability are what keep users using an app.