What to Test When Testing a Mobile App

Mobile app testing is a hot topic. The number of downloadable apps is barely comprehensible (Android Apps: 1,5 million apps; Apple’s App store: 1,4 million apps ) and the device fragmentation is nearly beyond imagination (24,093 Distinct Android devices seen up until August 2015). The only way to develop apps is to test on as many devices as possible through mobile app testing.

This post will focus on what you have to test when testing a mobile app. 

The first thing to understand when engaging with mobile app testing is that there is no easy way and no standard way of doing things, even though apps have standard requirements.

Other important elements when testing mobile apps are the target devices (the devices you choose to test your app on) which should be chosen wisely, usually between 10 - 30 devices and relevant for your target market. Those devices should include, based on territorial coverage, the most used screen sizes and OSs platforms and versions. All the information needed can be found online, starting from screen sizes and OS.

So, as you can see, there is a lot of work to do before actually start testing. We put together regularly the top 10 devices of different countries based on their territorial coverage.

An example of top devices used worldwide would include devices with screen size between 4.0” - 5.2” with resolution between 640 x 1136px - 1080 x 1920px . Next step is to look for the market shares based on OS and manufacturer. Android (87% market share), followed by Apple (13% market share) share the market with a majority, with the majority of the Android devices running Marshmallow (34%) and Lollipop (39%). The majority of Apple device run iOS 10x.

For easier selection, it would also be a good idea to find the different Android market shares based on manufacturer.

With these information you can go ahead and use websites like gsmarena.com to filter out devices with the above mentioned characteristics.

When you actually start testing you need approximately 50 use cases for each OS platform (Android and iOS). Use cases test the above standard requirements, in other words, they test what users might do when using an app. They need to be extensive to find as many bugs as possible to keep users happy.

Different tests are run during different life-cycle stages of an app. Tests can be run manually or automated depending on the performed test. Different tests are performed by different departments (development, QA, and production), depending on the app’s lifecycle stage.

As mentioned before, there are mainly two types of approaches: the manual approach (manual testing) and the UI automation approach (automated testing).

The manual approach is used for Checkups, Usability, Edge Cases and Performance, but it’s not meant for repetitive tests. There are pros and cons to this approach:

Development is an ongoing process, there is no start or finish when talking about app development, but continuous testing for continuous improvement that result in continuous delivery: it’s the never ending cycle of continuous integration!

So let’s think about having new builds or releases each week and every minor improvement needs to be tested. Those minor changes result in 100 different use cases (50 for Android and 50 for iOS) that need to be tested on 20 different devices. It’s about 2000 different combinations to test each week… each week! It’s impossible to do this manually: the answer to this would be the UI automation approach.

The UI automation approach is used for Regression, End-to-end, and Integration tests. Test scripts need to interact with your app and the tests need to be black box tests: you test exactly what the users will see. The test will check if the behavior and the rendering of the app is what was expected. The regression test is very important: it checks if something that was working broke with recent changes.

But as you can’t do everything manually, you can’t automate everything either.

The pros and cons to this approach are:

Pros:

Cons

To wrap it all up, manual testing and test automation always go together, because you cannot test comprehensively with just one testing method. Outlining a clear testing strategy and doing a neat research for the key devices in your target market is another very important part of your mobile testing project.