Mobile device testing presents unique challenges. There’s a lot to keep up with—mobile OS updates, new devices and features, application frameworks for web, native, and hybrid apps, and a new breed of DevOps tools to ensure the mobile apps you ship are reliable and perform well.
Therefore, you have a lot of decisions to make when planning a mobile testing strategy, and there is a lot that can go wrong. With the right approach, however, you can make mobile testing a key advantage for your organization.
Below are five strategies every mobile testing team should keep in mind in order to reap the most from their QA efforts.
1. Mobile testing on real devices
Emulators (and simulators - there are differences, but for the sake of simplicity I will refer to emulators throughout this post) are great for early stage testing, and they have a place in the overall QA process. However, if all your tests are run on emulators, your app will fail in real-world scenarios. Real mobile devices let you test many key device capabilities like GPS, carrier and connectivity issues, battery drain, call and SMS interruption, camera functionality, and more.
Emulators test the app to work on all devices, but don’t take into consideration each device’s unique configuration. For example, if there’s an issue that affects only Samsung Galaxy devices, you’re not likely to find it using an emulator.
With the number of devices available, it can be overwhelming to test your app on every device and OS combination—but that is not possible considering the number of combinations you’d need to test for. The best approach is to identify the most-used devices among your user base and test on them. If your testing covers 80% of devices, you can be fairly confident of your test results.
Maintaining a device lab in-house is expensive, inefficient, and distracts you from the tests you need to run. Instead, renting real mobile devices in the cloud by the minute is a better alternative. Typically, device clouds like TestObject (a Sauce Labs company) have a wide range of phones, and stay updated with new devices regularly.
2. Automate mobile testing
Automated testing is even more essential with mobile apps than traditional apps. With automation, you can run tests in parallel across numerous emulators and real devices. This speeds up test results, and helps bring your apps to market faster. Plus, you can save configurations before running tests, and reuse them whenever needed. When shared across teams, this can save a lot of time.
Not every test should be automated. Some tests are best performed manually, especially if they are run rarely and require a lot of setup. Test automation with mobile requires the right set of tools. To start, Appium is the most popular open source mobile test automation framework that supports both iOS and Android, and lets you write tests once and run them on both platforms.
3. Keep pace with your web app
With the rise of continuous delivery and continuous deployment, web apps are being released on a daily basis. This is possible because the app is released to users directly. However, mobile apps need to be approved by the OS providers (iOS and Android), who act as gatekeepers. They can enforce stringent policies that slow down or even block releases.
iOS and Android have been taking steps to enable faster releases. For example, enabling auto-updating of apps has made it easier to update an app without annoying users with repeated notifications and permissions. Similarly, iOS TestFlight and Android’s beta tests let you release early and frequently to an opted-in list of users who can test new features for you.
Android even lets you release your app in phases with progressive updates. This way, you can deal with bugs and issues that come up for the first set of users, and once the app is fairly stable, release it to your entire user base.
4. Mobile performance and load testing
It’s great to test earlier in the development cycle to identify bugs before they make it to production, but when it comes to mobile performance and load testing, tests are best done later in the cycle. You could leverage numerous cloud-based emulators to mimic user behavior when testing for traffic spikes. Additionally, a tool like JMeter can help with load testing at the protocol level. Apart from this, you’ll also need to test performance on a number of real devices to check for issues like memory leaks, and interruption of other apps or calls.
Android’s Monkey tool generates random actions like clicks, gestures, and system-level events and stress tests applications. With performance and load testing, you need a variety of tactics to cover all bases, but it is essential to make sure your app runs smoothly post-release.
5. Use mobile-specific tools
It’s tempting to get by with manual functional testing done by humans, but mobile testing done the right way requires purpose-built mobile testing tools. Fortunately, today, there are many capable tools for various objectives in mobile testing.
Appium leads the way as the most capable and popular mobile testing platform. Along with it there are many other tools—Robotium is an easy-to-use testing tool for Android. It records any action you perform in the app, and saves it as a test case. Similarly, Espresso helps you write and automate UI tests easily. And Google’s EarlGrey is the iOS alternative to Espresso.
Testing mobile apps is not an easy task, and the landscape keeps changing. However, the underlying principles stay the same. Both emulators and real devices have a place in mobile testing. Release features to beta testers early, and make sure you do performance and load testing in production. Finally, use a combination of testing tools according to their strengths. This combination of the right principles and tools will ensure your mobile apps are a key advantage and not a liability for your organization.
Twain began his career at Google, where, among other things, he was involved in technical support for the AdWords team. His work involved reviewing stack traces, and resolving issues affecting both customers and the Support team, and handling escalations. Later, he built branded social media applications, and automation scripts to help startups better manage their marketing operations. Today, as a technology journalist he helps IT magazines, and startups change the way teams build and ship applications.