Mobile testing continues to grow as mobile devices become more important in our every day lives. According to Statista in 2021, 92.6% (4.32 billion) of internet users accessed the internet through mobile devices (smartphones and tablets). This rate will surely continue to grow.
Mobile application testing (MAT) is the process of testing mobile apps to ensure that mobile apps work as they should on any device or browser. Mobile testing can be done manually on real devices and emulators/simulators or with automation.
Mobile testing helps improve user experience with software, apps, and more. With proper MAT, you can improve QA to catch bugs on the front end and backend, test APIs, and simulate different operating systems.
Just doing device hardware checks has been called “mobile testing” in the past. MAT differentiates itself from the earlier term by widening the scope of testing to include the software of a device (the app), what a user will experience when they use the app, and, ultimately, to aim for increased user satisfaction.
Manual testing is a human input, analysis or evaluation. This approach is user-centric, focusing on explorative ways of monitoring, whether a mobile application meets user requirements and expectations. You should test your app for look & feel and for usability, making sure that it is user friendly. You should not be using manual testing for all your testing, but just for about 20% of them, for the rest you can use automated testing
Automated testing is another mobile application testing approach. While the aim of manual testing is to test the “user experience,” automated mobile testing aims to test all the functionality that characterizes an app, and ensures that all features and functionalities work as expected (for example, regression testing) after code changes and app updates are implemented.
Given the benefits and capabilities of each testing method, an integrated testing approach that incorporates both manual and automated tests is the way to go. And with this approach, a best practice is to automate 80% of your test cases while performing 20% manually.
Here are the specific test cases that should be automated:
Automate the most frequent test-cases
Automate test cases that are easy to automate
Automate test cases that have predictable results
Automate the most tedious manual test cases
Automate frequently used functionality
Automate test cases that add the most business value
Automate test cases that can have high cost implications when the associated feature breaks down in production
Learn more about manual vs. automated mobile testing.
Beta tests allow mobile development teams to validate the applications with “real users” in real-world environments. It helps teams to get feedback from end users who represent an app’s actual user base. By testing application performance and functionality with a beta group prior to general release, teams gain a deeper level of validation that the software meets end-user requirements. They also significantly reduce the risk that an unforeseen application problem post-deployment will necessitate the application release to be rolled back or require the team to push out an update quickly in order to fix a problem with the production release.
Read more about the best practices of mobile app beta testing to learn how beta testing works, why it’s important, and how to get the most out of beta testing processes.
Mobile app testing has three major points of focus, each complementary to the others.
App functionality tests include:
Core steps of what the app should do
Exception Conditions - Incorrect input
Dependency Failures - When third party dependencies are not available
Platforms - Different browsers and mobile devices at various screen resolutions
API testing can test dependencies, expected interactions, and isolate where a failure is.
Beta testing can point out where the app may be flawed, either in it’s execution or in customer experience areas
This is testing specific to the device itself, and usually requires physical devices rather than simulators. This includes, but is not limited to:
Network outages, dropped packets, slow network
Battery drain and heat
Interruptions (Alerts such as calls, text messages)
Gestures (Swipe or force touch)
Sometimes called "non-functional" testing, attribute testing explores qualities of the software outside of pure functionality.
Single user performance testing
Code scanning for security vulnerabilities
Attribute testing may be overlooked by some quality efforts, but it can make or break an app. An app that has an easily exploitable security hole, for example, will be shunned by users, and can force liability onto the developer.
How the app gets used in real-life is important to the decision about testing on production code. Testing on production removes the layer of abstraction introduced by the QA-designed testing environment and brings tests closer to the actual use cases. If it is possible to do so, performing production testing will reduce unwanted surprises at release time.
The mobile app testing tools vary by the specific type of app that is under test. Mobile application types include native, hybrid, web, and progressive web apps.
Native apps run on iOS or Android. iOS apps present as an IPA binary file. The file may be tested with open-sourced Appium and/or Apple’s proprietary XCUITest framework.
Android apps are built into an AAB/APK package. Appium and/or Espresso frameworks can be used for this popular mobile OS. Read more on how you can choose the right mobile test automation framework.
Hybrid apps are essentially web apps that have an application wrapper which is designed to be independent of iOS or Android. This gives the hybrid application the ability to access all of an operating system’s specific capabilities.
Hybrid apps can be tested using the Appium test automation framework, native frameworks (XCUITest, Espresso), as well as Selenium, when the webpages are isolated.
Web apps work with mobile native browsers, like Chrome, Safari, Mozilla’s Firefox, and other “vendor” browsers like Samsung Internet. Because they are pure web applications one can test them by using the Selenium and Appium test automation framework. Though Selenium is a pure web tool, it can be very useful in aspects of MAT.
Progressive web apps are traditional web applications that are enhanced with modern web technologies, allowing them to provide a more app-like experience.
These are web apps that use emerging web browser APIs and features along with traditional progressive enhancement strategy to bring a native app-like user experience to cross-platform web applications. Progressive Web Apps are a useful design pattern, though they aren't a formalized standard.
Appium, Selenium, or JS based Native frameworks like Cypress, Playwright, and others can be used to test PWAs. The JS based Native frameworks allow users to test PWAs on a desktop web browser.
Real device testing is the practice of installing the latest build of a mobile app on a real mobile device to test the app’s functionality, interactions, and integrations in real-world conditions. Real device testing is a recommended component of a comprehensive mobile app testing strategy, especially when used in combination with virtual devices (Android emulators and iOS simulators).
Organizations committed to their mobile application quality efforts are faced with making an important choice regarding where to run their mobile tests. The default for development teams in large enterprises is to use real mobile devices. While this gives them more accurate test results, it is not ideal for scaling and automation of testing.
In an attempt to move away from testing on physical devices, some organizations have switched to using emulators and simulators for their mobile app testing. However, it’s a best practice to incorporate a combination of real devices and virtual devices (mobile emulators and simulators) for the most comprehensive mobile app testing strategy. Let’s dive into what emulators and simulators are and when they should be used for mobile testing.
A mobile emulator, as the term suggests, emulates the device software and hardware on a desktop PC, or as part of a cloud testing platform. It is a complete re-implementation of the mobile software written in a machine level assembly language. The Android (SDK) emulator is one such example.
A simulator delivers a replica of a phone’s user interface and does not represent its hardware. A simulator is a partial re-implementation of the operating system written in a high-level language. The iOS simulator for Apple devices is one such example.
A real device is the actual hardware (plus OS and built-in support resources) on which your software will run in production. For mobile software, it's the mobile phone or tablet. For specialized industrial, scientific, or medical monitoring software, it's the actual monitoring device.
Easy to Provision
Easy to Scale
Detect Hardware Failures
Advanced UI Testing
Easy to Maintain
Here’s a summary of when to use real devices and emulators and simulators in your testing:
What to Test
Functional testing for large integration builds
UI layout testing
Mobile web testing
Manual/ interactive testing on physical devices
Unit/ System testing
Error monitoring and reporting
Hardware dependencies (CPU, GPS etc.)
Display testing (pixels, resolutions)
Replicate issues to match exact model
Push notifications (real services)
Natural gestures (pinch, zoom, scroll)
Read our white paper to know why automated mobile testing requires both emulators/ simulators and real devices for a best-of-breed mobile QA environment and comprehensive coverage.
A real device cloud is a mobile app testing environment that provides on-demand access to real iOS, Android, and other mobile devices (smartphones and tablets). Mobile app development and QA teams can test their apps on different device/OS combinations to get real-world feedback and ensure optimal coverage for their organization’s target customers. A real device cloud can integrate with popular mobile testing frameworks like Appium, Espresso, and XCUITest.
A critical decision for organizations that are defining their automated testing strategy is choosing the right automated testing framework that will help their development and QA teams efficiently write and execute automated tests. The market offers a plethora of frameworks to choose from, and the decision isn’t always easy.
To help you in your evaluation, we have created a comprehensive guide on how to choose the right mobile test automation framework that will provide you insight into:
Criteria to consider as you select a mobile test automation framework
Popular frameworks for automated functional testing
The unique benefits and limitations of each framework with a comparison view
Mobile testing is complex, and therefore you need a testing platform that can serve your needs and help you address the key challenges throughout the mobile app development lifecycle.
Sauce Labs end-to-end mobile quality solutions can help you achieve quality at speed throughout all stages of the mobile app development lifecycle (from development to production), with a secure and scalable test cloud, extensive device coverage, optimized mobile beta testing, and best-in-class error monitoring and reporting.
To provide comprehensive guidance on everything you need to know before choosing the right mobile test automation framework, this whitepaper dives into the three popular solutions available today: Appium, XCUITest and Espresso, and lists out the key factors you should consider when determining when to choose which framework.
Learn how using a real device cloud can help resolve mobile app test errors and help teams streamline debugging.