Better Together: Real Devices, Emulators & Simulators for Mobile Testing

Posted by Clinton Sprauve in Mobile Development & Testing

For several years now, using emulators and simulators to test mobile applications has gotten a bad rap. This is based on the perception that if you’re not testing on a real device, you’re not testing at all.

While real devices give more accurate test results, using them alone is not ideal for scaling and automation of testing. Some startups and SMBs may ignore real devices altogether as they’re too expensive, and opt for the more convenient option--emulators and simulators. The reality is that effective and efficient mobile app development requires both emulators/simulators and real devices. The focus of this article is on the importance of emulators and simulators, why they’re still needed for mobile test automation, and how they can augment real devices in a mobile testing strategy. 

Relying on Real Mobile Devices Alone Cripples Mobile QA

Software testing went from testing in the data center to testing web apps in the cloud using browser automation tools like Selenium. However, with mobile, many large organizations prefer to do the bulk of their testing on real devices. While the use of automated testing is on the rise, many apps are still built on a developer’s Mac or PC, and are manually tested on the few devices available in a device lab. This results in inefficient launches that are focused on functionality and user interface; they ignore important factors like stability, networks, and laggy client performance. Post-launch, developers rely heavily on users to act as their debugging layer; developers then fix the issues in the next update, or the next. Low-quality releases are reflected in poor ratings that flood the app listing, which result in fewer new installs, lower daily average users (DAUs), and eventually lost revenue.

Using Emulators and Simulators Helps Scale and Automate Mobile Testing

In an attempt to move away from testing on physical devices, some organizations have switched to using emulators or simulators for their testing.

An emulator, as the term suggests, emulates the device software and hardware on a desktop PC, or as part of a cloud testing platform. It is a complete re-implementation of the mobile software written in a machine-level assembly language. The Android (SDK) emulator is one example.

A simulator, on the other hand, delivers a replica of a phone’s user interface, and does not represent its hardware. It does not run the real device OS; rather, it’s a partial re-implementation of the operating system written in a high-level language. The iOS simulator for Apple devices is one such example.

Emulators are much faster to provision than real devices, as they are software-driven. Additionally, they enable parallel testing, and test automation via external frameworks like Appium or Espresso. Selenium revolutionized the world of web app testing by pioneering browser-based test automation. Today, Appium is its counterpart for mobile app testing. Appium uses the same WebDriver API that powers Selenium, and enables automation of native, hybrid, and mobile web apps. This brings huge improvements in the speed of tests for organizations coming from the manual world of testing on real devices. Emulators enable parallel testing in a way that can’t be achieved with devices in a lab. Because tests on emulators are software-defined, multiple tests can be run on tens of emulators at the click of a button without having to manually prepare each emulator for the tests. Further, automation is easier with emulators as the tests can be executed without manual intervention, and be controlled remotely.

Of course, for those times when more control is needed, manually running the test individually on an emulator is always an option. However, QA teams that start to use emulators may swing to the other extreme of stopping all testing on real devices. While this speeds up the testing process, it comes with a critical drawback — emulators can’t fully replicate device hardware. This makes it difficult to test against real-world scenarios using an emulator. Issues related to the kernel code, the amount of memory on a device, the Wi-Fi chip, and other device-specific features can’t be replicated on an emulator. It’s not enough to test on emulators alone. Real devices are an important part of the QA process.

Emulators/Simulators and Real Devices Are Better Together 

The ideal QA strategy employs a mix of emulators and real devices. This option addresses the scalability and cost inefficiencies that come with real devices, while retaining the ability to test under real usage conditions. It provides the best of both worlds. Once the decision has been made to add emulators to the mix, there may still be uncertainty about which tests to run on emulators, and which tests to run on real devices.

One way to decide where to run mobile tests is based on immediate testing needs. For example, if an app is in alpha stage, pixel-perfect UI testing isn’t necessary, and if the team wants to run multiple low-level tests in parallel, emulators are the best bet. On the other hand, if the goal is a revamp of the user interface of an app, and the look and feel and exact color shades matter, it may be best to lean towards real devices.

This hybrid approach of picking and choosing where to run which tests is a great way to start small and not be overwhelmed by all the changes. The key is to start somewhere, and build upon the starting point.

Organizations that rely on real devices alone handicap their mobile testing efforts. Emulators are complementary to real devices, but they can’t deliver the real-world environment that a device can deliver. Real devices and emulators, when used together in an automated testing environment, enable QA teams to get the most out of their mobile testing efforts. And, mobile testing in parallel across multiple platforms helps speed up tests while optimizing costs. Any organization that competes in the mobile marketplace cannot afford to ignore the value of using both real devices and emulators in their QA efforts.

Learn more - download our white paper, Automated Testing Requires Both Emulators and Real Devices.

Discuss: Better Together: Real Devices, Emulators & Simulators for Mobile Testing
0 Comments

Free Trial

Get access to a free 14-day trial version, or contact Sales for more information.