There are different strategies for mobile testing, but when it comes to simulators/ emulators and real devices, it’s not really an either/or approach. Each solution has its benefits and drawbacks, depending on the stage of an app’s lifecycle. Simulators/ emulators are great for early stages, but testing on real devices before releases is crucial to ensure an app’s success. “A recommended approach is to find a healthy mix of emulators, simulators and real devices to get the best out of your test automation.” (Gartner, Market Guide for Mobile App Test Automation Tools, Maritess Sobejana). We will highlight advantages and disadvantages of iOS and Android emulators as well as real devices and we will also give an affordable solution to test on both!!
Mobile Device Emulator vs Simulator: What is the difference?
Many people use either of these terms to refer to a digitalised object that mimics a real object and that can be used for testing purposes, but in fact simulators and emulators are two very different things:
An emulator has the goal of taking the place of the real (in our case) mobile device. It duplicates every aspect of the original device’s behaviour, both hardware and software. It basically simulates all of the hardware the real device uses, allowing the exact same app to run on it unmodified, and all of the software.
While a simulator sets up a similar safe environment to the original real-life device’s OS, but it doesn’t attempt to simulate the real device’s hardware. So what you will see is the OS and the interface of the device you want to use, but you won’t experience all the problems the hardware might cause. Some apps may run a little differently and that’s the main reason why simulators aren’t very reliable.
Advantages and disadvantages of simulators and emulators
Both of the options mentioned before are widely used solutions when it comes to mobile app testing. The feeling you get is that emulators are better, because they both simulate software and hardware, but that’s not all you want to test when testing an app on a mobile device. As we know, everything comes with upsides and downsides.
Advantages and disadvantages of Android and iOS emulator
- Simulates both software and hardware
- App runs unmodified
- Helps you find unexpected behaviour
- Tend to be free and open source
- Can be connected to an IDE for early testing during development
- It is a fairly inexpensive solution
- Mobile device emulators are very slow (because they simulate both hardware and software)
- A mobile device emulator doesn’t take into consideration factors like battery overheating/drainage or conflicts with other (default) apps
- Setting up a good emulator takes time and it is expensive
- They may be incompatible with the app or app elements, meaning that you will need to create patches here and there to keep on using the emulator
- Emulators may support only certain OS versions
Advantages and disadvantages of iOS and Android simulator
- Fast (because they simulate only the software)
- Relatively easy to set up
- Can be used to study the behaviour of an app
- Helps you find unexpected behaviour, but it can also give you false positives
- It doesn’t take into consideration the hardware
- Apps may run a little differently
- Simulation results may be difficult to analyse, due to incomplete data
Both solutions are an optimal solution for early testing when it comes to mobile testing. There is no need to set up an expensive solution with real devices in the early development stages of an app. Android and iOS simulators and emulators are perfectly suited for this purpose, with easy setup of different OS versions as well as a cheaper solution for a high device coverage.
Advantages and disadvantages of real devices
Well for starters, real devices give you a real feeling of the app, how it performs on real hardware and all the factors that may contribute to make an app work or crash.
Advantages of real devices
- You test in a real environment with very real conditions (weather, location, network, etc.), providing real insights on the functioning of the app
- Screen resolution and brightness can be easily tested in a series of different lighting scenarios
- Performance of a real device is faster compared to other virtual options
- Testing interoperability is easier
- Easy to replicate bugs users are having on their devices
- No false positives, as it happens with virtual testing solutions
- Real devices allow to test push-notification, geolocation and the devices build-in sensors like orientation and wifi connectivity
- Battery drainage issues derived from your app or your app in correlation with other default apps
- You will be testing on the real OS, with the manufacturer’s tweaks, not plain vanilla OS versions
- Test for incoming interrupts, like SMS and calls
Disadvantages of real devices
- There are thousands of mobile devices and having a proper testing pool that contains wide range of devices becomes really expensive
- The maintenance of the device must be taken into account as well
- It is difficult to get devices that are only available in foreign countries People located in other offices or working remotely will not have access to the device
- It is an expensive and time consuming solution
The solution: mobile cloud testing
We have now analysed all viable mobile testing options we could find, comparing advantages and disadvantages, liability and the costs. Based on our research we can easily say that mobile cloud testing is the best solution that is currently available.
Testing in the cloud means accessing hundreds of iOs and Android simulators and emulators as well as real devices, multiple OS and OSs versions, directly from any browser. There is no need to buy mobile devices or set up emulators/simulators, you can test on hundreds of mobile devices right from your browser!
What if something breaks? You don’t have to care, because you’ll be always testing on a functioning device: no maintenance!
And the best part is yet to come: different mobile testing teams dislocated in multiple locations can work on the same project, see the progress, the tests that were executed and on which devices.