Ten years ago I saw a Selenium test that could click on a capital "I" that was one pixel wide. I was not capable of clicking on it myself. Around that time, a lot of people began using phrases like "End to end is further than you think." Simulators and emulators, after all, could not simulate heat, poor memory management, low power, and other problems.
Then again, that was ten years ago.
The teams I work with today tend to have a stronger focus on the pipeline of delivery work, along with eight, sixteen, or perhaps thirty-two tests at the same time. This sort of work isn't exactly a great fit for physical devices plugged into a cable and running to a laptop. Managing that many actual devices in the cloud is no picnic either.
In other words, there may be a time and a place for simulators, emulators, and physical devices. The challenge is which tools to use where.
Let's talk about it.
Scaling your tests
A single machine to perform builds is certainly a way to start; it was how Extreme Programming started. That takes a single machine and allows it to run tests on one phone.
Most modern engineering processes want to run tests on every new commit pushed to the branch, and possibly on every branch. Even with just one branch, if the team adds ten minutes of new automation per sprint, that phone is going to be bogged down within a couple of months. Plus, you likely don't want to run on the same device every time -- you'd like to switch things up, running on a tablet, or an older device, or a different operating system.
Enter Simulators and Emulators.
The place for emulators
Emulators re-create the entire device, down to the chip, in software. You can think of it almost as virtualization, as it is possible to run Windows on a Macintosh laptop with a tool like VMWare -- even back when the computers ran on an entirely different Central Processing Unit (CPU). Emulators don't have heat or battery problems. They may perform at a different speed than the real thing. Still, when it comes to testing of normal functions under normal conditions, an emulator is going to do the same thing as the real device. Which is strange, given their reputation for being different.
Because tests on emulators are software-defined, multiple tests can be run on tens of emulators at the click of a button without having to manually prepare each emulator for the tests. Further, automation is easier with emulators as the tests can be executed without manual intervention, and be controlled remotely.
The problem with emulators is speed and memory. That extra level of processing, like virtual machines, leads to a lag. Yes, you can run emulators in the cloud, but when it comes to full speed ahead, most people in reality use simulators, the emulator’s less realistic, but easier to run cousin.
The place for simulators
Simulators create the appearance of the device in software. They do not re-create the device all the way down to the CPU, but instead mimic the look and feel. Apple, for example, has been frantically uniting their mobile operating system, iOS, with their desktop, OSX, for years. The Safari browser in the two is essentially the same. Through the user of shared code libraries, Apple can easily put a desktop simulator in their XCode coding tool. Plus, it runs much more quickly than the emulator.
With a smaller memory footprint than an emulator, easier to "spin up" and down, and easier to run in parallel, simulators look like the quick fix for a Continuous Delivery Pipeline or even a simple test run. Until you consider that the simulators are the least like the real thing. Simulators represent what the tool provider expects the device will do … not what it will actually do.
As it turns out, you want to use all three options.
And physical devices as well
If any one thing I've done will be remembered in ten more years, then it will probably be for championing the balanced breakfast strategy. That is to say, it takes several things. Yes, test by hand with physical devices. Sit on a bus and go through different cellular networks. Leave the phone on with the application active and see how fast the battery drains or how much network you use. Feel the application, viscerally, in your hand. Test every feature with a real device by hand. Right alongside you can take some small subset of those experiments and describe them in so much detail they can be executed by a computer on every regression run.
Best of all, you can run real devices in the cloud. Several companies, Sauce Labs included, have ways to connect to real devices, to either run test automation, or, perhaps, to have a human look at a device/operating system combination that you don't have available. Done as part of human exploration, real devices in the cloud can be inexpensive and easy.
Running tests in the cloud has benefits too, in that you can run a great number, you can run several at the same time, and you can get a video of test run results. The downside of this, in general, is higher cost and slower test results.
Putting it all together
To get the most out of their mobile testing efforts, organizations need both real devices and emulators and simulators. One option promises more accurate test results, while the other delivers greater agility.
Coded correctly, you can change the type of device at runtime for any test run. A general test run could use simulators (easy, fast, cheap), perhaps the overnight run uses emulators, and the occasional regression or "what if" or "big browser upgrade" run could use real devices. Create a strategy with a balanced breakfast, and you may find you can have speed, cost, and coverage, all at the same time.
It's a great time to be in testing.
Want to learn more? Download our white paper, Automated Mobile Testing Requires Real Devices & Emulators/Simulators.
