Mobile Testing Strategy: Planning for Scale
Every good developer knows you should test your app. Testing is a crucial step in the development process that allows you to ensure the quality the app you’re releasing. No one wants to release buggy or regression-filled apps, and testing can help you catch problems before the critical app store submission.
But when it comes to the testing ecosystem, mobile testing is a relatively young field, still undergoing a lot of change and evolution. This places developers and companies building mobile applications in a tough spot. How can teams developing for mobile determine what strategies to use to inform their mobile testing, and what tools and solutions can they use in their mobile testing process?
As the makers of the leading testing platform, the team at Sauce Labs never stops thinking about how to make mobile testing efficient and effective. We’ve thought a lot about mobile testing, and we’ll share our thoughts about how you can build a mobile testing strategy that is efficacious, powerful, scalable, and hassle-free. We’ll also discuss some best practices and thoughts on how to develop your mobile testing processes.
Key Challenges of Mobile Testing
There are many challenges to mobile testing, some of which have similarities to challenges of traditional web app testing, and some which are unique to mobile. We’ll highlight some key challenges and how to address them.
One of the biggest challenges for companies mobile testing is device support. The number of models and types of smartphones on the market today is already large, with more always entering the market. Each device tends to have differences, and deciding which devices to test can be a challenge. While everyone wants to support as many devices as possible, adding devices can greatly increase testing time as well as cost. We believe that the most economical solution to supporting a maximum number of devices while keeping costs low is to use emulators and simulators to run automated tests. Buying or using third party infrastructure services to gain access to real devices can be useful for seeing how your application functions in “real world” conditions, but is significantly more expensive and more difficult to scale than using emulators. The ability to increase device support easily is one benefit to using emulators, but there are others as well, which we will discuss later.
Another related difficulty that mobile testing poses is that of platform support. In addition to the ever-increasing number of devices on the market, there are also a number of OS platforms, and many companies are developing for an increasing number of them. It has become much more important to build both Android and iOS apps, and as more platforms, such as FirefoxOS, enter the market, mobile companies will need to adapt to support those as well. This can be difficult, as many frameworks and tools focus on a single platform, since OSes tend to differ wildly from each other. But in order to build a mobile testing strategy with longevity, companies need to be able to easily change what and how they are testing to keep up with technology. This comes down to using tools that are flexible and can stand the test of the ever-shifting mobile development landscape. An automation framework (such as Appium) that focuses on being cross-platform and versatile can help. Choosing a framework that allows for testing on a variety of platforms and languages gives you the flexibility to continually evolve.
A third challenge is the challenge of developing truly agile processes for mobile development. While continuous integration has become standard for most web development projects, the same cannot be said of mobile development. Sauce Labs believes that if mobile development is to truly mature, strong CI processes and tools need to become as ubiquitous as in the web development world. This is another reason we believe that automated mobile testing is so important. While manual testing is a useful practice, ultimately it is automated testing that can be run as part of CI that will give developers the power to develop faster and more easily. The mobile marketplace already changes faster than the web marketplace, and developers need a way to continue to make changes and improvements to their apps as easily as possible. That’s why we believe mobile CI is important to consider when thinking about a mobile testing strategy, and automated testing is an important part of that.
Other benefits of automated mobile testing
It’s clear that testing is an important step in the development cycle to ensure you’re releasing high quality apps. As we’ve already discussed, automated testing can help you support more devices and move towards CI for mobile development. Automated testing, while challenging, can also be beneficial in other ways.
Speed of Testing
While manual testing can be useful, it is a slow and costly process. By automating your testing, you can run a suite of tests in minutes that would take a manual tester hours. Speeding up your testing can allow you to increase your test coverage so you can test more code and thus be more confident you are releasing bug-free apps.
Not only is manual testing slow, but increasing your testing can be difficult since you’re mostly dependent on hiring more testers to increase your testing capabilities. With automated mobile testing tools, increasing which platforms you are testing on and running significantly more tests is easy. The ability to reuse and adapt existing tests also increases your testing capabilities. And by using a cloud testing platform, you can scale your testing to fit your needs without the lead time needed to increase your local infrastructure.
Another advantage automated testing provides is cost savings. You can spend fewer resources on manual testing, which tends to be expensive, while increasing your test capabilities. Using a third party infrastructure can be even more cost-effective, because it eliminates the need to buy and maintain your own infrastructure, and third party solutions can even increase developer productivity.
So we know now that automated testing is beneficial in many ways to the mobile testing process. Many advantages of traditional web automated testing also apply to mobile automated testing. Automated mobile testing allows you to increase test coverage, scale your testing, maximize device support, and save money. That being said, balance is always key when it comes to testing, and for many companies, a blend of automated and manual testing is the right choice.
Another aspect for which balance is a good mantra is testing on real devices vs. emulators. Emulators can’t give you the pixel-perfect resolution you might need for some testing or allow you to see how your app functions in conjunction with the quirks of real-life phone hardware. But they do allow you to do cost-efficient testing at scale and are a powerful tool to have in your mobile testing arsenal. A healthy mix of real device and simulator/emulator testing can give you the test coverage you need at a reasonable price.
Finally, when building a mobile testing strategy, be sure to leave room for growth, both of the marketplace and your own needs. It’s always important to choose the tools and practices that fit your needs, but it’s helpful to think about what may be coming in the future, to minimize the threat of having to spend precious time and resources redoing or revising systems that are out of date. Choosing flexible and cross-platform testing tools and using easily scalable third-party infrastructure are two ways to do this.
Building a mobile testing strategy is hard, and requires a significant amount of care and thought. But the payoff is large, and allows you to release the best apps possible, develop and iterate faster, and stay ahead of a constantly changing industry.
Sauce Labs - Scalable Mobile Testing