You Can Have It All: How to Maximize Your Software Testing Coverage

In a world where the Android ecosystem alone has spawned 24,000 different types of mobile devices—to say nothing of iOS, a smaller but still diverse device market—deciding which devices to support when writing and testing software can be a challenge. When you factor in multiple types and versions of operating systems and web browsers, building a software testing strategy that covers as many software environments as necessary may feel like a decidedly impossible battle.

Faced with these challenges, some organizations are tempted not to test their software for a broad range of environments at all. Others may lack a systematic software testing strategy, relying on ad hoc tests and suffering shortcomings in software quality as a result.

Yet while the dizzying number of different combinations of device types, operating systems and browsers may seem impossible to manage, the fact is that modern software testing solutions make it feasible for software delivery teams to tame this challenge. While testing software for every possible environment is not realistic for most organizations, it is eminently possible to develop a software testing strategy that offers broad coverage but is also practical to implement and manage.

This guide explains how to identify which types of environments to prioritize as part of a software testing strategy, as well as how to implement a testing routine that provides testing coverage that is as broad as your business needs.

What Is a Software Environment?

When we mention software environments, we mean the type of environment in which an application might run. A variety of factors determine what a given end-user’s software environment looks like and how it causes an application to behave.

The three main factors in shaping a software environment are:

1. The make and model of hardware device that hosts the application. The hardware device could be a PC-compatible laptop, a Samsung mobile phone, an iPad, a smart watch or something else. As noted above, the number of different types of Android-based phones alone is in the tens of thousands, which provides a sense of how many types of distinct devices exist.
2. The type and version of operating system that hosts the application, such as Windows, Linux, Android or iOS. While the number of distinct operating systems in widespread use today is limited, the versions of the different operating systems are numerous. Android has seen sixteen major version releases, for example, as well as several point updates within each version category. There are hundreds of different Linux distributions.
3. If the application runs in a web browser, the type and version of the browser that hosts it. As with operating systems, there are a relatively small number of major browsers (Chrome, Firefox and Windows Edge are the main options today), but each browser family contains many distinct versions.

It is worth noting that other variables can impact application behavior, too, such as the version of the various software libraries that an application depends on and language and region settings on the host device. Your business may or may not choose to incorporate these variables into its testing strategy, depending on whether they are important for your target market or not. If you only release software for English-speaking users in North America, for example, testing for different language and region settings is not likely to be a priority.

Deciding Which Environments to Prioritize

Given that hundreds of thousands of possible device/operating system/browser combinations exist, it is not realistic to attempt to test your applications for every possible environment. Instead, your software testing team should identify the types of environments that are most important for meeting your business needs, based on the following factors.

Region

The market share of various devices, operating systems and browsers varies widely between different regions. iPhones are more popular in North America than in the world as a whole, for example.

Identifying which types of environments are most common among your target users based on the regions in which they live is therefore a useful first step in deciding which environments to prioritize for testing. The “Mobile Reference Testing Guide,” which offers data about environment popularity across different regions, is a useful resource for this task.

Types of Applications

Another obvious factor to consider is the types of applications you plan to build. If you only build native mobile apps that don’t depend on a web browser, for example, you eliminate the need to test for different browser versions—although you may also limit the portability of your application, because applications that depend on functionality that is native to a specific device are harder to migrate to other devices than those that run in a web browser.

Striking the right balance between application functionality and ease of testing is thus another useful step in planning a software testing strategy.

Types of Devices

Do you build software for all types of devices? Or do you target, for example, only mobile devices and PCs? How might your device support needs change in the future? Obviously, there is no need to test for devices that you don’t currently support.

That said, it is wise to future-proof your testing strategy by ensuring that you can easily extend testing support to more devices if needed in the future. Testing for as many types of devices as possible can also provide business benefits by enabling an omnichannel presence, as explained in the following section.

Maximizing Environment Coverage

While it will be necessary to limit the number of environments that you support based on the factors described above, this does not mean that you should not strive to test for as large a combination of devices, operating systems and browsers as possible.

The benefits of broad test coverage for your business are many:

• Broad testing enables you to achieve an omnichannel presence by releasing software for multiple types of devices. Limiting yourself to only mobile or only PC devices, for instance, prevents you from being able to engage users across all of the devices they own. Omnichannel engagement is particularly important given that the strong majority of consumers now use multiple types of devices on a regular basis.
• By supporting more environments, you can support more users. Although your focus today may be only on certain demographics or regions, you don’t want to prevent your business from expanding to other groups because of a lack of broad software test coverage.
• Software deployment techniques and technologies change quickly. You may deploy your application today using virtual machines, but switch to containers in the future. Or your software delivery process may become more complex and target both types of technologies. You want to ensure that your testing strategy can accommodate change such as this.

Achieving Broad Coverage

It may not seem feasible to test applications for a broad range of environments, especially if your business has limited IT resources. By taking the right approach, however, it is quite possible to achieve broad environment coverage without overextending your testing team.

A test strategy that enables broad coverage in an efficient way is defined by three main factors:

1. A scalable, low-maintenance, cloud-based testing infrastructure. If your ability to run more tests or test for additional types of environments is limited by the size of your on-premises infrastructure, you will not be able to achieve broader coverage or the business benefits that come with it.
2. Automated testing. Compared to manual tests, automated tests take much less time to execute. They are also much more scalable because they can run in parallel, and can be easily adapted to support new environments. Automated testing is key to an efficient, flexible testing strategy.
3. The ability to test on real devices as well as emulators or simulators. Testing applications in emulated or simulated software environments is typically faster than testing on actual devices. However, in order to obtain the most accurate results, it is sometimes necessary to test on actual devices. A software testing strategy that is both broad and deep should therefore include the ability to test on both types of environments, depending on specific needs.

Conclusion

A certain amount of picking and choosing will always be necessary when deciding which types of software environments to include in your testing strategy.

However, by designing a testing process that makes use of cloud-based test infrastructure and automated testing, it is possible to achieve broader test coverage—and, by extension, to maximize your business’s ability to support new and existing users no matter which types of devices, operating systems and web browsers they might use.

You can, in other words, have the best of both worlds: cost-efficient software testing as well as broad environment coverage that not only supports your business’s needs today, but can also scale as your business grows.