Technological advancements and the explosion of devices across multiple platforms means hardware and software developers have a much more difficult job. They have to keep up with the demand to develop and roll out new products. One of the most significant issues is accounting for the differences in system response, when responding to mobile traffic rather than to internet traffic.
Applications must be tested to make certain they run responsively on key platforms and across numerous networks. Effective functional testing eases the pressure on device manufacturers while allowing application developers the time to collect applicable metrics that improve product quality. A New Set of Challenges Testing mobile applications is completely different and significantly more complicated than testing traditional desktop and net applications. Mobile devices don’t just emulate the desktop environment — they have their own set of requirements. Mobile app testing is far trickier because of the following key aspects. Device Variation: Mobile applications run on devices that have different:
Operating Systems (OS) such as iOS, Android, Windows and BB;
Versions of those OS; and
Manufacturers such as Apple, Samsung, Nokia, Motorola and LG.
When an application needs to run on multiple OS, devices, and varied screen sizes, QA teams are faced with the challenge of ensuring the application functions in every environment. The below graphic of the iOS support matrix alone shows the complexity with a single series of devices and OS type. [caption id="attachment_12588" align="aligncenter" width="587"]
source: iossupportmatrix.com
Availability of Mobile Testing Tools: Desktop and web application-based testing tools cannot be used for testing mobile applications, so a new testing framework is required. Some of the frameworks currently available for writing test scripts are:
Network Challenges: You must be able to effectively emulate various bandwidth rates, because your end users will be operating on a variety of networks and bandwidths. You also must be able to test geographically isolated loads accurately to signify real-world traffic. Screen Size and Densities: With the diversity of screen sizes and densities available today, you must also be able to test your mobile app on different screen configurations so that it:
Fits on small screens;
Takes advantage of the additional space on larger screens while still looking good on smaller screens. For example, the difference in screen size between iPhone 4 and iPhone 6+;
Is optimized for both landscape and portrait orientations.
Peripheral Mode: When testing in mobile, there are physical elements that also need to be considered, unlike with web applications. You may need to consider wireless and wired peripheral mode testing.
Wireless to the device such as near field communication (NFC), Bluetooth or stylus; and
Wired internal to the device such as a headphone jack or external testing with cc reader or bar code scanner;
Native Applications and Browser-Based Applications.
These are the variables that come with a mix of native and browser-based applications. Native applications reside on the user’s device and communicate over HTTP(s). The browser-based application uses a modified version of a browser to access applications online. As many companies use both applications to offer solutions to their customers, you must support testing of all types of mobile applications.
Taking Challenges on "Strategy First"
What do these testing challenges mean for web developers and site owners? Primarily, for every web application designed, you must also address a strategy to test the product in the mobile space. Building an app with all the features and functionality needed by the client and user is important. Having a rigorous mobile testing plan in place before the mobile app is deployed is even more crucial to its success. Mobile applications are becoming more and more sophisticated, significantly increasing the requirement for functional testing. To tackle this, organizations that require app testing are always exploring alternatives to traditional manual testing. Mobile automation testing is a highly effective approach to mobile app QA. It provides significant business returns when executed using the right tools and infrastructure, while factoring in cross-platform challenges. From sites to web applications to native mobile applications, test automation tools bring full-featured functional testing to mobile platforms.
Strategy for Mobile Test Automation
Automated testing can sometimes be a black hole. The best thing to do is automate something and measure that against a precise objective that can be measured in a realistic timeframe. Knowing your hard or soft Return on Investment (ROI) goals helps as well.
Mobile testing requires a balanced approach, and the key is understanding your company’s mobile strategy.
Target Device Selection: You simply cannot test applications on every device that exists. However, cloud-based device emulation tools allow you to increase that breadth more and more. The best approach is analyzing the market and choosing a representative device that reduces the effort of executing multiple test cases. A few factors to consider are OS version, screen resolution and form factor (smartphone and tablets) while also ensuring the multi-device and multi-platform compatibility of the app.
Emulators: Emulators mimic the software and hardware environments found on actual devices and provide excellent options. Options like the ability to bypass the network and use a real-world environment via modem where actual users run and interact with those applications on their devices. At the same time you will need to test on a physical device, but this can usually be done more ad-hoc for most applications. Find the right mix of emulators and physical devices to provide the best results!
Tool Selection Criteria: Test automation tools create a framework to systemize the testing of mobile native and web apps across platforms. For instance, Appium is one of the few frameworks that can develop cross-platform scripts in multiple scripting languages. Collaboration between iOS and Android developers is crucial to ensuring every element that looks the same has the same accessibility label and builds a testable app. An important step in this process is creating a list of requirements to review when choosing a tool for evaluation. Some questions to ask as you determine your requirements are:
Am I looking for a Behavior Driven Development (BDD) framework?
Do I need to support native, hybrid, mobile first (responsive web design) or mobile web?
Do I need to support diverse mobile platforms such as Android, iOS or Windows?
Am I looking to test locally or in the cloud to reduce the cost of ownership?
Am I looking to test against emulators or real devices?
Do I need a framework that supports cross-platform scripting?
Do I need a framework that offers an easy interface for tests to auto-generate scripts? For instance, scripts can be automated without any programming or scripting language knowledge.
Am I practicing continuous integration and need a tool that integrates into the broader environment seamlessly?
Am I looking for a framework that will attract existing developers to contribute to my automation goals?
Take a step back and consider your resources. Does your QA team have sufficient programming knowledge for automation development? For automation, you must have people with some programming knowledge. If not, do they have the technical capabilities to easily adapt to the new technologies? Answers to these questions will help guide your team to picking the best framework, and also understand the capabilities of the team to execute on testing.
Automation Environment: Automation environment and setup depends on the approach to testing. Automation testing approaches are either cloud-based or local. Cloud-based testing provides web-based automation platforms that can be accessed from anywhere in the world with good internet connectivity. It is one standard way in which one can achieve native and hybrid types of test automation. And the “automation-from-anywhere” feature is a big advantage. You can run scripts from your test framework on most of the cloud solutions. Local-based testing involves setting up tools in a test environment and leveraging either emulators/simulators or physical devices to automate testing using popular open-source tools such as Appium, Espresso, or Kif. The additional consideration with on-premise testing is the breadth you can cover, and the time to set up these environments.
Mobile Testing, The Practice
Though the techniques and tools used to automate mobile application testing are complex, we learn a lot from the days of client/server desktop applications. But a few extra mechanisms are required for mobile automation, and these are:
Using a mobile test framework such as Espresso or Kif makes it easy for developers and testers to write scripts in the native programming language and opens the door for pair programming;
Identify the requirements and categorize them based on the mobile application type—Native, Hybrid or Mobile Web;
Identify the scope and device that meets the requirement;
Identify an automation tool that will a best fit. Filter the best tools by performing Proof of Concept (POC) that would prove that the automation can produce real results;
Design the framework architecture based on the initial requirement.
Test automation strategy runs in parallel with the framework design, which details the technical scope, test environment, and running scripts on emulators/physical device with appropriate automation based approach.
Mobile Device Matrix: In addition to the testing mechanisms, you have to consider the matrix of device type, device OS, and device browsers to test against. The size of this matrix is significantly larger than in web application testing because of the dependency of device types since each have different screen resolution possibilities and different features. For example, the new thumb print authentication in some iPhones is not available in others, but applications can leverage this feature. Templates for building a mobile automation device matrix require:
Operating Systems - customizations, missing libraries, driver issues;
Screen Size - rendering issues, usability, missing layouts;
Pixel Density - density independence, missing layouts;
Aspect Ratio - X,Y calculations, overlapping panels, display issues;
System on a Chip (SoC): hardware performance, instruction set, battery signal;
Carrier: network protocol, speed, responsiveness, packet loss.
Conclusion
To build a successful test mobile automation strategy, all stakeholders must understand the business value of automation. Further, all teams, including development, need to buy in to the process. Picking the right automation tool and building the right testing environment are typically the most difficult challenges when implementing a successful test automation process. However, you do need to focus on the long-term success of your product, so it’s helpful to address these important questions when starting the process:
What are your automation objectives?
Will your current development processes need to change?
What test environment support do you need?
What skills will you need?
Process and Organization + Environment + Technical + Resources + Scope and Roadmap = Test Automation Strategy. Mobile testing tools are relatively new, and developing an automation strategy specific to your organization ensures that you will get the most business value out of your automation tool. A crucial piece to laying the groundwork for mobile app testing is building a well-balance testing portfolio that includes automated unit testing, integration testing, WebView testing, automated UI testing and exploratory manual app testing of the UI. Appium shows great promise in this area. Making the leap into mobile testing today is not easy. The good news is a there are many tools that are being developed that support mobile testing efforts, but the biggest part of the challenge is integrating your testing into your existing environment. The important thing to know is that just because it is hard does not mean you should reduce your efforts compared to your web application. This would result in a disconnect of software quality, and could ultimately destroy your mobile application strategy from within.
Greg Sypolt (@gregsypolt) is a senior engineer at Gannett and co-founder of Quality Element. He is a passionate automation engineer seeking to optimize software development quality, coaching team members how to write great automation scripts, and helping testing community become better testers. Greg has spent most of his career working on software quality - concentrating on web browsers, APIs, and mobile. For the past 5 years he has focused on the creation and deployment of automated test strategies, frameworks, tools, and platforms.
