If you’re like most app development and QA teams, you’ll discover that adopting automated testing requires clearing some hurdles. This is because, like most innovations, test automation creates some challenges that organizations need to overcome before they can reap the full benefits of an automated testing strategy.
1. Lack of Automated Testing Expertise
The most common pain point that QA teams tend to experience when they move to an automation-based testing strategy is a lack of expertise with test automation.
This challenge is to be expected. If your QA engineers were previously accustomed to performing testing by hand, they likely lack experience in writing automated test scripts or using automated testing frameworks, like Selenium and Appium.
The obvious solution to this challenge is to educate your QA engineers, of course. But you don’t have to limit your approach to forcing your QA team to take tutorials on writing test scripts or pore over Selenium documentation.
You might also think about how to leverage the coding skills that your organization already possesses to help your QA team learn to write automated tests faster and more effectively. Ask your developers to assist in the automated testing learning process. The coders you have on staff may not have backgrounds in QA engineering, but they should be able to wrap their heads easily around writing test scripts and learning a new scripting framework. By involving developers in the automated-testing migration process, you can speed your QA engineers’ ability to gain the skills they need in order to take full advantage of automated testing.
2. Lack of Automated Testing Infrastructure
Another common challenge that organizations face when moving to automated testing is a lack of infrastructure for executing their tests.
To solve this problem, you have two options. The first (and the one to which you might be most tempted to turn) is to build your own automated testing environments by setting up software like Selenium server on your own on-premise infrastructure.
That approach works, but it’s less than ideal, for two reasons: First, setting up automated testing software will take time, especially if your team doesn’t have prior experience configuring these tools. Second, an on-premises test infrastructure is difficult to scale. If you plan to increase the number of automated tests that you run, you’ll have to add more infrastructure, which will be costly and take time.
The other, better approach is to use a cloud-based testing solution, where the environment is already set up to let you run tests using any framework you choose. An automated testing cloud is also highly scalable; you can use as much infrastructure as you need to run as many tests as you want at the same time.
3. Not Knowing Which Test Automation Framework to Use
The market offers a plethora of test automation frameworks to choose from, and the decision isn’t always easy. As you evaluate mobile testing frameworks to determine what's suitable for your organization, you should ask some key questions to help uncover your needs and understand the impact of a new tool on your organizational processes, tech stack, and teams.
The following are popular test automation frameworks:
Selenium - Selenium is the industry standard in open-source automated testing framework for web applications. Selenium WebDriver is a great starting point for teams moving from manual testing to automated testing because of its breadth of coverage with client drivers. Selenium WebDriver is essentially language agnostic and allows for automation testing in languages such as JavaScript, Java, C#, Ruby, and Python, among others.
Appium - Appium is a leading open-source test framework that allows for cross-platform (mobile) native test automation, as it supports both iOS and Android platforms. It was derived from Selenium to extend automated testing functionality to mobile apps.
XCUITest - XCUITest is the testing framework that ships with Apple’s XCode development system, and as such, it is the most popular framework for performing basic unit-level testing of iOS apps.
Espresso - Espresso is the testing framework that comes built into Android Studio and is designed specifically for functional testing of Android applications.
WebdriverIO - WebdriverIO (not to be confused with Selenium WebDriver, which supports only browser testing) is a JavaScript test automation framework for both browser and native mobile testing. WebdriverIO is best for organizations that have extensive and broad testing needs that extend beyond just basic JavaScript tests.
Cypress - Cypress is a JavaScript testing framework that can perform both front-end and (to a certain extent) backend testing. Cypress is a good choice for developers who use JavaScript (or Node) for both server-side and client-side development.
TestCafe - TestCafe offers the advantage of supporting both JavaScript and TypeScript as the languages for writing testing scripts. TestCafe is an excellent choice for testing any desktop or mobile JavaScript apps, provided they are client-side only. If you need server-side testing support or mobile testing, look elsewhere.
Playwright - Playwright, released by Microsoft in January 2020, is open source, and is one of the newest JavaScript-based test automation frameworks available. Playwright is an attractive choice for developer-friendly, cross-browser testing that supports multiple languages, including but not limited to JavaScript. But if you need to test for Internet Explorer, you’ll need a different framework. You may also want to wait a bit for Playwright to mature further if you are worried about having to handle rapidly changing APIs.
Puppeteer - Puppeteer is a great alternative to the newer and less stable Playwright. Puppeteer is the go-to test automation framework for developers who only need to support Chrome and Chromium. For cross-browser tests, frameworks like Playwright provide a similar experience.
Detox - Detox is an open-source JavaScript-based test framework for React Native applications. It works cross-platform, accepts tests written in JavaScript, and works well with emulators and simulators. Detox is designed for gray box testing, allowing monitoring of the app from the inside, and is known to reduce test flakiness as it is well synchronized with the app’s activity.
EarlGrey - EarlGrey is an open-source iOS UI automation framework developed by Google. It supports only iOS and hence requires tests to be written in Objective-C or Swift. Some benefits of EarlGrey include easy integration into iOS, simplicity in writing tests, and enhanced built-in synchronization features which help to increase test stability.
4. Deciding Which Tests to Automate
Not all tests should be automated, but most unit, integration, and performance testing can be easily automated. Consider the following criteria when deciding which tests to automate:
High value and business critical: Tests that ensure your app keeps running smoothly and uninterrupted, such as smoke tests, sanity tests, and regression tests.
High volume, repeatable, and/or tedious: Tests scripts that are time consuming or need to be repeated many times, such as testing different browser/OS combinations and running high-volume batch tests.
Predictable results: Tests that are both repeatable and have predictable outcomes, such as stress tests and load tests.
Frequently used functionality: Tests frequently used functionality that comes with high-risk conditions for both the device and sensitive user data, such as loss of functionality (e.g., loss of data connection), unexpected high data usage, battery draining 2-3 times faster than normal, and security or privacy exposure.
Impossible to perform manually: Tests that can’t be performed manually. For example, a manual test may return an issue that you’re unable to diagnose. Automated tests can not only identify an issue but also indicate the cause of the issue.
Test data and environments: Tests that automate tasks like setting up environments and test data.
Your mileage will vary, of course, but in general, these pointers are a good place to start when you’re deciding which tests are the best candidates for automation.
About the Author
Chris Riley (@HoardingInfo) is a technologist who has spent over 15 years helping organizations transition from traditional development practices to a modern set of culture, processes and tooling. In addition to being an industry analyst, he is a regular author, speaker, and evangelist in the areas of DevOps, BigData, and IT. Chris believes the biggest challenges faced in the tech market are not tools, but rather people and planning.
