Posted Apr 1, 2021
Here’s a look at the top five such frameworks, with tips on the pros and cons of each.
Not to be confused with Selenium WebDriver, which supports only browser testing, WebdriverIO is a test automation framework for both browser and native mobile testing. It’s highly extensible and can automate tests for a wide variety of apps running in any mainstream browser or mobile OS.
WebdriverIO’s ability to support a broad set of testing needs makes it a great framework if you develop multiple types of apps and want a single framework for automatically testing them all.
WebdriverIO uses Puppeteer out-of-the-box. It also comes with integration to Mocha, Jasmine and Cucumber and the whole setup requires just a couple of npm commands.
The main limitations of Cypress include limited support for testing mobile apps: it can handle mobile apps that run natively in a browser, but not other types. It also currently only supports three types of browsers: Chrome, Edge, and Firefox.
TestCafe also supports all of the major desktop browsers, including Chrome, Internet Explorer, Edge, and Safari. It has some limited support for browser-based mobile apps, but in general it is not designed for mobile testing.
A drawback of TestCafe is that it only supports client-side apps. It’s also designed primarily to be an end-to-end and regression testing framework, and does not work well for unit testing or integration testing.
As of right now, perhaps the biggest drawback of Playwright is that, because the framework is relatively new, it is still evolving, and it doesn’t offer as many integrations as more mature frameworks. The lack of Internet Explorer support is also a limitation, as is the fact that Playwright requires a custom browser that must be downloaded each time you want to run a test. This adds time to testing routines, and also means that you can’t test in the actual browser that your users are using. Finally, like Puppeteer, Playwright needs to be integrated manually with test frameworks like Mocha, Jasmine or Jest.
If you like Playwright but wish it were more stable and mature, you should consider Puppeteer, which has been around in stable form since 2018.
The main limitation of Puppeteer is that it only fully supports Chrome and Chromium, so it’s not a true cross-browser testing solution. That said, the Firefox nightly builds are currently compatible with Puppeteer as well, thanks to collaboration between the Puppeteer team and Mozilla developers. Puppeteer support for stable versions of Firefox is in the works.
In sum: Puppeteer is the go-to test automation framework for developers who only need to support Chrome and Chromium. For cross-browser tests, frameworks like Playwright provide a similar experience.
How many types of apps and configurations do you need to support? Frameworks like WebdriverIO are the most extensible.
Do you need to test mobile as well as desktop apps? If so, frameworks that broadly support both types of apps, like WebdriverIO, are a good choice.
How robust and mature do you want the framework to be? Frameworks like Puppeteer, TestCafe, and WebdriverIO are tried-and-true.
Do you need a framework that can handle front-end as well as back-end testing, like Cypress?
How quick and easy do you want it to be to write, configure, and run tests? Currently, frameworks like Puppeteer and Playwright require some relatively complex configuration, especially if you are not using them for the narrow use cases that they support out-of-the-box.
Chris Tozzi has worked as a journalist and Linux systems administrator. He has particular interests in open source, agile infrastructure, and networking. He is Senior Editor of content and a DevOps Analyst at Fixate IO. His latest book, For Fun and Profit: A History of the Free and Open Source Software Revolution, was published in 2017.