Is Selenium the Best Test Automation Framework?

As far as web test automation goes, Selenium is certainly the most popular toolset. That popularity is uncontested. A survey on test automation finding that 54% of the respondents were using Selenium might be surprising only in that the number is so low. Many of the alternatives to Selenium, Software-as-a-Service products that record and re-run entirely in the browser, still use the Selenium as the core driving technology. Separating popularity from fitness for use takes a little bit more effort, which is our goal today.

Over the past twenty years, Selenium has also grown from a technology to drive a single browser and execute tests, to a standard protocol for programs to interact with a browser as if they were a human, an in-browser test record/playback tool, a distributed computing grid to run multiple tests at the same time, and more. Selenium is now a full suite of products that combine to create something much more powerful. It is also an open source project which makes very few, if any barriers to use and contribution. In addition, both open source and commercial companies have built frameworks on top of the core open source products, in particular frameworks, our focus today.</p>

The term "framework" is frequently overused in software development and testing. Here's a typical definition for framework, from 2007:

Wikipedia defines a framework as "A basic conceptual structure used to solve a complex issue". It also warns that "This very broad definition has allowed the term to be used as a buzzword".

When I use the term, I mean any support, infrastructure, tool or “scaffolding” designed to make testing easier, and (often) automated.

For today, let's refer to a test case as a piece of isolated code that can run, operates software under test (in our case, a web-browser going to a web page), and produces pass/fail results. Companies tend to build hundreds to thousands of these tests, and want to run them against all kinds of browser and operating system combinations, at various sizes, then get information on what passed/failed, collect metrics on those statistics over time, and so on. For our purposes, a test framework is a tool to organize the running of tests and the reporting of statistics.

The framework may create reports and dashboards, it may integrate with Continuous Integration to provide one single place to call functions.

Put simply, the Framework is the thing calling, managing the execution of, finding meaning in, and reporting information back on the individual test cases.

The core technology of the Selenium project is WebDriver, the technology to drive a browser. WebDriver consists of two components, the "client" code libraries that allow programmers to access the web, and the "browser driver" technology to open Firefox, Edge, or Chrome, and command it to open web pages, click and type. On top of those technologies, programmers create test cases that can run as executables, inside an IDE, or as part of something else.

Selenium Frameworks typically store all those test cases in a hierarchy (such as a set of folders or a class tree in a programming language), synchronize the running of those tests, perhaps interacting with the grid to run them on. That makes switching out the grid a matter of changing the configuring of the framework, in one place - the browsers to run on, which tests to run, the integration with visual testing, and so on.

Some frameworks were created from the beginning to integrate with Selenium, others had it bolted on. If your team is already using a framework that supports Selenium, it may be best to stick with that one, allowing the team to add end-to-end web-browser test automation yet keeping all the tests in the same place.

Choice in Framework does lead to the question: Is Selenium the right automation tool?

As we mentioned earlier, WebDriver actually sends commands over a protocol that is a World Wide Web Consortium (W3C) standard. Every major browser vendor, Microsoft, Google, Mozilla and Apple, support that standard. That means a computer program designed to test chrome and change one setting, from browser from *chrom to *ff, and it will run Firefox.

The tool is also the most popular, which means getting support, training and hiring staff will all be easier than any niche tool. To create tests, developers write code in a programming language, which can be almost any popular programming language. That means the testers can write tests in the same programming language as the production code -- or the programmers can write their own tests.

Writing the tests in the same programming language (or at least one the programmers understand) and having them expressed as text, not "locked" in a testing application in some binary format, and especially keeping them in version control, makes it much more likely that the programmers will consider tests a first-class artifact and keep the tests passing along with the code. This prevents automation delay.

Selenium was initially developed for Functional Test Automation of web applications. Because it can run hundreds of browser / operating system / screen combinations in the cloud, Selenium can also be used for compatibility testing, especially with a tool like Sauce Labs. The Appium project is a sort of second-cousin project extremely similar to selenium that extends the capabilities to test native iOS and Android applications. Sauce Labs can also record and re-use Selenium functional tests as performance/load tests.

The term functional testing can fail to identify changes in the user interface that "work" but now render awkwardly or improperly. Visual testing is a technique to catch those incorrect renders, and, yes, Selenium can support it.

Adding a framework will provide the reporting, timing, perhaps running in parallel capabilities to Selenium that will allow it to do things such as integrate with Continuous Integration and Continuous Delivery. The question is - which framework?

Below we'll discuss some popular Selenium test automation frameworks.

jUnit/xUnit. Perhaps the most popular unit-test framework. The term jUnit can be used both for the specific Java framework but also synonymously with xUnit, where x is a short name of the technology. cppUnit is for C++, pyUnit is the unit test framework for python, and so on. In most cases these run inside of the development environment, so a programmer can make a change, press F5, and see the tests run that tie to that class, project, or the entire suite. Originally xUnit was designed for low-level code, still, many companies find great success in writing tests in xUnit. Here is a code sample in Java/jUnit/Selenium:

class SauceLabsHomePageTest { @Test public void site_header_is_on_home_page() { System.setProperty("webdriver.chrome.driver", "/where/you/put/chromedriver"); WebDriver browser = new ChromeDriver(); browser.get("https://www.saucelabs.com"); WebElement href = browser.findElement(By.xpath("//a[@href='https://accounts.saucelabs.com/']")); assertTrue((href.isDisplayed())); browser.close(); } }

Microsoft. Microsoft Visual Studio has its own set of infrastructure. The two most popular tools are probably MS Test, a commercial tool, and nUnit, which is a free open plugin for Visual Studio. The tools generally run on the Common Language Runtime for .Net, which means any test automation tool can work in any .NET programming language. Programmers that write code in Visual Basic or C# can write tests in F# or PowerShell or any of the twenty-six CLI languages that Visual Studio supports.

TestNG A java-based alternative to jUnit, TestNG can run as a stand-alone or as a plugin for most popular java editors, including Eclipse, IDEA Intellij, and Netbeans. The test community seems to be using TestNG (which stands for "Next Generation") for higher-level tests.</p>

WebDriverIO. Built from the ground up to be a web and mobile automation framework, WebDriver.io works with code written in Node.js, a javascript programming language. Node and Javascript are popular technologies that might be.</p>

Cucumber. A tool to take near-English code written in given / when / then format, Cucumber can support Selenium-Webdriver code libraries. Jeff "Cheezy" Morgan's Cucumber with Cheese may be the standard reference for this approach.</p>

Selenium is open source, vendor-agnostic, has plugins for every popular development environment, every popular programming language, every popular open-source test framework. Selenium also has a series of frameworks that were developed as from-the-beginning, Selenium-native frameworks.

Selenium is also the most popular automated web browser testing tool.

We think there may just be a reason for that.