Faster Regression Testing With In-Sprint Automation

Searching for User Interface test automation on Google will yield a lot of results for tools, but not much in the way of good practice. Search for experiences with GUI test automation and you may get some code (in the wrong programming language) or you get some … advice. The advice is lately to include a handful of UI automation stereotypes -- UI automation is unstable, UI automation is slow, record/playback is brittle, teams will spend more time on maintaining test code than they will actually testing. There is a hint of truth to those, people really do have problems building UI automation, but they don't have to.

This article covers some important aspects of automated website testing -- strengths and weaknesses, building tests, framework selection, and how automation fits into a software development strategy.

The first step, before taking a look at the many options for tools and before building proofs of concept, is to ask the question 'why'.

Arguably the most common reason companies start a UI automation project is because “regression testing takes too long.” Each release, they have a week of development on new features, then new code is handed off to testers, and then there is a week of testing and bug fixes on that new work. When everything is done, someone still has to do some investigation hoping to discover whether or not all the recent code churn introduced any surprising problems. This last part, the regression testing, either takes too long or is crunched for time. This is where someone gets an idea that all that hands-on regression testing could be automated and done instantly.

The end result of this is usually a halfway build suite of tests that don't really return much information. And, even less time because that group of people that was doing regression testing is now performing regression testing and building automation at the same time. The truth is that regression testing is taking too much time because the development flow is bad.

That is the problem to fix.

Here is an alternate strategy for UI automation; use it to propel features forward into production faster.

UI automation is typically developed on a lag of at least one sprint. The test code being written right now might be for a code change that has been in production for weeks or months. At best, this will only tell you when something on the page changes. Why not build automation in-sprint?

In-sprint automation looks something like this. Developers write their new features sprinkling in unit and API testing where appropriate. Once the UI is close to done, they should be working more closely with a tester. That tester can begin building the bones of an automated UI test. They might write a couple of lines of code to navigate to a page, and start populating some text fields and have to stop to investigate a field that isn't accepting an input 20 characters long. The developer would take a look at that point, fix a bug, and restart their local server. Then the tester would get back to building the test in that same write a line or two, stop to investigate something interesting and then start again.

By the time the important coverage for that feature is built, the new code change has been explored and had rounds of bug fixing. When the change is ready to be committed, there are unit tests, API tests, and a handful of automated UI tests. The feature is basically ready for production. This also has the side effect of preventing too much UI automation. At some point, the pair can decide that they have explored enough and have just enough automated coverage that it is time to move on to the next task. As the consultant, Matthew Heusser famously quipped “The story is not done until the automated tests [for it] run.”

Once the process in place, regression testing really is faster and development flow is a little more smooth. Instead of having to have a hand-off between groups, code comes to regression tested already testing, with checks that are passing in selenium for that feature. Add a little exploration and a green bar in the Continuous Integration system dashboard and the team can ship code for the sprint. The coverage problem also gets much easier. Instead of a person performing similar tests in 5 different web browsers, a script can run in all of the browsers in parallel while a person looks for far more important problems.

This strategy and approach are crucial for teams that want to deliver software in a cadence that approaches Continuous Delivery.

There are a number of APIs and tools available to help people build UI automation. One easy way to categorize them is whether the person building tests records steps they perform in a browser and later adds assertions, or if they are writing code.

Telerik Test Studio and HP Unified Functional Testing (UFT) are popular record and playback tools. Using tools like these, a non-technical can work mostly on their own to create tests. They can create object libraries to refer to page elements, reusable functions to call when some series of steps have to be performed repeatedly. These are usually extensible with either a proprietary programming language, or something like C#. It can be difficult to create stable tests using record and play back tools.

Watir and Selenium WebDriver are popular for testers that are willing to write some code. WebDriver is overwhelmingly popular in terms of usage. Programming tools like these allow testers to create completely custom UI automation frameworks. APIs also allow test developers to take advantage of design patterns like the Page Object. The Page Object allows people to design reusable pieces of code based on pages in the software they are testing. This creates a scenario where code can be made more stable. Tests using Page Objects usually contain less code and are much easier to maintain.

Any of the tools mentioned above are fine, a team can end up with useful test automation regardless of whether they choose the record and playback tool, or the programming API. Which you choose will depend on the staff available. Record and playback tools work well with non-technical testers. Though, it is important to remember that no matter how much a tool claims to be code-free, some programming will eventually be required. Also, having a deep understanding of both testing, and automation is still required.

The coded API options, such as Selenium WebDriver, offer far more flexibility and are generally easier to hook into Continuous Integration systems, or delivery process like Continuous Delivery, but they require testers that are not afraid of learning some amount of basic programming skill.

The first step to build a test for a user administration page using WebDriver would be to build the Page Object. This class should contain methods such as enterFirstName(), enterLastName(), setBirthDate, deleteFirstName(), clickSubmitBtn() and so on. Each of these methods is completely self-contained. for example, enterFirstName() would check that the element exists, that it is visible and ready to be manipulated, and then enter text.

The first test for that user administration page might look like this:enterFirstName("Justin")enterLastName("Rohrman")setBirthDate(03/02/1981)clickSubmitBtn()

Those steps would be followed by assertions based on what you are expecting the page to return.

There is a fine line in automated UI testing between a shallow test suite that doesn't help you learn about quality in the important parts of your product, and suites that are overkill and cross that threshold of requiring constant care and feeding. (The more checking you do, the more maintenance the test will need.) One single user creation test for an administration page probably isn't enough. Automating every test idea that enters your head is probably too much. An approach in the middle, Create Update, Delete (CRUD), might make more sense.

This test would do something like the following:

This sounds like a complex scenario for web testing, and it potentially is. Good architecture and the use of design patterns like the Page Object can make this both useful and reliable.

If there were a downfall to the Page Object, it would be that a lot of tests will fail when an element on the page changes. That problem is true regardless of which development pattern is used. The strength of the Page Object pattern is that the fix only needs to go in only one place -- the page object -- instead of every test. The combination of the right design pattern, and a good API mean you only have to change code in one place to get things moving again. This is important considering it is in the nature of software to change.

There are as many ways to get UI automation right as there are ways to mess it up. Start by understanding the team’s goals: what problems is the development team having and how can UI automation smooth those. Next, build a strategy to fit those goals: how much UI automation is needed, what places need coverage, and where should the team start first. Then, select the right tool, a design pattern that works with tool, and start building tests. After that, the game is coming up with good test ideas, converting those to code, knowing when to change the strategy ... and when to stop making new tests.