Benefits of Automated Testing

Kick off a test at any time to run in the cloud - regardless of your location in the world. When Apple and Google release a new browser version, upgrade a cloud server and kick off the test seamlessly. No more having a tester disappear into a conference room and come out in three or four days, proclaiming it is "probably okay."

Once a test script exists, it can be rerun on different form factors and devices at the touch of a button. For that matter, if you assemble the test as components, or building blocks, you can reuse those components on other tests. That means only writing login once, and re-using it. When your application adds a new drop-down for "client type" (person or business), you can change the login function in one place, and suddenly all the tests pass again. The more detailed the human test scripts, the more they will either go out of date or require expensive maintenance work to keep current.

In the past, a passing automation run meant it was possible for the software to run through at least one specific scenario. The computer was incredibly bad at realizing that a change made the screen "look funny" or created a usability problem. In the worst of cases, an entire form field could be hidden, or locked from receiving keystrokes, yet the automation could find a way to type into it anyway. In those cases, the passing test did not even guarantee that it was possible to run through the scenario!

Modern test automation tools provide video playback of not only failing, but even passing test runs. Reviewing these play-by-plays, perhaps at 2x or faster speed, can combine the human power to notice problems with the computer's ability to do the exact same thing over and over again. Having a video of a failure can shorten debugging time from hours to seconds.

Once you've created a test that can run in the iOS platform, you can scale that out of thousands of combinations of device, operating system and browser. Android’s fragmented ecosystem offers hundreds of thousands of choices. If the application is a straight web application that can also run on a laptop, there are millions of combinations. You do not have run these all at once. With a separate test run for each commit, you are probably running a dozen a day. Create a list of the hundred most popular combinations and rotate through the test runs, achieving reasonable platform coverage for free. Or select the ten most important tests and run them on a hundred devices overnight, for not much more in terms of cost. Better still, cloud-based test performance means you no longer need to choose between some minimal amount of hardware that takes a long time for a test run versus building out a test lab at cost of hundreds of thousands of dollars.

The cloud, plus the repeatability of the tests, means you can run the exact same test on multiple platforms at the same time, looking for minute differences. This can provide insight into programming practices and even performance. Consider the difference in knowing time for a test run on an old iPhone, along with average page response, compared to the newest version.

It is very possible that two developers, working at the same time, introduce changes that break the software -- or introduce a merge problem that breaks the software. This is seriously broken; "unable to login" broken. The human tester finds the problem a few hours after it is introduced, and is told that "shouldn't happen", or waits for a new build. The next business day the tester gets in with the developers over the failure, who say it "works in the development environment," or perhaps drag in operations. Finally, on the fourth day, the issue is resolved.

If your experience is better than this, you are more fortunate than most. Or, perhaps, breaking login is rare, yet it is common to find problems in test, or at least the unanticipated consequences of a change.

Having a regression test suite run on every build finds the problem early, identifies what change made the problem, and points the problem right back to the developer who created it. That prevents waste, delay, finger-pointing, and drastically reduces the time spent debugging and fixing.

Most teams leading with human testing  have a "regression test", or "final check before release" test cycle. This cycle is expensive; it might take a week. With a long test cycle, it will be hard to ship often. Over the years, I have twice worked with teams that shifted from monthly releases to quarterly releases as a form of "process improvement." 

Moving to less frequent releases allows the team to spend less time as a percentage writing new code. However, the amount of change come regression time will be greater, which will require more regression test cycles to release. Those more cycles make testing look more expensive, leading to pressure to ship less often. In economic terms, this is a vicious cycle.

Tooling can drastically change those numbers. Combined with modern tactics in software engineering, like multiple deploy points, resilience, and quick time to recovery, the team can create a virtuous cycle, where more frequent deploys lead to less change deployed, less testing needed, and faster value for customer.

Once a "read-only" test script exists you can flip it around and use it to monitor production. Add altering and you can find out when features go down, or if a configuration flag is mis-applied within seconds. Add timing to that to get performance monitoring of production -- true, end-to-end, actually-customer-experience performance monitoring - for about the cost of a busy lunch hour. Again, add alerts when a page load exceeds a reasonable threshold to find and fix problems immediately.

As an application ages, it tends to add logic and database size, and performance decreases over time. It doesn't take much more to add timing to notice differences in performance. That means you can identify features that are trending toward problematic performance before the customers even even have a problem!

Again, start with a test suite of independent tests that mimic the user. Then run them simultaneously at scale for insight into performance. Insights here are two-fold. First, you can look at the performance counters that you wrote for the production monitoring above. Or, have humans explore the functionality in an environment very much like production that is under a similar load.

If a team is adding new code every sprint, then the "space" to "cover" with tests will grow every sprint. With automated tooling, that work is relatively flat: Code a story, code some tests. With human testing, the amount of space to cover grows linearly. In sprint one, test 10 stories, in sprint two, test 20, in sprint three, test 30. Testing either needs to become ever more advanced random sampling, or else testing time needs to grow. 

Done well, test tooling leads to reduced test cost over time. 

[Authors note: Yes, this is a complex discussion that needs a great deal of nuance, but that's okay, I literally edited the book on the topic.]

Sadly, there is usually always a downside to everything, testing styles included. 

The best tooling takes some small subset of what the users might do—the most frequent or important actions—and institutionalizes them to run over and over again, exactly the same. Trying to cover all the possible uses and combinations is a fool's errand. That leaves plenty of room for minor bugs, and also room for things that bug the customer. Remember, the computer is testing for compliance to specification; it is always possible to the specification is wrong, or out of date. Most companies use automated testing to find bugs in simple, frequent, core operations, e.g. to log into the app, create a new account or send email when the password gets forgotten. That’s what automated tests do.

App crashes in specific scenarios must still be tested manually. As we know, machines are very advanced, but the applications for "true" Artificial Intelligence are limited and expensive, at least for now.

One other thing that automated testing won’t do is test the effective usability of the design, e.g. where the buttons are positioned, and how easy it is to actually use the app. That still has to be done by manual user-friendly tests.

In conclusion there are benefits and drawbacks to both automated and manual testing. Some would argue the distinction is helpful, as plenty of manual work is computer-assisted. Sauce Labs believes that independent test execution and evaluation by computers is different, and should be treated differently, than when a human is in the driver's seat. This article had the purpose to show how automated testing can help you to make your testing more efficient.

To achieve the best results you will need a combination of both types: automated testing for repetitive, simple use-cases; and manual testing for reproducing specific bugs, complicated use cases and to ensure the best user experience possible.

Through the solutions that we offer, we are positive that we can help you find one that satisfies your needs. At Sauce Labs, we have experienced solution architects and testing engineers who can work with you to develop a suitable automation strategy, and implement the test scripts for you. Contact us for more information.