If your test automation team’s directive is to automate X amount of tests, and you have no strategy as to which tests they should focus on, you are wasting your time. Before you begin writing your first line of automation code, make sure you have a strategy in place. Otherwise, you will have a ton of ineffective tests to maintain.
Don’t Choose a Random Goal
How many times have you been told that the goal of the team is to have X amount of test coverage? This is an arbitrary value picked out of the sky. What is it based on? If a UI automation team were to cover 80% of the stories in a sprint, they would never get done in time. We all know how fragile UI automation is! How many times will a designer make a change that directly affects the UI and breaks the test? This is almost manageable during a sprint while you are working closely together, but how about when the product is sent to be translated to another language? The translator inevitably comes back with suggestions to allow for phrases more common and translatable. Bugs might be entered and UI changes made by a maintenance team with no heads up to the automation team, and Bam! — You have broken tests that need to be investigated.
Not only is a random goal meaningless, it’s hard to track, and a pain when you have to prove you’ve met your goals. A test automation strategy should aim for the spirit of a goal, but should not be strictly enforced to a value.
The Old ‘80-20’ Rule Strategy
Most of you have heard about the Pareto Principle. In software, per Lowell Arthur, “20 percent of code has 80 percent of the related errors.” Dr. Joseph Juran, a long time quality evangelist, recoined this to, “the vital few and the trivial many”, which he later changed to the “useful many” so as not to discount the remaining 20 percent. A QA team that needs to optimize their ability to discover bugs with the fewest amount of resources should take heed to this. The more you automate, the more you have to maintain. If you’ve ever automated, you know that is the true bear. Use your bug tracking system to show trends in bugs over X amount of time. You should be able to identify the riskiest areas of your code. Keep digging and you might find the few areas that cause most of your bugs. For the 80-20 Strategy, this is where you should focus first.
The “Focus on Customer-Reported Bugs” Strategy
Most companies considering automation already have a manual process in place. It’s probably pretty effective, or the company wouldn’t still be in business. This means they are also effectively tracking both internally and externally client-reported bugs. How many times have you heard something has been fixed, and later it recurs? Is that not frustrating? Customers can accept that bugs happen, but to have a bug come back really makes them question your quality standards. A strategy to focus on customer-reported bugs will prevent this gaffe. This strategy is simple. Every customer-reported bug MUST have automation built around the fix. If you did a proper root cause analysis you should know the best place to apply the automation. Every client-reported bug that is fixed should have a process in place to ensure this occurs:
Have a means to identify this type of bug in the bug ticket.
Include the Root Cause Analysis results in the ticket, plus the proposed prevention solution.
Based on where the automated test is enforced, include validation of this in the peer review checklist.
Monitor for future related issues.
This strategy is simple, and should make for a good starting point for an automation effort just starting out. But it is so powerful that it should be part of every effort.
The CRUD Strategy
This is an extremely simple strategy to implement. You don’t even have to investigate to start. My teams have found this simple strategy to be the most effective in catching bugs early during CI. The CRUD (Create, Read, Update, Delete) strategy is basically what is says. Write simple tests that cover each of these basic functionalities. Applying this strategy will probably cover 80% of all customer activities — bonus! Plus, while it might be redundant, it can be applied across all levels of testing, and is probably the easiest thing to automate. This will allow your automation teams to become familiar with the infrastructure while getting a large bang for the buck.
Always Be Reviewing
I’ve always loved the movie Glengarry Glen Ross, and the “Always Be Closing” scene. You should be applying a similar principle to your automation strategy:
Make it a point to review the effectiveness of your strategy. Have you noticed a downward trend in bugs? Have the tests found any bugs? (Make sure you identify bugs found by your automation.) What percent? Don’t be afraid to discard ineffective tests. They just eat up resources. Always be reviewing! Joe Nolan (@JoeSolobx) is a Mobile QA Team Manager with over 12 years of experience leading multi-nationally located QA teams, and is the founder of the DC Software QA and Testing Meetup.
