One of the interesting things about automation is that it frees you up from time-intensive manual testing, allowing you to spend time on strategic elements—because if you do not spend time on strategy, your capabilities as a team will not grow. And part of that growth means focusing on valuable metrics— metrics that will help you learn, and improve your processes. Once you have processes in place, the next crucial step is to invest in automation. Automation helps you work faster, and makes your work consistent, traceable, and shareable, which is also imperative. All this comes only after establishing the right KPIs (key performance indicators). Automation: Deliver Faster, from Months to Minutes Ask yourself this question: without CI (continuous integration), how long would it take your organization to deploy a change that involves just one line of code? For instance, say your organization sets an objective to deploy a change in production within 30 minutes. To achieve this objective, everyone has to agree on the tools and processes that are needed for an easy button approach (aka continuous integration). Let's review the roles, team responsibilities, and the CI process.
The product manager is responsible for defining product features and providing feature acceptance scenarios. The QA team has two defined roles: QA lead and automation engineer. The QA lead is focused on management of the engineers, and oversight of all automation engineers' deliverables. The automation engineer is focused on writing and testing acceptance criteria scripts. Developers are responsible for accepting feature request(s), writing feature code, and taking on some test script responsibilities. The easy button process involves 6 milestones, beginning with an open feature request, then refining the feature request, development ready, in development, code complete, and finally, continuous integration.
The creation of this easy button process, and the disciplines and tools specified within, are driven by the need to "unclog bottlenecks” after code is complete. Implementation of automated processes is critical in order to enable teams to release code continuously, with minimal dependence on each other or external support teams. This will eliminate bottlenecks, and pushing risky releases that require an army of people to release and test manually. A lot of people hear "process" and think "waterfall." That assumption doesn’t apply in this case. By crafting a methodology to continually release code, we facilitate a high degree of flexibility, and allow product and application development teams to quickly react to market demands. Once in practice, the "process" itself fades into the background. The discipline of doing the things that feed into the process becomes the most important: creating good requirements, writing quality acceptance tests, writing tests in parallel with development, developing with feature flags, etc. When you do each of these things well, the "process" will happen almost automatically. It is an intricate machine, and each part must function properly for the whole process to work. To make continuous integration or continuous deployment a reality, we have to look carefully at what is working and what is not working.
Measuring speed: What are KPIs, and why are they so important? KPIs are the backbone of a development project. They are used to determine if a project is on the right track to success (think: health check), and identify where improvements need to be made to meet goals. Everyone wants to implement a CI environment these days. An established CI environment is where the benefits of KPIs will help measure how every code commit is doing. Statistics focus people, help measure success, and identify where improvements are needed. What do you measure? What makes a successful KPI? You first need to understand who your target audience is before you can craft a clear list of metrics to capture. Let’s start by defining a list of possible metrics for developers:
Number of builds PASSED or FAILED
For a FAILED build, identify WHO broke the build, and send notifications instantly to the individual or group
Measure of code coverage. An instant understanding of code coverage every time something new is committed without any tests. If the coverage goes down, something is wrong.
Tests run before code merge. Testing code before and after it is merged will allow you to decrease the amount of times a master build is broken.
How will developers benefit from this? These measurements will increase the developer's confidence, credibility, create more accountability, and provide a clear representation of code quality. How does test automation benefit from KPIs? It will identify the project’s health status, and indicate areas in your test automation that may require improvements. For example:
Capture the automation execution time per code commit. This will allow you to evaluate whether your environment is scaled accordingly.
Quarantine tests that have random success. Stop flaky tests by budgeting time to fix them. (This part is IMPORTANT: Understand the nature of flakiness before fixing the test.)
Capture test automation PASS-FAIL rates per code commit. This sends a notification to automation engineers, and lets them know who committed feature code, and which test failed.
Capture the CI duration time, from code commit to ready for stage and production. This identifies whether the goal of “x” minutes to deploy to stage or production is being met. Recalibrate the CI environment to build faster by splitting tests and build processes into different instances, so everything will finish even more quickly.
Build small and lean automation tests. Capture and alert automation engineers when individual tests exceed “x” execution time allowed (i.e. a goal of 5 minute maximum). Take the opportunity to investigate why an individual test is taking too long to execute and decide to refactor the test, if needed.
KPIs are indicators, not end-states. They indicate where strong and weak points exist, and should be used to call attention, or used as monitors against a baseline. They are not themselves a goal or a solution. Lastly, a reminder: any measurement is better than no measurement, but a good metric is clear—not subjective. An appropriate metric is discrete and means something to the project. Build your own standards for tracking your KPI dashboard. Make sure dashboards are assembled for everyone on your project. The key to all of this is to automate the creation of KPI information, NOT to manually create any type of metrics produced from a CI environment. Greg Sypolt (@gregsypolt) is a senior engineer at Gannett and co-founder of Quality Element. He is a passionate automation engineer seeking to optimize software development quality, coaching team members how to write great automation scripts, and helping testing community become better testers. Greg has spent most of his career working on software quality - concentrating on web browsers, APIs, and mobile. For the past 5 years he has focused on the creation and deployment of automated test strategies, frameworks, tools, and platforms.
