Achieving Continuous Integration (CI) excellence through the test automation role is evolving, and will be entirely different in the future. At Gannett - USA Today Network, where I work in a quality assurance role, the change has started by blurring the lines between Test Automation and DevOps.
Let’s take a closer look at CI excellence by dividing it into three tiers: the practices, how to do it, and the new test automation experience.
Setting Expectations for CI
Setting expectations and clear goals up front puts you on the right path to a successful model. Some of my recommended goals are:
A dedicated champion of CI in place
All product teams practicing CI
Community ownership
Building a repeatable and disposable infrastructure
Ensure that the team is building quality into every stage of the pipeline
Thinking about shifting left
Publishing changes to everyone
To adapt to changes, we need to be proactive and seek improvement continuously. New technologies and methods are introduced almost every day. Take the time to evaluate new solutions, and don’t be afraid to fail fast and move on. When you do find a new solution, share the experience by demo frequently. Publish new features and changes by writing up blog posts and Confluence pages. Or hold a hackathon to coach team members on it.
As an example, we create self-paced interactive workshops for all new technologies and tools we roll out to our teams. Building confidence naturally happens, and will expand based on sharing the team's success stories with others. This is one reason why I’m sharing our story about CI excellence (and hope you all can share your success stories with me the next time we cross paths).
CI Ownership
In the traditional model, there was NO QA involvement and zero ownership in our process. QA was blind in the model and had no insight, and no team collaboration. Now, as a community, we are sharing the testing and continuous integration responsibilities with developers. This community movement is allowing us to start blurring the lines between Test Automation and DevOps. Some of the daily tasks now handled by Test Automation include developing automated scripts, coaching the team on how to build testing into every stage of the pipeline, sharing and owning CI responsibilities, defining and paving the path for CI standards, helping to script the CI pipeline, and acting as the quality gatekeeper of product releases.
When we introduced this culture change, it was best for Test Automation to champion this movement and be the sole owner of Chef Jenkins development. To prove this new concept, the mobile team allowed us to pilot this new CI model, and now it is the blueprint for all other teams.
Here are examples of community CI responsibilities:
Test Automation: Leading Chef development for Jenkins Master role cookbook; contributor to the Chef Jenkins Workers role cookbook; scripting the Jenkins pipeline file
Developers: Leading Chef development for Jenkins Worker to build, test, and deploy the application
PaaS Site Reliability Engineer: Assistance with Scalr, Chef, recycle instances, and anything to optimize experience
Defining a Single Path to Production
You can build a CI pipeline for any application, whether it’s designed to run on a desktop, mobile device, or server—anything. The approach you take to continuous integration starts by defining a single path to production and outlining the pipeline stages.
source: https://goo.gl/qBWbvx
The infrastructure stage is a critical stage, and our PaaS team does an excellent job of building quality into all the self-service tools rolled out to the teams to ensure they work as expected.
In the pre-commit stage, the developer or test automation engineer executes all the tests in a local dev environment before creating the pull request, and someone on the team reviews the changes before creating the pull request.
The product stage is an important step to outline all the stages, and provides a clear picture of how to define your continuous testing strategy, along with your path to production.
Define a Continuous Testing Strategy
The items listed in the graphic below are best practices for defining a continuous testing strategy. Yet they’re more than just best practices. They’re essential if you want to deliver applications continuously, with confidence.
source: https://goo.gl/Mn9xOd
Design CI Architecture
Continuous integration success starts by automating the entire cloud infrastructure to be disposable. We started down this path to change how we build, test, and deploy products. Now, let's take a few minutes to review our CI architecture in the cloud and some of the latest DevOps tools we’ve used. This new architecture allows us to build and test every PR, and scale up as many Jenkins Workers as needed to build and test hundreds of markets concurrently.
source: https://goo.gl/KtiRMP
We are using Chef to download, install, and configure Jenkins and application dependencies on our servers. Docker is new to our CI architecture, and we will be rolling out the solution later this year. We use Scalr to manage our servers in the cloud. Scalr sits on top of Amazon Web Services and Google Cloud Platform. Scalr allows us to quickly bootstrap our Chef cookbooks as a self-service tool, and control our cost.
Jenkins is our continuous integration tool of choice. The core part of this architecture is made up of repeatable and disposable Jenkins Workers. The Jenkins Workers are configured from our Chef cookbooks, and that allows us to scale up as many Jenkins Workers as needed to build and test hundreds of markets concurrently. In order to manage all of the different packages for mobile, desktop, and anything, we use a tool called Artifactory. Artifactory is designed to be a central hub for all of our packages built from Jenkins. Sauce Labs is our testing service that lets us test, and interacts with the browser. Amazon S3 allows us to store the nightly Jenkins backup and retrieve the backup whenever we need to re-spin a new Jenkins Master server (disposable).
New Relic aggregates logs and events from Chef, GitHub, Amazon, Jenkins, etc. on a rolling timeline, and this information can be handled in many different ways. Code committed and merged to GitHub synchronizes with our continuous integration tool by using webhooks.
This architecture has completely removed the previous bottleneck, and we can build, test, and deploy hundreds of apps in less than 30 minutes.
Setting CI Standards
What happens if you don’t have clearly defined standards? If you don’t, the entire process becomes a mess, and everyone goes in separate directions. When multiple teams are adopting CI, it is critical that everyone is on the same page. It starts with proper documentation and educating the various teams with conversations. Now, let’s review some of our standards, starting with how to create the CI pipeline:
source: https://goo.gl/4Yyx06
Test Automation Engineer Responsibilities
The graphic below will help you visualize the responsibilities for the new Test Automation role, and how to start blurring the lines between Test Automation and DevOps tasks.
source: https://goo.gl/tuCpbE
Conclusion
The vision is clear. It requires a culture change. Like any change, it will result in early adopters, and resistance from non-believers. I believe quality is everyone's responsibility. The test automation team will help strategize and facilitate testing, and teach teams how to incorporate those strategies into their development. The goal of this project, for us, is to come together and define our continuous testing strategy by enabling testing for every stage of the CI/CD pipeline. At Gannett | USA Today Network, Test Automation shares the testing and continuous integration responsibilities with developers. This community movement is allowing us to start blurring the lines between Test Automation and DevOps. The mission is for all teams to be practicing continuous integration and continue to shift left to enable faster feedback from testing and integrating code. Now the team can find problems sooner rather than later. It takes a COMMUNITY to foster this movement to build a repeatable and disposable infrastructure and CI ownership. We are achieving CI excellence with test automation engineers contributing and driving the strategy.
Greg Sypolt (@gregsypolt) is a senior engineer at Gannett and co-founder of Quality Element. The last 5 years focused on creation and deployment of automated test strategies, frameworks, tools, and platforms.
