There is a plethora of CI/CD tools available. They’re all designed to do the same basic thing. Their costs are similar (many are free), and they are all, for the most part, easily accessible.
That begs the question: How do you pick which CI/CD tool to use?
I’ll explain in this post. There are five things to keep in mind when choosing the right continuous integration and delivery (CI/CD) tool for your projects, which I outline below. I believe if you follow these guidelines, it will enable you to find a tool that meets your team’s needs and budget.
1. CI/CD Hosting Requirement
This may or may not be significant to you, but in some cases this might be a mandatory requirement. Which path should you take—self-hosted or cloud-based solutions? I believe there are advantages and disadvantages to both platforms. The critical question to ask yourself before choosing or exploring a CI/CD tool solution is, What is your company policy on hosting your application and data storage? And is it required to be behind a firewall in your own private infrastructure?
If yes, this will narrow down your options, and will require a dedicated team to maintain the infrastructure
If no, then you're in luck, since there are a lot of great options for both on-premises or in the cloud.
In the cloud, they oversee some of the onboarding, maintain the infrastructure, security, maintenance of CI/CD tools, continuous support, and more.
With on-premises, you will need a dedicated team to maintain the infrastructure, manage on-site security needs, and you determine the cadence of maintenance updates. If you manage your private CI/CD infrastructure, it allows full flexibility regarding expansion/customization.
2. Budget
You don't need a massive budget to practice continuous integration, delivery, and deployment. You do need to understand where you are planning to spend your CI/CD funds.
If there are no restrictions, before deciding on the cloud-based solution you need, answer the following questions to determine the plan and cost of the service.
Is your team building more than 10, 20, 30, or more per day?
How many concurrent builds are required to support development team demands?
How many users need access—10, 20, unlimited?
How many days, weeks, or months of data retention is required?
A key thing to consider with restrictions is that self-hosted requires a dedicated DevOps team to build and maintain the CI/CD infrastructure, with an understanding of how to optimize cloud cost when the service is not in use.
There is no silver bullet, and both methods require a budget to run a CI/CD tool. Take the time to understand the cost of both self-hosted and cloud-based. Even open source tools like Jenkins have some level of cost to run and maintain the tool. Time is money!
3. Features
Every great team needs loads of features, the flexibility to customize missing features, and the ability to quickly scale up to handle demand, automatically. CI/CD features are the critical part of the evaluation. You want a tool that makes the build, test, and deployment process adaptable and straightforward. It shouldn't be difficult or complicated. I am not going to say this is a complete list of CI/CD features. I’ve focused here on what I believe are the core features you need, along with a few questions you’ll want to consider to help choose the right CI/CD tool for your project:
LDAP/AD
Pipeline as code such as Jenkins files, YAML files, etc.
Docker and Kubernetes support.
Easily configure your compute and memory needs.
Easy setup—up and running within minutes.
Scale up capacity based on demand for hundreds or thousands of builds at the same time.
Ownership costs vs. someone else managing the infrastructure, such as in a cloud-based CI/CD tool solution.
What is the number of concurrent builds and users required? (With in the cloud or enterprise solutions, licenses limit you on concurrent builds and users.) The more concurrency you need comes with a price.
User interface—In my experience, some of the tools are easier to use than others. Who will be logging into the CI/CD tool?
Open source solutions will lack quick support, and cloud solutions will have quick support turnaround.
Does the CI/CD tool meet your security and regulatory requirements?
Easily integrates with the following services:
Amazon Web Services (AWS), Google Cloud Platform (GCP), Azure, etc.
Package repository such as Artifactory.
Cloud-based testing solutions such as Sauce Labs, Applitools, and QASymphony.
All your quality checks from linting, unit, integration, Selenium, visual, and more.
Notifications via e-mail, Slack, GitHub pull request status, and more.
What plugins are available out of the box? If you don't see plugins you need, can you write your own custom resources?
Monitoring
Built-in tool solution
Do you need to integrate third-party monitoring tools?
Run builds and tests in parallel
Back-ups
On-premises, you will need to build your backup system.
Cloud-based solution with automatically built-in backups.
How to manage all secrets, such as API keys, security credentials, etc.
SSH for easy debugging.
Supports Node.js, Java, Ruby, Python, Golang, and more.
4. Community Support
What makes a strong and growing CI/CD tool solution? Active community message boards for levels of expertise, users sharing knowledge, best practices, lessons learned, and openly sharing ideas. The best support for any tool is excellent documentation that stays up-to-date.
5. Return On Investment
CI/CD investment has a lot of value when you consider the range of tangible benefits that it brings to a development team.
Continuous quality checks with every stage of the CI/CD pipeline, reduction in defect cost by shifting left with automated testing.
Instant feedback on code quality for a specific feature.
Increasing the visibility of product quality.
Builds confidence when releasing new features to production.
Faster release cycles.
The list could go on forever. Everyone should be practicing CI/CD.
Conclusion
The ultimate goal is to find a solution allowing developers to focus on engineering and managing the entire pipeline to build, execute all quality checks, and deploy with ease. Every team has their own specific requirements—The five points above are a solid foundation to help your team narrow down the choices to find the right CI/CD tool for your needs and budget.
Greg Sypolt (@gregsypolt) is Director of Quality Engineering at Gannett | USA Today Network, a Fixate IO Contributor, and co-founder of Quality Element. He is responsible for test automation solutions, test coverage (from unit to end-to-end), and continuous integration across all Gannett | USA Today Network products, and has helped change the testing approach from manual to automated testing across several products at Gannett | USA Today Network. To determine improvements and testing gaps, he conducted a face-to-face interview survey process to understand all product development and deployment processes, testing strategies, tooling, and interactive in-house training programs.
