Continuous Integration - The Heart of DevOps

Missed our earlier webinar, "Beyond the Release - CI That Transforms"? Check out the recap below.

In this webinar, we discussed the power of CI and possible considerations for the future. One of the more interesting aspects of the webinar was seeing the modern pipeline with the new Sauce CI Dashboard alongside Cloudbees' automation templates. We also conducted a small CI survey at the beginning of the webinar, and ended with a Q&A.

Continuous Delivery and Deployment (CD) steal the show in DevOps conversations, but the reality is that Delivery and Deployment are not for every organization, nor are they already widely adopted. In order to move on to delivery and deployment, organizations must get Continuous Integration (CI) right — unless they were built from day one with the DevOps framework, and did not have to fit the processes into existing environments.

The reason CI is so powerful is that it allows you to dip your toe into the modern delivery pipeline without the risk or complexity of building out delivery and deployment all at once - and with the potential of failure. You can consider CI as the on-boarding for DevOps. And from a process and tool standpoint, CI is nearly identical to delivery and deployment, it means that once you get it right you can easily move on.

I'm pleased to say the results of the Sauce Labs CI survey were almost exactly what I expected, served with a side of surprise. For me, the most interesting aspect of the survey results is how they appear to be in conflict with the perceived high CI adoption and success rates already existing in the market. Let’s look at the results among 500+ attendees:

What Types of Automated Tests do you run?

6% of the attendees are not running automated tests at all! This was astonishing to me. I expected 1% at most, especially given the audience, because they are already familiar with automation. At a minimum, I would expect all companies to automate Unit testing. However there is a high likelihood that what this 6% is saying is that they have automated tests, just initiated manually. The results also showed that many are running functional tests. This is great! However, only 27% are running integration tests. This is troubling because compared to the reported 45% who state they are doing CI already, the lack of integration testing would seem to contradict that statement. I suspect that this is a definition problem, where some may define CI as being simply a shared testing environment, and not really the CI process as described in the webinar.

Do you have an Integration environment?

77% of the audience reported having an integration environment, yet only 27% have automated integration testing. This could be an indication that infrastructure was the focus over process.

And the theme continued.

Where are you with CI?

45% have it down! This also surprises me, as the earlier numbers indicate that 74% have not yet completed automation of the entire stack. Without full CI automation, it is a stretch to go to CD, where you have no choice but to automate everything. But the fact that 53% of organizations have not or are just starting is consistent with my observations.

Now I'd like to showcase my favorite part of the webinar, the Q&A. We received a great set of questions and were able to address most during the event, but here are the ones I want to highlight with additional responses:

Q: What are the challenges of using CI in a Cloud environment? A: One of the misconceptions about CI infrastructure is that it is one location, and static. The idea of integration infrastructure is that it can be spun up and torn down on demand, and the best system is used for the process. For example, it is not necessarily true that all functional tests run with Unit. The biggest challenge of using Cloud solutions is integration and morphing existing processes. Where CI fails is most often in the lack of planning around the process. If the process is solid, then it’s relatively easy to introduce any number of integration environments, and have oversight on their results and repeated usage.

Q: What cultural factors are absolutely necessary for CI to (a) happen and (b) sustain? A: Shared motivation for results, no barriers, and no ownership. Integration environments are like the application/code café. Everyone comes together. Which means that the creation of these environments needs to be autonomous, and open. For example, there cannot be a ticketing process to obtain CI VMs or access to Cloud solutions. And there needs to be flexibility in who can do what. So this means there cannot be any barriers between IT, Dev, and QA. QA should be able to suggest changes to the entire team, for example. You can achieve this by defining a shared objective that is all about results. The results equal finding bugs and resolving them faster. With this goal, more commits and more iterations in the integration environment will happen naturally. This drives more releases per commit, more automation in the CI environment, and more interaction among the entire team.

The beauty of the CI process is that failure is inevitable, and when issues occur, they have little impact on anything but time. So adding sensitivity to the environment only limits the ability to use it. What needs to be well established are configurations on the environment, such as framework versions, etc. This is where using a Cloud solution is nice because it ensures that consistency. Deploy, deploy, test, test as much and as fast as you can.

Q: How does CI help maintenance of a small application when the cost of maintaining CI is a bit high for the application? A: It is true that the larger the team and application, the more justifiable the cost of infrastructure and tooling around CI. But on the flipside, the ability to set up API calls and Webhooks for smaller applications is easier than for those that are large. This is generally due to less dependencies and the number of integration points. For small applications, the goal should be CI that is 100% PaaS-based in Cloud testing environments, so that the ONLY effort is integration. This comes at a low cost when it is done by developers while they are coding.

Q: How effectively can we integrate a test automation suite into CI? A: The basis of this question is a bit concerning because it would imply that the automation was developed in a silo. Good automation should be transportable from ad-hoc boxes to a standardized CI environment. And the engine that drives the combined set of automation is fairly easy to wire up. What is missing in this question is the type of testing.

One of the biggest benefits Sauce brings to functional testing specifically is the offloading of massive testing grids that come at a large dollar and opportunity cost. And here, like many other Cloud solutions, the integration point is an API with a secure tunnel to your on-prem or Cloud IaaS testing environment which interprets the results, runs scripts from your testing suite, and manages the rinse and repeat.

Q: Does optimized and good usage of CI actually have much of an effect on quality or release schedule? It's only an automation of the checkout-build-deploy-test process, so it should technically not have any effect on quality of product or release schedule, right? A: Ouch. This one hurt my soul a little. If CI is not substantially increasing the quality of your application and the number of releases you are working on, you're doing it wrong. It should not just be a matter of automating existing processes. Instead, it should go like this:

Developers' commits end up in fresh CI environments, therefore there will be more frequent releases to CI, so it follows that there will be more frequently run automated tests. With more tests, more bugs are caught before delivery, thus the cost per bug is less because it happens earlier on in the process and there are fewer bugs in production. It follows that the cost of resolved bugs is less, both in dollar and opportunity cost to the backlog ... and all this means that your customers are happier.

If you consider volume of releases only, you can find ROI. But you can go beyond that: if you have a great CI environment, you can fail forward with higher risk functionality. This means product improvements come much faster, so both the real and opportunity ROI are tremendous.

We often forget that proactive is far better than reactive. Similarly, we forget that the more bugs you have the more bugs you will have, so without CI you are increasing the cost of all downstream processes.

When you get CI right, you can move downstream to high-speed testing on mock applications, service virtualization, and pipelines driven by containers. On the flipside, if you do not get CI right, you cannot expect to move on to delivery and deployment.

Based on the survey and questions, it seems that there is a lot of confusion both on the definition of CI, and where the market actually is with CI maturity. It would indicate to me that there is a lot to learn when it comes to CI, and that there are a ton of possibilities for improvement as well.

Chris Riley is a technologist who has spent 12 years helping organizations transition from traditional development practices to a modern set of culture, processes and tooling. In addition to being a research analyst, he is an O’Reilly author, regular speaker, and subject matter expert in the areas of DevOps Strategy and culture and Enterprise Content Management. Chris believes the biggest challenges faced in the tech market is not tools, but rather people and planning.

Throughout Chris’s career he has crossed the roles of marketing, product management, and engineering to gain a unique perspective of how the deeply technical is used to solve real-world problems. By working with both early adopters and late, he has watched technologies mature from rough solutions to essential and transparent. In addition to spending his time understanding the market he helps ISVs selling B2D and practitioner of DevOps Strategy. He is interested in machine-learning, and the intersection of BigData and Information Management.