Continuous Integration: Why All The Fuss?

Posted Aug 9th, 2017

CI is a combination of tools, philosophies, and practices that help teams ship quality software at a faster pace.

Continuous Integration is a combination of tools, philosophies, and practices that help teams ship higher quality software at a faster pace. CI focuses on bringing traditionally difficult, late-hitting events as early in to the process as possible by automating processes that had been manual. This removes bottlenecks of manually intensive actions and also reduces the risk of huge technical issues hitting late in a release cycle.

What is Continuous Integration?

CI revolves around a server that’s responsible for orchestrating tasks the team defines. Many open source and commercial CI servers are available, including Jenkins, Travis, Team City, and Team Foundation Server to name a few. All popular CI servers are extraordinarily customizable; all additionally have a large environment of plugins and adapters supporting other tools and systems involved in delivering quality software: think of reporting, static code analysis, deploying database changes, etc.

CI is not a panacea for undisciplined, unskilled work; to the contrary, it’s a highly disciplined approach that requires time to implement and optimize. That effort pays off very quickly as teams spend less time on painful merging, test execution, environment configuration, etc. Instead, the teams are able to focus on delivering value for their projects.

Merge Your Code. Constantly.

Initial implementations of Continuous Integration focused on one specific step: preventing the extraordinary pain around infrequent, late-in-cycle merges of a project’s codebases. Far too many teams don’t regularly integrate and build their codebases. Simply checking in to a shared code repository (Git, SVN, TFS, whatever) does nothing to ensure a particular check in hasn’t broken the build entirely—much less injected a bug.

At their basic, CI systems monitor a code repository. When a check in is made, the CI server will pull the latest changes and build the software. Nothing more. That may sound anti-climactic, but think of the many issues this can head off simply by running a compiler and linker over the entire updated codebase: unchecked syntax errors, interface and module versioning problems, missing dependencies, and similar problems are all immediately identified simply by running a basic build of the system!

Moving to CI for nothing more than builds can often dramatically improve a team’s performance simply by making merge and integration pain much smaller, more frequent, and much more manageable!

Automate Your Deployments

Deploying code to environments is one of the most time consuming, manually intensive steps right after merging and building. As a result, many teams use a CI task to push changes to various environments. With a CI server, scripts can be created and tested to automate the arduous work of provisioning and configuring environments for code, then pushing data, dependencies, and the system itself to those environments.

Run Your Tests Every. Single. Time.

CI quickly evolved from “just” merging and building code to becoming a full on automated delivery pipeline. This involves bringing as many different quality checks in to the automated process as possible. One of CI’s largest benefits is being able to run all automated tests after every significant change to the codebase. This ensures that not only are syntax and link checks being run, the more critical checks around code quality, acceptance tests, performance, security, and other checks are constantly being evaluated.

What Types of Testing Fit in Continuous Integration?

CI excels in helping teams keep a constant monitor on the quality and health of their codebases and systems. Wrapping your team’s quality checks and testing in to your process enables you to automatically run tests including:

  • Unit Tests
  • Integration
  • Baseline Performance
  • Code Quality
  • Security
  • Accessibility

And, yes, functional testing as well!

Unit and Integration Tests

Unit and integration tests are easy targets to wrap in to a CI environment, especially unit tests.

Unit Tests

Unit tests, because of their lack of dependency on any external systems, are simple to wrap in to CI. There’s no requirement for environments, no database needed, no container for web services, etc. At a bare minimum, teams should look to incorporate unit testing from the very start of any CI effort.

Integration/System/API Tests

Integration, system, or API testing (depending on your vernacular) are also great targets for bringing in to CI environments; however, they’ll take more work as they’re generally dependent on active web services, a database, etc. This involves some form of deployment to an actual environment—Docker, virtualization, etc.—as well as some provisioning.

Baseline Performance In a CI Environment

Performance testing is often ignored in CI environments. That’s understandable, as performance testing generally involves specific checks on near-production hardware using heavyweight tools.

However, baselining or A/B testing is a high-value performance check that can be brought in to CI environments with a significant payoff. Baselining takes a much simpler, lightweight approach to performance. The goal is not to validate whether the system performs as expected with real-world data and hardware, but instead looks to identify regressions in performance.

Baseline tests expect a consistent environment, not a real-world one. If the environment stays the same, then any significant degredations in benchmark timing are due to regressions in system code—mangled exception handling in a web service, a badly thought database index change, or a bug introduced into a database accessor, for example.

Automated Functional Testing

Functional testing often takes up a huge amount of time in any project. The time required to run through complex workflows and business requirements is significant. Ergo, functional testing is a common target for automation.

Good functional test suites are a wonderful candidate for bringing in to a CI environment; however, care needs to be taken to make sure the suite runs in a performant fashion. Functional tests, by their very nature, are long-running tasks. Because of this, functional tests generally should be executed in a well-constructed parallel execution environment that meets all browser, operating system, etc. requirements.

Executing functional testing reliably in a CI environment means first solving a number of problems mentioned earlier in this article: merging, building, deploying, and execution/reporting of test suites. It’s much easier to get a stable functional test run once those initial steps are solved. At that point teams can start rolling in their functional suites.

Bringing Sauce Labs Into Your Continuous Integration

Running functional test suites in a CI environment can create significant challenges for teams. Not only do teams need to automate deployments, stand up environments, deploy code, etc., they also find themselves having to manage browser configurations, mobile devices, and infrastructure for running execution agents.

Sauce Labs dramatically reduces this complexity by providing a number of services designed to drastically reduce the effort for teams to quickly incorporate fast, effective functional test execution in their CI pipeline.

Empowering Execution

Sauce Labs’ services give teams all they need to start wrapping their functional test suites in to their CI systems. Sauce hosts execution-related services in their own cloud, and can even work through corporate firewalls via Sauce Connect, an encrypted proxy that enables communication to websites that aren’t publicly accessible. (Be nice to your network security folks and make sure to coordinate with them first!)

Sauce also handles scaling out execution to handle multiple instances of browsers. Teams don’t need to create and manage their own infrastructure just for execution. Instead, that’s handled “hands-off” by Sauce.

Perhaps one of the largest challenges for any team is mobile device management. Few teams have the financial resources to purchase, configure, and manage their own matrix of mobile devices, operating systems, and features. Solid test coverage requires a mix of emulators, simulators, and real devices. Sauce Labs’ Real Device Cloud takes that burden off teams, enabling them to focus on high-quality testing.

Integrating Sauce Labs Into Your CI

Sauce Labs directly supports the most widely used CI systems:

  • Visual Studio Team Services
  • Bitbucket Pipelines
  • Bamboo
  • Team City
  • Jenkins

These integrations handle configuration, execution (with direct support for Sauce Connect), and reporting. Additionally, the teams behind Travis CI, Solano, and Circle have added their own support for Sauce Labs execution.

Focusing on Quality With Continuous Integration

Teams looking to maximize their value and quality output have to move past manually intensive operations and leverage the years of advances in Continuous Integration. Automating repetitive, intensive tasks lets teams focus on higher-value activities. Moreover, Sauce Labs’ support for parallel execution, physical mobile devices, and out-of-the-box integrations with popular CI systems make it a snap for teams to quickly roll out effective automated testing.

Written by

Jim Holmes


CI/CDUnit TestingIntegrationsAutomated testing