Accelerating Releases with Quality: Contract Testing vs. E2E Functional Testing

Posted Sep 9th, 2021

How much contract testing vs. E2E functional testing will help you accelerate releases without reducing quality or raising QA costs?

Good API testing should give developers confidence that services will run and perform as expected in production. But how much contract testing vs. E2E functional testing is really needed for quality at speed?

What Is Contract Testing or Consumer-driven Contract Testing?

The most common open source library for contract testing is called Pact. We will let Pact define contract testing themselves:

“Contract testing is a technique for testing an integration point by checking each application in isolation to ensure the messages it sends or receives conform to a shared understanding that is documented in a ‘contract.’”

Plainly speaking, contract testing is a lightweight form of API testing that strictly checks the content and format of requests and responses. Unlike functional tests, contract tests do not validate the logic or consumer flows of APIs. Contract testing really focuses on ensuring that spec files such as Swagger or OpenAPI and RAML properly fulfill contracts between API consumers and producers.

SL API Testing Laptop Bot v1 SDET - Developer Bot no text

Contract testing can also validate spec files in an automated way by capturing how an API consumer and producer communicate with each other. This is typically done in a protected, static environment: tests are run against mocked (not live) environments, allowing contract tests to compare isolated API responses to the contract for immediate attention if something is wrong.

When and Where Contract Testing Makes Sense

During the very early stages of building or updating a mobile app or web platform, API developers are challenged to reduce time and costs, whether working locally or in the pipeline. Contract testing may then make more sense than end-to-end testing to achieve the right balance of risk and speed. 

Consider the following observability checklist before committing to contract testing:

  • Are you building or changing internal APIs where you control the development of both the consumer and the provider?

  • Are the consumer and provider both under active development?

  • Can the provider team easily control the data returned in the provider's responses?

  • Are the requirements of the consumer(s) going to be used to drive the features of the provider?

  • Is there a small enough number of consumers for a given provider that the provider team can manage an individual relationship with each consumer team without being overwhelmed?

By checking one or more of the list items, your confidence level may increase for contract testing. But even if you check all of the items, contract testing will not close certain QA gaps such as verifying data semantics and validating complex integrations in real world scenarios. A sprawl (“hairball”) of isolated contract tests would be needed to do the work of a single E2E functional test.

Benefits of Contract Testing

Contract testing is popular for helping many teams accelerate development while lowering costs. It's particularly useful in modern workflows requiring high iteration among scattered stakeholders. Key benefits include:

  • Contract testing is more flexible for continuously evolving codebases

  • It will verify applications earlier than end-to-end tests can

  • Validation flows naturally and quickly due to the minimal “testing” being done

  • Both the producer and the consumer are tested

  • Isolation provides a way to reduce the scope of an issue

As more organizations rush to embrace the APIs that power all of their business-critical services and apps, (consumer-driven) contract testing may help developers maintain velocity to ship with quality on time.

Weaknesses of Contract Testing

While contract testing is a solid method of testing, it does have many weaknesses that make it not an optimal method to drive consistent quality at speed. Weaknesses include:

  • Both sides of the integration must be contract tested in the same way; human error often emerges from the sprawl of isolated contract tests that run in siloes

  • Testing APIs where the consumers cannot be individually identified (e.g., public APIs), resulting in unreliable tests, false-negatives

  • Performance and load testing require far greater testing observability, checking whole API consumer flows and side-effects

  • Situations (paradoxes) where you cannot load data into the provider without using the very API that you're actually testing (e.g., public APIs)

  • Situations where you cannot control the data being used to generate the provider's responses

  • Testing new or existing providers where the functionality is not being driven or altered by the needs of particular consumers (e.g., a public API or an OAuth provider where the API is completely stable)

  • Testing providers where the consumer and provider teams do not have good communication channels

  • Testing "pass through" APIs in which the provider merely passes on the request contents to a downstream service without validating them

Contract testing is a fast and robust method of testing APIs during the early stages of design and development. Optimally, contract testing is applied to a team that is purely creating internal APIs with a limited number of consumers. However, if your organization is creating a robust API program that will have public APIs where third-party consumers will be consuming the APIs, a more robust method of testing may be needed. That is where functional and end-to-end testing really shine in helping development teams thrive as they attempt to accelerate releases with shorter testing times.

Modern E2E Functional Testing: Simpler, Faster, Cheaper

In summary, functional and end-to-end API testing are vastly more insightful than contract testing because the main aim is to determine how multiple APIs can collaborate to achieve meaningful results. While contract testing focuses on the contract of each single endpoint, functional testing makes sure that the API consumer can fully support the user story’s goals, which often require dynamic combinations of APIs.

An ideal quality workflow should allow teams to painlessly choose between contract testing and E2E functional testing—without significantly impacting velocity and time to market. This is the mindset that helped to drive Sauce Labs’ new approach to API E2E functional testing (free for all Sauce Labs users).

Currently, Sauce Labs API Testing can feed off contracts (OpenAPI for example) to kickstart functional tests quickly and easily. Although it is not quite contract testing, it is a holistic testing of contracts. But stay tuned: Sauce Labs is working on contract testing, and exclusive tech previews of Sauce Labs contract testing may be announced soon.

Sign Up (or sign in) to your Sauce Labs account to begin using Free API Testing and Monitoring.


Written by

Patrick Poulin and Simone Pezzano (Co-founders of API Fortress, a Sauce Labs company)


Topics

API Testing