This blog was first published on September 9th, 2021. It has been updated with the most current information.
What Is Contract Testing?
Contract testing is a lightweight form of API testing that strictly checks the content and format of requests and responses. Unlike functional tests, contract tests do not validate the logic or consumer flows of APIs. Contract testing really focuses on ensuring that spec files such as Swagger or OpenAPI and RAML properly fulfill contracts between API consumers and producers.
Contract testing can also validate spec files in an automated way by capturing how an API consumer and producer communicate with each other. This is typically done in a protected, static environment: tests are run against mocked (not live) environments, allowing contract tests to compare isolated API responses to the contract for immediate attention if something is wrong.
When and Where Contract Testing Makes Sense
During the very early stages of building or updating a mobile app or web platform, API developers are challenged to reduce time and costs, whether working locally or in the pipeline. Contract testing may then make more sense than end-to-end testing to achieve the right balance of risk and speed.
Consider the following observability checklist before committing to contract testing:
Are you building or changing internal APIs where you control the development of both the consumer and the provider?
Are the consumer and provider both under active development?
Can the provider team easily control the data returned in the provider's responses?
Are the requirements of the consumer(s) going to be used to drive the features of the provider?
Is there a small enough number of consumers for a given provider that the provider team can manage an individual relationship with each consumer team without being overwhelmed?
By checking one or more of the list items, your confidence level may increase for contract testing. But even if you check all the items, contract testing will not close certain QA gaps such as verifying data semantics and validating complex integrations in real world scenarios. A sprawl (“hairball”) of isolated contract tests would be needed to do the work of a single E2E functional test.
Benefits of Contract Testing
Contract testing is popular for helping many teams accelerate development while lowering costs. It's particularly useful in modern workflows requiring high iteration among scattered stakeholders. Key benefits include:
Contract testing is more flexible for continuously evolving codebases
It will verify applications earlier than end-to-end tests can
Validation flows naturally and quickly due to the minimal “testing” being done
Both the producer and the consumer are tested
Isolation provides a way to reduce the scope of an issue
As more organizations rush to embrace the APIs that power all of their business-critical services and apps, (consumer-driven) contract testing may help developers maintain velocity to ship with quality on time.
Weaknesses of Contract Testing
While contract testing is a solid method of testing, it does have many weaknesses that make it not an optimal method to drive consistent quality at speed. Weaknesses include:
Both sides of the integration must be contract tested in the same way; human error often emerges from the sprawl of isolated contract tests that run in silos
Testing APIs where the consumers cannot be individually identified (e.g., public APIs), resulting in unreliable tests, false-negatives
Performance and load testing require far greater testing observability, checking whole API consumer flows and side effects
Situations (paradoxes) where you cannot load data into the provider without using the very API that you're actually testing (e.g., public APIs)
Situations where you cannot control the data being used to generate the provider's responses
Testing new or existing providers where the functionality is not being driven or altered by the needs of particular consumers (e.g., a public API or an OAuth provider where the API is completely stable)
Testing providers where the consumer and provider teams do not have good communication channels
Testing "pass-through" APIs in which the provider merely passes on the request contents to a downstream service without validating them
Contract testing is a fast and robust method of testing APIs during the early stages of design and development. Optimally, contract testing is applied to a team that is purely creating internal APIs with a limited number of consumers. However, if your organization is creating a robust API program that will have public APIs where third-party consumers will be consuming the APIs, a more robust method of testing may be needed. That is where functional and end-to-end testing really shines in helping development teams thrive as they attempt to accelerate releases with shorter testing times.
API Testing for the Full SDLC
In summary, functional and end-to-end API testing are more insightful than contract testing because the main aim is to determine how multiple APIs can collaborate to achieve meaningful results. While contract testing focuses on the contract of each single endpoint, functional testing makes sure that the API consumer can fully support the user story’s goals, which often require dynamic combinations of APIs.
Not all test cases require more than contract testing. The ideal quality workflow should allow teams to painlessly deploy desired ratios of contract testing and functional testing for projects—without significantly impacting velocity and time to market. This is the mindset that helped to drive Sauce Labs’ new approach to API testing for the whole SDLC (free for all Sauce Labs users). Sauce Labs users can generate API mocks and bidirectional contract tests from OpenAPI specs. And then contract tests can easily be extended into functional tests, integration tests, and load tests as well as API monitors in production. In this way, teams can get the best of both contract and functional testing to achieve optimal confidence for microservices and CI/CD pipeline success.
