Schema-based API generation
In a schema-based approach like Swagger, GraphQL or gRPC, the payloads can be generated from (or checked against) a schema, ensuring that the shape of the request and responses are compatible.
Assuming you always deploy in a way that includes no breaking changes, this will ensure that the shape of the communication payloads are not the source of any incompatibility, but it still leaves you vulnerable to semantic and behaviour changes.
Contract testing is an easy way to provide the missing value in this scenario. A well-written contract testing approach will:
- Track which versions are compatible with each other
- Protect against breaking semantic changes (eg,
AdminvsADMINfor a field that accepts strings) - Protect against broken behaviour changes
Version tracking
If your communication code is autogenerated, then you still need a way to track which versions are compatible with each other - are you deploying a client that is generated with exactly the same schema as the server?
The deploy check step in contract testing allows you to roll forward and roll back comfortably.
Semantic changes
Schema approaches are syntactic rather than semantic - for example, gRPC prevents syntactic breaking changes by ensuring that all fields have default values. However, this is a bit like saying “all our endpoints use json, so we can't send incompatible responses”. Just because the payload can be parsed by the consumer doesn't mean it has meaning to the consumer.
Contract testing uses detailed examples to mitigate this problem - it ensures that all the fields that you rely on are present in the response, the exact payloads can be understood by the client, and will be returned for the request that the client is really sending.
Schema expressibility
A related problem is that schemas often allow syntactically correct but
meaningless responses. For example, a User object's type field might be
defined as string when the type is really "admin" | "user" - which means
that Administrator would not be valid, but would pass the schema.
Optional fields compound this problem - in some payloads, the presence of one field might depend on another field's value. Asserting that your test doubles are valid according to the schema doesn't protect against mistakes in which fields are expected to be present together.
The detailed examples of contract testing ensure that you are never relying on an inappropriate combination of fields or field values.
Behaviour changes
Even if you have asserted that the type system has the same semantic and syntactic meaning between the two services, you can still come across behaviour changes. Just because a particular payload is possible to produce, it doesn't mean it will be produced.
Contract testing ensures that your consumer really does produce the requests that the provider understands, and that the provider really does produce the responses that your consumer understands.
Will contract testing help me?
All tests are just risk reduction. Schema-based approaches provide some safety over manual implementations, so depending on your needs, the risks might be acceptable. If you are comfortable with the risks above, then you don't need contract testing.
To answer the question of whether contract testing would help you, look at whether you currently experience communication-related outages in your existing deployment and testing practice.
Our experience is that contract testing is an effective and lightweight way to improve deployment confidence with schema-based APIs.