Skip to main content

Frequently Asked Questions

This document is aimed at newcomers to contract testing. Experienced users may find answers to their questions in the best practices section,

If you still have questions after reading this document and the best practices section, please open an issue with your question and we'll answer it as soon as possible.

What is mocked during a contract test?​

A contract test is divided into two halves. First the contract is defined (which includes some testing). Then the contract is verified, In each test half, only one side is mocked. For example, for a contract written by an HTTP client:

  1. While writing the contract, you invoke your real client code, and your contract testing framework provides a mock server that confirms that you really do send the request you said you would.
  2. While verifying the contract, the contract testing framework provides a mock client that behaves exactly like the client that was tested in step 1. This client is used to query your real server, running (usually) locally, or in a test environment.

The contract records the behaviour of the mock used in step 1, which is then validated against the behaviour of the real provider in step 2.

How do I write examples that depend on each other?​

This question is often asked by people testing endpoints that create or modify data. "How do I make a test for the getUser request depend on the createUser request?"

Instead of making examples dependent on each other (and the corresponding explosion of complexity and potentially brittle tests), each contract test example runs independently. This is achieved with state definitions and state setup handlers, which you can use to set up the user required by the getUser request first.

How do I tell ContractCase that a field is optional?​

You write two examples - one with the field, and one without. The reason for this is that a test which allows optional fields would still pass even if the field was never present. You wouldn't be sure that your provider could ever generate that missing field.

Remember also that fields that your consumer isn't using don't need to be described in the contract. Any additional fields are ignored.

How do I test an array field that might have zero items in it?​

If your arrays can be empty or have contents in them, you need two examples. One, using the appropriate array matcher, and one with an empty array literal.

For the same reason as optional fields, the array matchers (like arrayEachEntryMatches(itemDescription)) won't match empty arrays. If empty arrays passed those matchers, then the provider would pass the test even if it could never generate any contents inside the array.

Can I ensure that my whole API surface is covered?​

The short answer is: it depends. Note that in a contract test, you don’t actually describe the whole payload that you consume, just the parts your consumer needs. This is advantageous because it’s not a breaking change to rename or remove a field/endpoint that no-one is using.

For the long answer, have a read of the best practices section on API coverage.

Do we need contract testing if we use schemas like Swagger, GraphQL or gRPC to generate our communication code?​

Yes, contract testing can still provide value in this scenario.

A well-written contract testing approach will:

  1. Track which versions are compatible with each other
  2. Protect against deployments that are broken due to communication issues
  3. Protect against breaking semantic changes (eg Admin vs ADMIN for a field that accepts strings)

If your communication code is autogenerated, then you still need a way to track which versions are compatible with each other.

You also need a way to track and prevent breaking changes - most schema based approaches do this with manual rigour. Manual rigour is effective, but fallible.

All schema approaches are syntactic rather than semantic - for example, gRPC prevents syntactic breaking changes by ensuring that all fields have default values. However, this is a bit like saying “all our endpoints use json, so we can't send incompatible responses”. Just because the payload can be parsed by the consumer doesn't mean it has meaning to the consumer.

Technically, you never need any testing, as all tests are just risk reduction. Schema-based approaches provide some safety over manual implementations, so depending on your needs, the risks might be acceptable. The question is whether you want to live with the risk of deploying breaking API changes, or whether you are successfully avoiding breaking API changes with your current approaches.

For more on this topic, see contract testing vs schema tests