End-to-end (E2E) tests have a reputation for being “flaky”, unstable and unreliable. This can be down to how the application is built, how the tests are written or how the test data is set up, where the tests are being run for example on your CI workers or due to environment issues. Not only does this impact the reliability of your tests but it also affects the reputation of them causing statements like “shall we just turn them off”. If you want to scale your E2E test suite then read this brilliant article which explains the maths and gives a strong case for getting started now. Also, if you already use contracts (see definition if you are new to contract testing) that have been verified as part of your unit tests then you can use these contracts as the mocks in your E2E tests. Double win!!
Scenarios where contract testing can be applied
There are two scenarios where I think this can be applied, I’ve seen both in my experience around end to end tests which can cause different issues depending on which approach you choose to take. Some even use a hybrid approach of the two.
Problem Statement
End to end tests in an integration environment - where you have end to end tests which are run in an integration environment, trying to capture the “true” user viewpoint within your test. Either the fully integrated environment is spun up as part of the pipeline or the environment is permanently available. These tests go through the user journey which often spans multiple webpages, including features such as login, check out, etc. The aim of automated tests is to be deterministic, however in this approach multiple dependencies can break this principle. These tests offer a high degree of confidence but are often flaky, unstable and therefore become unreliable.
Mocked dependency end to end tests - whereby you are testing the user interface in isolation and mocking out the API, third party libraries, etc. I see this approach when it comes to “end to end” where the tests mock out the API requests to external services to enable them to run in continuous integration more reliably or touch microservices which the team doesn’t own. Therefore the tests do not reflect the integration environment and can be confusing when they are named “end to end”.
In both these scenarios I see a huge value in integrating API contracts as your mocks/stubs within your end to end test suite. Allowing you to run your tests reliably within continuous integration with the safety of knowing the API contracts have been verified by the provider of the API service. Therefore ensuring your tests remain stable and reflect the integration environment. In terms of comparison, I’m comparing directly to running your E2E tests against an integration environment, as in my experience this is generally where E2E tests run. However, using API contracts in your E2E tests would actually reflect production as you can point your stubs to the contracts tagged with the production version in the Pact Broker (Pactflow).
Another hidden benefit of using a contract testing approach over running your tests in a test environment or mocking manually, also reducing your maintenance effort and costs.
Clarifying E2E definition
To be clear before we continue, I am talking about Horizontal End to End tests here. Spanning multiple microservices including multiple RESTful API services importantly. Your typical user journeys or critical user paths which demonstrate that your core journeys are covered and your application can convert users to purchase or something like that.
Toolset for stabilising E2E tests
To demonstrate the introduction of contracts to your end to end tests, I’m going to be using Cypress & Pactflow (with a React App) to demonstrate how you could improve the stability, reliability and more realistic representation of the real service. Hopefully with a little experience with JavaScript and the basic understanding of contract testing you can follow the example provided. For a quick and easy to digest set of slides around how contract testing works, check out Pactflow's “How Pact contract testing works” slides.
Contract testing solution for E2E tests
I’ve chosen the second scenario but instead of using static mocks, using contracts as stubs (for more information checkout consumer Pact docs). The reason for this being these tests are faster, have less dependencies and easier to maintain.
Microservices End to End Example
For this example I will be using this repo: react-cypress-pactflow, based on the show Great British Bake Off.
To get things setup you will need an HTTP request framework, for this example I am using Axios.
#package.json
{
...
"dependencies":
{
"axios": "^0.24.0",
"cypress": "^9.1.0",
...
}
}The process I went through to come up with this solution was initially to intercept the API response within the before statement in the spec file. With the intercept calling out to the Pactflow stub. But this didn’t work because the intercept was not waiting for the stub to be returned before replying with the data. So I realised when making external calls in Cypress you need to wrap the external API in a cy.task, something like this:
#ingredients.spec.js
before(() => {
cy.task('getIngredients', '/ingredients/choco')
.then((data) =>
{
cy.intercept(`http://localhost:5000${url}`,
(req) => { req.reply(data) }).as('cake')
})
})
The string ‘getIngredients’ within the refers to a task within the plugins folder of Cypress like this:
#plugins/index.js
const axios = require('axios')
module.exports = (on, config) => {
on('task', { getIngredients(url) {
const result = axios.get(`https://demoOrg.pactflow.io/pacts/provider/Cake_API/consumer/Web_App/version/0.0.1/stub${url}`,
{ headers: { Authorization: 'Bearer xyz', }
})
.then(response => {
return response.data })
return result },
})
}As you can see from the cy.task('getIngredients’, ‘/ingredients/choco'), I passed the url I want to stub. Then within the task itself I pass that to the Pactflow stub functionality which would look like this .../stub/ingredients/choco. This is amazing because it allows me to dynamically add stub fixtures to my Cypress tests.
I believe this approach can not only stabilise your end to end test suite but make it reflect the production environment more closely while encouraging the development of contracts to support your release process by offering confidence to release with high-value automated checks.
You can read more about contract testing here on my blog, listen to my podcast or complete my FREE mini course walking you through this microservices end to end test example with API Contracts step by step.