Recently I did a workshop on how to test microservices with my new team. I joined as tech lead this summer and are getting into the business domain and the technology.
The technical setup is a landscape of Java and .NET systems and services running in AWS. We have a monolith .NET application that has aged reasonably well. We are migrating to microservices to get elasticity and to reduce cost. Yet, while our microservices built within the last three years use powerful frameworks and packages, they are lacking when it comes to documentation and test. Technical debt in the making. Not a road to sprint down blindly. You don’t want your new code to turn legacy this soon.
So I did a peptalk on how to test microservices. It was well received so here it is for a wider audience (you).
As always, we need to understand the what and the why before addressing the how.
Why: Engineering Nirvana
We don't test because it is fun. We test because we want to work in Engineering Nirvana: A place where it is easy to do the right thing. Where we ship new features with little effort and where the impact of making mistakes is small. Not just for the senior guy but for everyone in the team.
One reason we like microservices is that changes can be built and changed in isolation without restesting everything else. This is not always true! Your microservices can have a complex web of dependencies that is a fearsome barrier for newcomers to get their head around. Without knowing the big picture, any change can take out production for what you know. A microservice architecture alone will not send us strait to Nirvana, something else is needed (hint: testing is a big part of the answer).
What is a microservice
One way of looking at a microservice is this:
The API handles incoming requests from consumers, doing basic validation, logging and orchestration. The Domain model is where the juicy details of the business logic is expressed. Almost always a service requires bundled and external dependencies to implement its functionality.
This is view is very similar to how you would describe a component back in the days. This difference is in the dependencies and this is where things get complicated. Databases that are slow to respond. Services run by business partners that fail exactly when most inconvenient. Building a microservice that keep flying even when everything else around it crashes left and right is a challenge.
To build a great microservice architecture, being on top of your dependencies is key. Interestingly, understanding your dependencies is also key for how to test a system. If you can’t keep your dependencies constant while testing, you are testing a moving target. Then you are heading down a road of endless guessing at why things stopped working. That kind of testing requires you to be religious to stay sane.
What frequently happens is that you end up with a microservice that looks like this:
An underdeveloped domain model and bloated logic in your controller and in your data access layer. The size of the service grows organically through many small increments and it is always easier to add logic to existing structure than to refactor and grow new classes. Unfortunately, this makes it hard to test and very soon hard to change. So testing is the canary in the cage! If your microservice is hard to test, it is legacy!
What to test
So let’s look at the drawings and consider: Where is the complexity? Where is the impact of mistakes high?
API
Good tools and conventions help us create a well designed interface that captures well formed request, rejects badly formed requests, and sends back well formed results. The API is a good high level expression of what your service does and you will want regression tests at this level to cover the main usage scenarios. Mistakes here are likely to have a big impact but are also likely to be found early as all calls into your service pass through here.
Domain model
This is where the heart of your business logic resides. If you make a mistake here, the service will most likely not do what you expect it to do. Use your detailed knowledge of the domain and the code to test agressive here. Traditional unit tests love this part of the code. A well designed domain model is easy to cover tightly with unit tests. Test Driven Development. Clean Code. Domain Driven Design. This is where you live out the dream.
Bundled dependency
You need to make sure you understand what these dependencies do and how to make them do what you want. If they fail, it is almost always because you didn’t use them correctly. You would wish that once they work, they will keep working. However, third party components tend to release patches and upgrades all the time. A few regression tests may well be worth the effort. At least with charts and packages, today you have extremely good control of the third party components you take in.
External dependency
Ideally you trust your external dependencies to work. After all, you didn’t write them so it is not your problem when they fail? Not so. At the very minimum, you want to know fast when external dependencies fail as it will save your a ton of troubleshooting effort. Automated high level tests that are robust to minor changes is the way to go.
How to test
Cool, so we know why we test and what to test. So bring out the stone tablets and let's engrave our fundamental beliefs. These rules work in our domain and setup. YMMV. One size does not fit all.
In the spirit of the Agile Manifesto, we begin by formulating our preferences:
Preferences
We prefer...
- No code over complex code. If a situation occurs rarely and can be handled manually this is preferably to writing complex code.
- Automated test over manual test. If a manual test can be automated, it should be.
- Manual test over complex test code. If automating a test leads to code that is hard to maintain, we prefer manual tests.
- Testing in production over maintaining complex tests. If creating and maintaining the state needed to run a test reliable requires maintaining complex code, we prefer to test in production.
From these we derive our attitude and behaviour to testing microservices forming our manifesto for testing the software we create and maintain:
Testing manifesto part 1: Mindset
Test is in our hearts and minds:
- All our test code adds value
- We hold test code to the same high standard as production code
- We test before we commit
- We test after we merge to main
- We document how to test
- We follow the same approach for all services
Testing manifesto part 2: Tools
We use the right tool for the right job:
- We implement business logic in the domain model and unit test the domain model
- We integration tests the API and use mocks/stubs to control dependencies
- We validate configuration parameters and API requests using FluentValidation (rule engine)
- We do load test when needed
- We use manual tests to test the end-to-end flow
We do not write tests to reach a high code coverage with our tests. Measuring code coverage can be a great tool to see where adding a test can add value. But we only add the test if it adds value. It might be a good idea to write a test when we have discovered and fixed a bug. But we only add a test if it adds value.
We do not test the compiler. If we can express the business logic with code that can be verified by the compiler, we do not create tests.
We do not leave testing to testers. Testing is a natural part of the process of writing code. As is writing documentation. We do it because it adds value.
I think the part about keeping test code to production standing is really important. This will save you a ton of effort maintaining poorly designed tests and repairing a degraded test culture. Better to test just enough than to test everything. The other key to a succesful change is this:
The future is now
Don’t ask what your team can do for you tomorrow. Ask what you can do for your team today. Today's greenfield project is tomorrow’s legacy code. We do not start testing at some magic date in the future after which everything will be rainbows and unicorns. We begin today.
So let's see where this will take us. It was fun to turn 20 years of experience into workshop material. By the way, if you also want to kill practises of dark magic in your professional life, my game Death of Rapacus is available now.