When Tests Really Pay Off

Few topics spark such dogged arguments in development teams as testing, and the argument is almost always framed as a question of character: some are disciplined, others sloppy. That is why it never ends — a moral question has no answer, only camps.

This text treats tests neither as a virtue nor as dead weight, but as what they are: an investment. Every test costs something — writing it, maintaining it, running it, and the design it locks in. And every test delivers something — a change that becomes safe, a bug that never reaches production. The return on this investment varies enormously depending on what you test. Some tests pay for themselves many times over. Others cost money forever and never catch anything. So the only useful question is not "are we testing enough?" but "does this test pay off?".

Why teams argue about tests

The argument has two camps, and each is right about the other. One camp makes coverage a virtue: anything untested counts as unfinished, and the goal is a high percentage. Over time it suffocates under brittle test code that breaks with every restructuring without ever having caught a real bug — and change becomes expensive. The other camp experiences exactly that and concludes that tests only slow you down; it ships faster and pays later with bugs nobody saw coming.

Each camp has correctly diagnosed the other's symptom and overlooked its own cause. The mistake is the same on both sides: they treat "testing" as a quantity — more or less — rather than a question of selection. Coverage measures how much code is executed, not how much risk is covered. It is an input metric, not a value. As soon as you frame the question economically — what does this test cost, and what does it buy — the holy war dissolves. There is no right amount of testing. There are only tests that pay off and tests that don't.

One point makes this economics merciless, and it is almost always overlooked: a test is itself code. It is a second system that you write, read and maintain, that has to be touched whenever the production code changes, and that can itself contain bugs. That makes a bad test not neutral but worse than none at all — it costs ongoing maintenance, creates a false sense of security and punishes exactly the restructuring you want to encourage, without ever catching a real bug. Anyone who thinks tests are free sees only half the bill and later wonders why the suite slows the team down instead of carrying it.

What should actually be tested

The value of a test can be named. It is higher the more likely a bug is to arise at that spot, the more expensive that bug would be in production, and the more the test safeguards future change. Its cost is higher the more effort it takes to write and — above all — to maintain, the more often the tested code changes, and the more the test is tied to implementation details rather than behavior. A test pays off where its value clearly exceeds its cost.

That tells you what to test: the places where a bug is expensive and the behavior is stable enough for the test to outlast it. That means the domain core with its business rules, the paths where a bug does real damage — money, data integrity, security, anything irreversible — the stable public contracts, and the tricky calculations with many edge cases. The touchstone is not "is this line covered?" but "would a bug here be expensive, and will this test still hold next month?".

The value of a test is not limited to catching bugs. A good test is also an executable specification: it records what the code is supposed to do, in a form that cannot go stale because it runs along with the code. Whoever changes a rule immediately sees from the breaking test which commitments they are touching — the test makes visible the intent that is often only implicit in the code itself. This second benefit reinforces the first: precisely in the expensive, logic-dense places, the documented intent is as valuable as the caught bug, and both pay into the same investment.

The recommendation: select tests by the cost of failure and stability, not by coverage. The price is that a coverage number can look low even though what matters is covered — and you have to defend that against metric-driven expectations. We decide otherwise for code whose bugs would be cheap and immediately visible; there, even shallow testing is acceptable, because a bug without consequences shows up quickly and is cheap to fix.

Where tests have the highest ROI

The best place for a test is where three things come together: high failure cost, low rate of change and a lot of logic in a small space. That is exactly where the test is cheap to write, catches expensive bugs and lives a long time without needing maintenance.

Diagram: The value of a test rises with the cost of failure and falls with the rate of change.

Concretely, that means pure calculations with branching — pricing, tax or discount logic — which are cheap to test and catch expensive bugs; the domain invariants that must always hold; the contract boundaries to the outside world, where a break hits other people's systems; and every reproduced bug, pinned down as a regression test, because a bug that has happened once will happen again.

The recommendation: invest first in tests for logic-dense, expensive, stable code — and pin down every real bug as a regression test. The price is discipline in an uncomfortable place: the expensive core is often also the part you least want to touch. We decide otherwise for a calculation that is still being tried out and changes frequently; there we wait until it is stable instead of rewriting the test with every iteration.

Where tests generate more maintenance than benefit

There are zones with negative returns — tests that cost money forever and rarely catch anything. They are more dangerous than not testing at all, because they undermine both trust and speed.

The first area is trivial code without logic: wiring, pass-through, simple accessors. A test for it essentially checks that the language works. The second is volatile, exploratory code that changes faster than the test — you rewrite the test three times before it ever catches anything. The third and most expensive area is tests tied to the implementation rather than the behavior: mock-heavy tests that check how something happens, not what comes out of it. They break with every restructuring without catching a single bug — they punish exactly the improvement you wanted to encourage. And finally, testing third-party and framework behavior: what you don't own, you don't need to test; you trust it or encapsulate it.

The worst side effect of these zones is the flaky test — one that is sometimes green, sometimes red, without anything having changed. A single one that the team learns to ignore devalues the entire suite: when red no longer means anything, no test protects you anymore.

The recommendation: deliberately forgo tests in these zones, and where a test is needed, check behavior rather than implementation. The price is that the coverage number drops and you have to explain why that is a good thing. We decide otherwise for trivial code that has an expensive side effect — a simple adapter that nonetheless moves real money; then we test not the triviality but the expensive effect behind it.

The line between worthwhile and harmful can be condensed:

Unit vs. integration vs. end-to-end

The three levels are not a ranking but a trade of cost against confidence. Each checks something different, and each lies in a different place.

The unit test is cheap and fast, but it lies as soon as it mocks away the integration that actually breaks. The end-to-end test gives the highest confidence that the whole thing works, but it is slow, unstable and expensive to maintain. In between sits the integration test, which checks the seams — database, queue, third-party service — where most real bugs actually live. In practice it often has the best ROI, and in practice it is the one most often neglected.

Between the levels lies a decision that determines value or harm: how much you replace with stand-ins. A test double — a mock, a stub — buys speed and isolation, but every stand-in is an assumption about how the real part behaves, and that very assumption can be wrong. Mock away the database, and you are testing your own idea of the database, not the database. The rule of thumb again follows the economics: stand-ins for slow, expensive or non-deterministic edges; real dependencies where their interplay is exactly what carries the risk. The more you mock away, the faster the test — and the less honest.

Diagram: The shape follows the risk, not the dogma — the common inversion (many E2E, few unit tests) is the anti-pattern.

The recommendation: shape the mix according to where the risk lies, and give the integration level more weight than folklore suggests. The price is that integration tests need a realistic environment — a real database, a real queue — which costs setup and runtime. We decide otherwise for an application whose risk lies almost entirely in pure logic and whose seams are trivial; there the weight rightly shifts down toward unit tests.

Characterization tests

Sometimes you face code that nobody understands anymore but that still has to be changed. A special kind of test helps here: the characterization test. It does not describe what the code should do, but records what it does — including its quirks. It is not a quality verdict but a safety net: if the behavior deviates after a change, it fires before production does. It is often generated by recording the outputs of the existing code across many real inputs and comparing against that recording on every change — a comparison that makes deviations visible without anyone having to formulate the expected result by hand. That is exactly the gain: you can safeguard behavior you do not yet understand yourself.

Economically, characterization tests are peculiar. Their value at the moment of change is very high — they make a rebuild possible that would otherwise be too dangerous. But they are scaffolding, not foundation: once the old code has been replaced, you can often remove them again, because they lock in behavior that is not meant to stay. You build them for the rebuild, not for eternity.

The recommendation: put poorly understood code under characterization tests before you touch it, and treat these tests as temporary scaffolding. The price is that you invest effort in tests whose tested behavior you intend to replace. We decide otherwise when the old behavior is demonstrably wrong and nobody relies on it — then you don't freeze the bug in place but correct it deliberately and document the deviation.

Testing during modernization

When replacing a legacy system step by step, tests are the decisive safety net — but only if you use them correctly. The temptation to "first retrofit tests for everything" is expensive and mostly futile: you invest in tests for code you intend to replace anyway. The right approach is to put exactly the slice you are moving next under test — not the whole system.

Two things are economically decisive here. First, separation: migration and behavior change stay separate, otherwise no test can tell anymore whether a deviation is intentional or a bug. Second, parallel operation as a runtime test — running old and new together and comparing the results before the new one becomes authoritative. That way correctness is proven on real traffic without writing a large test suite up front.

The recommendation: during modernization, test exactly the slice that is moving, and use parallel operation as an ongoing comparison. The price is double execution for the duration of the comparison and the discipline not to mix migration with improvement. We decide otherwise where double execution would have side effects that cannot be safely repeated — payments, for example; there you compare against recorded traffic instead of live.

Testing asynchronous systems

Asynchronous systems are the hardest case, because you cannot assert the result synchronously: you send off a message, and at some point something happens. Test that with long end-to-end tests and sprinkled-in wait times, and you get exactly the unstable tests nobody takes seriously anymore. The expensive bugs in asynchronous systems don't sit in the success case anyway, but at the edges: duplicate delivery, retries, ordering, a receiver going down.

The economic approach breaks the problem down. You test each receiver in isolation with representative messages; you check the message contract so that a sender cannot silently break the receiver; and you specifically test the expensive paths — idempotency under duplicate delivery, behavior on retry and failure. Instead of relying on wait times, you make time and delivery deterministic in the test. Across the entire asynchronous chain, you test only the one critical path, not every combination.

The recommendation: test receivers, message contracts and the expensive failure paths individually and deterministically, instead of checking the whole asynchronous chain end to end. The price is that no single test gives full certainty that "everything works together" — you buy that with a few targeted integration tests on the critical path. We decide otherwise for synchronous processing, where the result is known immediately; there the simple, direct test is the cheaper choice.

Common mistakes

The same patterns keep turning the return negative:

• Coverage as a goal — an input metric that becomes an end in itself and rewards brittle tests.
• Tests tied to the implementation that break with every restructuring without catching bugs.
• The inverted pyramid: many slow end-to-end tests, few fast unit tests.
• Tolerated unstable tests, until red means nothing and the entire suite is ignored.
• Testing third-party and framework behavior you don't even own.
• No tests where money, data or security are at stake.
• Indiscriminately retrofitting tests onto legacy code instead of safeguarding only the slice you are changing.
• Testing asynchronous code with wait times and wondering about flakiness.
• Deleting a red test instead of fixing the cause.

Decision checklist

Questions a team can ask before writing a test. They are diagnostic questions, not verdicts.

• Would a bug at this spot be expensive — money, data, security, anything irreversible? The value of a test rises with the cost of failure.
• Will this test still hold next month, or does the behavior change constantly? Volatility eats the return.
• Does the test check behavior or implementation? Whoever checks the how pays for every restructuring.
• Is this the cheapest level that catches the risk — unit, integration or end-to-end? Not every assurance needs the most expensive test.
• Does the real risk sit at the seams — and are we testing them enough? Integration is the most often neglected.
• If a test is unstable: do we fix the cause or delete it — instead of ignoring it? One tolerated flaky test devalues them all.
• For inherited code: is the current behavior pinned down before we change it? What you cannot observe, you cannot safely change.
• Are we measuring the value — change made safe — or an input metric like coverage? The target is risk, not percentage.

FAQ

What test coverage should we aim for? None in particular. Coverage measures how much code is executed, not how much risk is covered — a high number over inconsequential code buys nothing. Aim for covered risk, not a percentage; the number is a by-product, not a goal.

Is test-driven development mandatory? No. TDD is a workflow, not a seal of quality. It helps where you can clearly formulate the behavior up front, and it gets in the way where you are still searching for the design. Use it as a tool, not a commandment.

Don't tests slow us down? Bad tests do — the ones tied to the implementation that break with every change. Good tests speed you up because they make change safe: you change things faster when a safety net tells you immediately whether something expensive has broken. The difference lies not in the quantity but in the selection.

How many end-to-end tests do we need? As few as it takes to cover the critical user journeys. End-to-end tests are expensive and unstable; their value lies in the certainty that the most important journeys work as a whole — not in checking every combination. Everything below that belongs on cheaper levels.

Should we retroactively test legacy code completely? No. Completely retrofitting tests onto code you intend to replace is a lost investment. Use characterization tests to safeguard exactly the parts you are changing next, and leave the rest until its turn comes.

How do you test eventual consistency? Not with wait times and long chains, but broken down: receivers in isolation, message contracts, and specifically the expensive failure paths — duplicate delivery, retries, outages. You make time and delivery deterministic in the test instead of hoping for the right timing.

Further reading

• Software that still runs in ten years — why the value of a test lies in the change it makes safe.
• Legacy modernization without a big bang — characterization tests and parallel operation as a safety net during a rebuild.
• API versioning without breaking customers — contract tests that catch a break before the customer does.
• Modular monolith vs. microservices — where boundaries run, and therefore where integration tests come in.

It is grounded in the Batunet Engineering Method: test where bugs are expensive, prove the hardest path first, design for the failure case.

A test is neither proof of diligence nor wasted effort. It is a bet that a particular change should stay safe. Good teams bet rarely — but when they do, they bet right.