A reproducible contract is more convenient than repeatedly explaining to an agent what must not break. But the number of automated checks does not establish their quality.
Consider an example built around Make targets, CI, and contract tests. A check can gradually stop protecting system behavior and start locking down one implementation. LLMs can accelerate that transition: another edge case easily becomes another parsing layer.
It starts with a useful contract
The requirement is simple: a particular check must run through one canonical Make target. The target must exist, actually execute the required command, and be used both locally and in CI. There should not be another path that performs the same check differently.
Contract tests are a good fit for this.
They protect an engineering agreement rather than a specific line in a Makefile:
there is one public way to run the check
→ local execution uses it
→ CI uses it
→ structural changes must not create a bypass
The problem begins when verification models command text in excessive detail.
The test can turn into a command parser
A review can easily suggest another hypothetical way to bypass the check.
What if the required name only appears inside echo?
What if the executable is referenced by its full path?
What if the command is invoked through xargs?
What if Docker receives a different --entrypoint?
What if two commands share the same prefix?
What about different quoting styles?
Each question looks reasonable on its own. But answering every question with more string parsing gradually gives the test its own model of shell and Docker commands.
It starts distinguishing arguments, calls, and plain data, recognizing different ways to invoke an executable, and handling syntactically different but equivalent commands.
The original requirement has not changed:
the required Make target must exist and run the required check.
Verifying a few lines of a Makefile now requires a small shell command interpreter. At this point, it is worth asking what the test actually protects.
Fragility can look like reliability
A brittle test is usually one that fails after a harmless refactoring.
In this example, changing quoting can require updating the test. An equivalent way to invoke the same command creates another parser case. Another valid Docker invocation means teaching the test to understand that form too.
The observable behavior has not changed.
The contract is not becoming stricter. The description of one implementation is. The more detailed that model becomes, the more tightly the tests couple the Makefile to its current shape, and the more expensive otherwise permitted changes become.
LLMs are not the cause, but they are an accelerator
The same thing can happen without AI.
LLMs simply make each additional layer of complexity extremely cheap.
A reviewing agent can invent five more ways to bypass the test in seconds. The implementation agent can support them just as quickly. The next review finds another set.
The loop looks like this:
find a theoretical bypass
→ make the test stricter
→ add more custom logic
→ find a bypass in that logic
→ make the test stricter again
Each iteration looks like an improvement. There are more tests, more scenarios, and a more convincing agent report.
Meanwhile, the question “what property of the system are we protecting?” starts to disappear.
An agent is very good at optimizing for a local task. If the task is “make this test harder to bypass,” it can keep strengthening the test long after doing so has stopped being a good engineering decision.
Return to the original contract
The solution does not have to be a better parser.
Make already knows how to read a Makefile and expand a target. Some custom logic can therefore be replaced with Make itself:
make --dry-run --silent <target>
If the target does not exist or cannot be expanded, Make can tell us. The test does not need to partially reproduce its behavior. Expanding a command and successfully executing it are still different checks.
Keep checks for properties that actually matter: public targets, the canonical path used locally and in CI, required dependencies, and real bypasses.
Checks for syntactic variations that do not change the contract need separate justification. Otherwise, helper markers, parsing logic, and mutations end up being maintained mainly to test the parser itself.
The goal is not fewer tests at any cost. It is clarity about what each test protects.
A contract test should not know more than it needs to
A good contract test does not describe the system implementation in maximum detail. It does the opposite: it leaves the implementation free to change.
If a command is written one way today and another way tomorrow, the test should fail only if an important property of the system changed with it.
I use a simple question:
If the implementation changes while the observable behavior stays exactly the same, should this test fail?
If the answer is yes, there should be a specific reason why that implementation detail is part of the contract.
Sometimes it is. For example, CI may genuinely need to call one canonical Make target because otherwise local and server-side verification can drift apart.
The exact quoting inside the command usually is not such a contract.
Mutation testing needs a boundary too
The number of artificial mutations that make a test fail does not by itself say much about the quality of that test.
A useful mutation should represent a real contract violation.
Removing a required command is useful.
Replacing the required target with another one is useful.
Creating a real bypass around the required check is useful.
Exhaustively trying syntactically different forms of the same shell command is not useful if syntax is not part of the contract.
Otherwise, the test suite starts protecting itself.
Constrain the agent’s tests too
Limiting implementation scope is not enough: do not refactor unrelated code, do not change neighboring components, add a regression test, run the required checks.
A test can exceed the scope of a task too.
I care not only about whether a test catches a bug. I also want to know why that bug belongs to the contract and whether the same property can be checked at a more stable boundary.
This matters especially during independent review by another agent.
A prompt like:
find a way to bypass this test
can easily start an endless cat-and-mouse game.
A better set of questions is:
what property is this test supposed to protect
→ does it actually protect that property
→ does it check anything beyond that property
→ can the same property be verified through observable behavior
That kind of review evaluates the quality of the contract rather than the creativity of the test author.
Overengineering can be automated too
If a requirement can be turned into a reproducible check, that is usually more reliable than leaving it as a textual instruction to an agent.
But there is a second half to the rule.
A test is also an engineering decision. It can be overengineered, coupled too tightly to an implementation, and eventually maintained for its own sake.
AI agents make writing tests cheap. They do not make deciding what is worth testing cheap.
That is why the number of automated checks is not a reliability metric for me on its own.
A good test should be strict enough to fail when the contract is broken and loose enough to allow everything else to change.