In “Public Explanation as a Test of an Engineering Solution”, I wrote about how reconstructing the actual order of events exposed a missing dependency in a CI/CD process.

That led to another problem. Writing accurate documentation once is not enough. The next pipeline change can make it wrong again.

In a DocOps project, I documented CI in a canonical document: which pipeline scenarios exist, which checks are mandatory, how jobs depend on one another, and which actions remain manual. I then turned some of those claims into contract tests.

The idea is simple: if documentation promises specific system behavior, some of those promises can be verified like any other contract.

Documentation does not become stale because people are lazy

CI configuration and its documentation have different lifecycles.

.gitlab-ci.yml is executed. If a change is syntactically invalid or breaks a scenario covered by verification, the problem can be detected automatically.

Documentation is not executed. The pipeline can change correctly, all checks can pass, while the old description continues to claim that a mandatory job still exists or that one stage waits for another.

From the author’s perspective, the task may genuinely look complete. The CI change works, verification is green, and the pipeline behaves as expected. Documentation simply was not part of the definition of done.

This is how documentation drifts away from actual system behavior.

A rule such as “remember to update the documentation” is a weak defense. Every author has to remember which documents depend on a particular configuration change.

Part of that dependency can be made executable.

The contract is not the entire document

The first dangerous idea is to try to prove automatically that the whole document matches the implementation.

That quickly makes verification brittle.

Paragraph order, wording, examples, and explanations are not part of CI behavior. If tests start locking them down, ordinary editing will break the build and the test suite will become a snapshot of a Markdown file.

I therefore selected only the claims that define the pipeline model:

  • which significant pipeline scenarios exist
  • which jobs belong to the mandatory verification path
  • which dependencies determine execution order
  • which jobs are manual or allowed to fail
  • where the canonical CI description lives
  • which related files must point to that document

These are no longer editorial details.

If the documentation says that a build starts only after mandatory checks, while the configuration allows it to start earlier, the documentation describes a different system.

If the README declares one document canonical but links to another file, that contract is broken as well.

The rest of the document remains prose. It does not need to become a test.

A contract test makes a changed guarantee visible

Without this kind of verification, a CI change can look like this:

change the pipeline
→ verify the behavior
→ get a green result
→ finish the task

A contract test adds another boundary:

change the pipeline
→ verify the behavior
→ verify documented guarantees
→ preserve the contract or change it explicitly

Suppose the documentation states that a build depends on the complete set of mandatory checks. A contract test verifies the corresponding dependencies in the CI configuration.

If one disappears accidentally, the test becomes red.

The red test does not know where the mistake is. The dependency may have been removed accidentally. Or the pipeline architecture may have changed intentionally, which means the old description should no longer be considered correct.

The check makes the disagreement visible. A person still decides which state is supposed to be correct.

That is the main value I found in this approach. Important documentation stopped being something I merely hoped to remember after changing CI. A mismatch between documented and actual behavior became an ordinary regression that could be seen before merge.

A green contract test can lie too

A contract test is not correct merely because it is called a contract test.

A simple link check made this especially clear.

One test was supposed to guarantee that the README linked to the canonical CI document. The first version effectively checked whether the expected path appeared somewhere in the link.

The test looked reasonable and stayed green.

But a value such as:

docs/ci-pipeline.md.invalid

also satisfied the check.

The test was therefore proving a weaker property than the one it claimed to verify. The README could point somewhere other than the canonical document while the test still passed.

After adding a regression case, the check had to become more precise. Accepted path variants are normalized, the URL fragment is separated, and the resulting path is compared with the canonical path as a whole.

The defect was small, but it exposed an important boundary.

Saying “we have a documentation test” proves nothing by itself. What matters is which violation can actually make the test fail.

If a check claims that a link points to one specific document, replacing that document with a similar-looking path must make the test red.

A contract test does not protect itself

There is another boundary that is easy to miss.

The author of a change can update the expected value in the test instead of updating the documentation. The check can be weakened. It can be deleted entirely.

No contract test can prevent someone from changing the test itself.

It therefore does not automatically guarantee that documentation and implementation remain synchronized. Its job is narrower: an accidental change in behavior no longer passes unnoticed.

A modification to the test becomes another signal for review. Changing the verified contract deserves the same scrutiny as changing the implementation.

For an intentional behavior change, I expect the diff to tell a consistent story across:

system behavior
+
verified contract
+
canonical documentation

Not all three parts need to change every time. If the documented guarantee still holds, the document should remain untouched. But if the guarantee itself changes, modifying the test without a corresponding documentation change requires an explanation.

The machine does not make this decision for the reviewer. It makes the decision point visible.

Do not write a second GitLab CI parser

Contract tests have an opposite failure mode: they can start modeling the entire .gitlab-ci.yml.

Then the project effectively gains a second implementation of the pipeline that also has to be maintained.

Every new job, renamed command, cache change, or internal detail starts requiring test updates even when the documented behavior has not changed. Noise increases, and a red contract test stops indicating an important problem.

I try to draw the boundary around an observable promise made by the documentation.

Adding an internal helper command may change nothing for the reader.

Removing a mandatory check does.

Rearranging technical details inside a job may be irrelevant.

Changing a dependency so that the build no longer waits for a mandatory check changes the pipeline model.

The same applies to execution conditions, manual jobs, and allowed failures when the documentation makes a concrete promise based on those properties.

A good contract test knows exactly as much about the implementation as it needs to verify that promise.

I now use the same principle more generally in my projects: verify the smallest stable observable contract.

Start by choosing the claims

Applying this approach does not require a universal documentation framework.

A better starting point is to open the CI documentation and find a few sentences whose being wrong would materially change how someone understands the process.

For example:

Claim:
the build starts only after mandatory checks

Behavior source:
job dependencies in the CI configuration

Verification:
all mandatory dependencies are present

Or:

Claim:
this job is manual and does not block the normal pipeline

Behavior source:
the effective job configuration for this scenario

Verification:
the job preserves the documented properties

Another example:

Claim:
the README points to the canonical CI documentation

Behavior source:
the link in the README

Verification:
the normalized path exactly matches the canonical one

A few guarantees are enough to start.

A good sign is that changing the corresponding behavior makes the test red.

A bad sign is that rearranging paragraphs or harmlessly editing prose does the same.

Not all documentation should be executable

Contract tests work well when documentation describes verifiable system behavior.

They are a poor fit for rationale, recommendations, examples, and ordinary prose. Trying to verify those mechanically usually creates more brittleness than value.

For me, “documentation as a contract” therefore does not mean turning Markdown into another programming language.

The idea is narrower.

If a document promises specific system behavior and that behavior can be checked in a stable way, the check belongs next to the other regression tests.

It does not guarantee that the documentation will always be complete or correct. It cannot prevent someone from intentionally weakening or deleting the test. It does not replace review.

It solves a different class of failure: the system has already changed while its canonical documentation still confidently describes the previous version.