Why Quality Gates Need Tests Too

The ashikov.ru repository has a single make check command. It runs the mandatory checks and builds the site. While reviewing those checks, I found two cases where they could finish successfully without fulfilling the contract they were supposed to enforce.

In one case, commitlint accepted an invalid commit message because an ignore rule was too broad. In another, the script that validates file endings could treat a failure to obtain the file inventory as if no violations had been found.

The fixes were small. But they also required regression tests for the checks themselves. A green result on a healthy repository did not show whether a gate knew how to fail when it should.

A Git error became a successful check

The file-ending script obtained the list of tracked files from git ls-files and passed it into a loop through process substitution. If the actual ending checks are removed and only inventory collection and counting are left, the structure looked like this:

set -euo pipefail

checked=0
while IFS= read -r -d '' entry; do
  checked=$((checked + 1))
done < <(git ls-files --cached --stage -z)

printf 'Checked endings of %d tracked Git text files\n' "$checked"

With a broken Git index, git ls-files exited with an error. The loop received no records, the counter remained zero, and the main script printed a success message and returned exit code 0.

The bug was not in the newline validation itself. It happened earlier, while preparing the input.

The <(...) construct runs the producer asynchronously. In this form, the exit status of git ls-files did not become the exit status of the main script. set -euo pipefail did not provide the required error propagation.¹

An empty inventory is not necessarily an error. A repository may genuinely contain no matching files. But “the inventory was obtained successfully and is empty” and “the inventory could not be obtained” are different outcomes.

Input collection became a separate step

Instead of process substitution, the file list is now written to a temporary file:

inventory=$(mktemp)
trap 'rm -f -- "$inventory"' EXIT
git ls-files --cached --stage -z >"$inventory"

The loop then reads the completed file. With set -e still active, a failing standalone git ls-files command stops the script before any content checks or success message.

The regression test creates a temporary Git repository and then points GIT_INDEX_FILE at a broken index. The script must return a non-zero exit code and must not print Checked endings.

This tests the required behavior rather than the shape of the shell code: failure to obtain the input must not turn into a successful check.

The fixup exception was too broad

The commitlint configuration had a different bug:

ignores: [(message) => message.includes("fixup!")],

A function in ignores can skip the whole message.² But includes searched for fixup! anywhere, not only at the beginning.

So the exception also matched an invalid subject such as:

invalid commit mentions fixup! in prose

Mentioning fixup! even in the body was enough to bypass validation of the subject. An exception intended for one kind of helper commit applied to unrelated messages.

The condition was narrowed to:

ignores: [(message) => message.startsWith("fixup!")],

The tests now cover both sides of the boundary. A real fixup! commit must pass, while an invalid subject that only mentions the marker in ordinary text must fail. A valid docs: explain fixup! commits message is also accepted: mentioning the marker is not itself a violation.

A test should prove the expected reason for failure

Checking only for a non-zero exit code is sometimes insufficient. A command may fail to start, fail to load its configuration, or stop for a reason unrelated to the rule being tested.

The commitlint regression test therefore checks both the exit code and the type-empty diagnostic for an invalid subject. The broken-index test requires a Git error and separately forbids the final success message.

Positive cases matter too. A check that always returns an error will reject every bad input as well. The test suite keeps valid commit messages and files with valid endings.

That gives three distinct outcomes:

Test behavior, not the implementation of the check

In “When a Contract Test Knows Too Much” I described how tests started parsing shell-command details instead of checking the contract. This case did not need another parser.

The tests run the real script and the real commitlint with the project configuration. They compare the observable result with the expected one. The implementation can change as long as valid input is accepted, violations are detected, and execution failures are not masked as success.

These scenarios are part of make test-checks, which in turn is included in the common make check. They are not a separate command that somebody has to remember after changing a gate.

The two defects produced false success in different ways. One check lost an error while preparing its input. The other skipped validation because its own exception was too broad.

So a successful run is no longer enough evidence for me that a mandatory gate is sound. I also want examples on which it must fail, and proof that it fails for the expected reason. “Could not check” must never mean “checked, no problems found.”