Independence, not more evidence, catches false done claims

Three separate times on Clarity, a "done" claim was false despite having tests, a live check, or a dashboard behind it. A commit that made an action atomic and conditional was correctly tested for both properties, while the actual flaw was an authorization gap: a caller supplied an identifier that was never proven to belong to the record. A commit preserving an ownership invariant passed a single threaded test while the real bug was a race between concurrent removals, which that kind of test cannot observe. A planning document claimed a metrics dashboard was collecting real usage, while querying the database directly showed every event came from a demo account.

The rule

The party producing the evidence also grades it. Tests written by the implementer confirm the implementer's mental model; they cannot falsify it. More evidence does not fix this. Independence does.

How to apply

• Never confirm a status by reading another document. Query the database, read the source, run the command.
• Write acceptance tests before the build starts, named in the work item, so the builder's job is "make these named tests pass" rather than "fix it and convince yourself".
• On any server action that takes a client supplied identifier, require a test that proves a foreign identifier is rejected, not that a valid one works. On any read then write, require a test that proves two concurrent callers cannot both pass the read.
• Have a verifier invoked by planning or architecture, never by the builder. A verifier the builder spawns and self reports on is self attestation with extra steps.
• Read real usage figures via a scheduled job, not asserted by whoever last touched the feature.
• Treat a merged migration as the record of what happened, not as proof of what is running in production.

This is not a coding lesson, it is a lesson about building with agents: when the same actor both builds and verifies, the result is confident, well tested, well documented, and wrong. See Clarity for the source project.

Corroborated on EaseMySemester v2

The v2 workflow separates the roles explicitly: a review step runs against the last landed commit and the issue's own "done when" clause, never against the diff alone, and every finding it reports is labelled as verified by running or inferred by reading, never left ambiguous. The review step exists specifically because reading a diff and pronouncing it fine converts an unchecked change into a blessed one. Every real defect it caught in that codebase, a stale session surviving a revoked login, an unguarded route, a data leak at a cutoff boundary, a length cap on the wrong field, was found by running something, breaking a specific line and watching a test fail, querying the live database, comparing against real generated output, never by reading carefully. Findings feed a separate fix brief for the builder rather than a second round of the same review, keeping the two roles apart end to end.

Tested again on Clarity 2: what independence buys when it is only context, not a vendor

Clarity 2's Maker and Checker started as two different models. They were later collapsed into one model, separated by context instead: the checker runs fresh, never having seen the build, and is handed only the acceptance criteria, the diff, and the test output. This still catches carelessness, drift from the stated criteria, and a claim nothing actually tests. What it does not catch, recorded explicitly rather than assumed away, is a blind spot the model holds in both contexts at once: two runs of the same model sharing one misjudgment will not disagree with each other about it. The remaining backstop against that failure mode is not a merge gate, since Clarity 2 also removed the last human-reviewed merge above each epic; it is the clinic session and a person watching the running app decide whether the built thing is the right thing at all, which is a different failure mode than a wrong build of the right thing. See Clarity 2.

The mobile PWA build's review register (issue #59) sharpened the rule further: a test can hold the wrong thing. A guard asserting a CSS class name in a string once certified a touch target as 44px wide when the class was never generated by the build, and it measured 23px in the rendered DOM. The same shape recurred four more times across that build, a shipped inline script diverging from the tested copy of it, a call-site literal untested even though the function it called was, a shared layout's chrome rule correct but its wiring across client-side navigation untested. The sharper form of the rule: a test for a rendered dimension, colour, or position must assert the computed value, not a proxy for it, and a pure function extracted for testability must have its caller covered too, or the test proves the rule without proving the product follows it.