Clarity's eval suite and the 86 to 100 investigation

Clarity's eval suite runs nightly and fails the build on a miss. It tracks a programmatic faithfulness gate that extracts every numeral from a narration and asserts it traces to a field in the input bundle, a golden set of 100 metric bundles spanning flat, sharp swing, sparse and zero division edges from a fixed seed, and an LLM judge for tone that is explicitly secondary and non blocking. Faithfulness pass rate, cost per narration, and p95 latency are tracked as things that can fail a build, not charts someone remembers to check.

The investigation

The first full run against all 100 bundles came back clean on infrastructure and narration failures, but scored 86 out of 100 on the faithfulness gate. The failing numbers in the sparse category were values like 50, 33.33333333333333, 75, and 66.66, exactly what dividing two small integer counts produces. The sparse generator creates periods with one to four transactions, so a model computing count over total lands precisely there.

The system instruction said every number must come directly from the JSON, but never forbade the model from deriving a new number by doing arithmetic on numbers that were individually real. The bundle already ships precomputed percentage fields, so the model never needed to compute them itself, and the trend module deliberately sets month on month change to null when there is no prior baseline, which read to the model as an invitation to compute a substitute.

The fix added two explicit rules: copy every number verbatim from a field, never derive it; and when a percentage field is null, omit the comparison or describe it without a number, never compute a substitute. The next run scored 100 out of 100, with judge scores for tone unaffected.

Why it matters

"Every number must come from the JSON" is not the same instruction as "never do arithmetic." A model can satisfy the letter of the first, because every number it states did originate somewhere real, while violating the intent, because the number it reports is one it computed rather than one the engine produced. This failure is more dangerous than outright hallucination because the model is doing arithmetic on real numbers, which is harder to spot by eye. See deterministic narration.