Clarity

Where this is: the correctness gate and the eval suite are shipped and running every night. The narration itself is still being tuned. I'm not happy with the writing yet, and I'm using Semantic Search to work on it.

What it is

A dentist I know runs a small clinic. He ran it out of a spreadsheet for sixteen months. This replaced the spreadsheet.

About 900 real transaction records now, entered between patients, on a phone about as often as a laptop. One clinic, built for one clinic, though I'm talking to a few other dentists.

There's an LLM feature: it writes a short summary of the month's numbers. That feature sits behind a plain function that checks it, and 100 test cases that run every night and fail the build if it lies.

A wrong number is worse than no number

This is a ledger. The dentist makes decisions off it. If the summary says margin fell 12% and it actually fell 2%, that's not a bad user experience, that's a bad business decision with my name on it.

So the check isn't a score I keep an eye on. It's a gate. A plain function walks every number in the generated sentence, traces it back to the row it should have come from, and if any figure doesn't reconcile it throws the whole thing away and shows nothing. No model, however confident, gets a number past it.

The other half of that decision matters as much: I separated what blocks a release from what doesn't. The number check blocks. There's also a model scoring the tone of the writing, and that one never blocks. It just tells me whether the thing is getting better. A subjective score is the wrong instrument to hold a release hostage to.

Pass rate, p95 latency and cost per call all get tracked the same way, as things that can fail a build rather than charts I remember to look at.

The suite that caught me out

100 test cases across four kinds of failure, generated from a fixed seed so the suite is the same every run. It runs nightly and fails the build on any miss.

The first full run scored 86 out of 100, and the failures were real. The model was working out percentages itself instead of quoting the numbers I'd handed it. Prompt constraint, fixed, written up.

A suite that passes on day one has usually been written to pass. This one found something, which is the only reason I trust it.

I organised a whole phase around a number that was wrong

For a while, one figure ran this project: entry took about 32 seconds, against about 7 in Excel. It was at the top of my notes in bold. I wrote a design review called Thirty-Two Seconds.

Then I asked the dentist about it.

He said he likes the entry screen. He specifically likes the autofill. And his spreadsheet was faster because it never asked who the patient was. It was dropdowns and an amount box, doing strictly less work. I'd been comparing two different jobs and calling it a gap. Three measurements, on someone still learning the tool.

I dropped the metric and rebuilt the roadmap around what he actually said: too many tabs, "History" sounds like patient history and this isn't a patient records app, and he wants to keep notes between visits.

Not all of that phase was wasted. The search ranking fix was a genuine bug and the contrast problems were real and measured. But it was aimed by me reasoning about him rather than asking him, and one question settled it.

The habit I took from it: when a number seems to be deciding the project, check whether the person using it agrees before building anything on top of it.

Making 200 model calls a day cost nothing

Nightly evals plus real usage is roughly 200 calls a day, and this has no budget.

Chasing quota errors turned up something I hadn't assumed: the limits were per model, not per account. 20 a day on one model where the account-level number suggested 500. That turns a hard ceiling into a routing problem. So there's a chain of eight models it degrades through, and the nightly run finishes on its own without me touching it and without paying for a tier.

One bug worth keeping: a retired model name returns a 404, and my retry logic didn't recognise 404 as something to retry, so the chain gave up on the first model instead of falling through. Errors now carry which model failed and whether the failure was a quota problem or something else. A fallback chain is only as good as its ability to tell why a call died.

What I chose not to build

The tone scorer never gates a release. The PhonePe QR automation he mentioned in passing stays unbuilt, because he was clear it wasn't now. And the Ctrl-key shortcuts I shipped are coming back out, because he told me he doesn't use them and expects Tab and Enter. Though not by turning the amount buttons into plain tab stops, because reaching the fifth one would then cost five presses, which is worse than what it replaced.

What's next

Two things. Whether I can run the narration model on my own server instead of calling a cloud API, which is exactly the question Semantic Search was built to answer, so I'm using it as the bench.

And a sensitivity threshold. A clinic's monthly numbers don't move much. If nothing has meaningfully changed since the last summary, there's no reason to generate a new one. The cheapest call is the one you don't make.

What I'd do differently

I'd have asked the question that killed the 32-second metric on day one. It cost a phase of work, and the conversation that resolved it took about a sentence. Everything I built in that phase was competent and pointed at the wrong thing, which is more expensive than building nothing.

I'd also make the demo and production tenants impossible to confuse. I've already closed a month against demo data once. That's not a discipline problem, it's a design problem, and I should fix it in the interface rather than by being careful.