One number got better and the other got three times worse

I changed one prompt in EaseMySemester and regenerated 809 answers. Declines on single part questions fell from 10.9% to 7.9%. Declines on multi part questions rose from 9.2% to 28.7%.

The first number is the headline. It moved the right way. The second number is what the change actually did.

What a decline is

The second version of the question bank answers rather than just ranks. Nine textbook volumes went in first, 7,731 chunks, just under four million tokens, embedded for about eight cents. The bank is 809 questions across 34 chapters and 218 topics, and every one of them got an answer for about 65 cents in total.

Some of those answers are the system refusing. A decline means the books in front of it do not support an answer, and that is the behaviour I want. A question bank that always says something the week before an exam is worse than one that admits a gap, because the student cannot tell the two apart and will revise from whichever one it produces.

So the decline rate is not a defect count. It is a coverage report. The useful question is not how low it goes, it is which questions get declined and whether that is defensible.

My prior on that was already wrong once. I assumed Biochemistry would decline most, because it leans on a single source. Biochemistry declines 6.3% of its questions. Physiology declines 16.2%.

The change, and the number I nearly shipped

The answers were coming back in the wrong shape rather than with the wrong content. An essay, where the exam wants five labelled points. So I wrote a per subject prompt overlay that states the shape an answer has to take, and regenerated all 809 with no errors.

Then I looked at the overall decline rate. 82 of 809, about 10%. Flat, roughly. Fine.

That is the number that would have let me close the ticket. It is an average over two populations that moved in opposite directions, and the larger one moved the way I wanted, so it came out looking like nothing happened.

Splitting it took one query.

What the change actually did

Telling the model what a complete answer looks like made it much stricter about admitting when it could only half build one. A multi part question is mostly where one sub part has nothing behind it in the books. Before the overlay, that question got a partial answer that read like a whole one. After it, it gets a decline.

So the honest reading is that the prompt did not break anything. It exposed a retrieval gap that was already there and was previously being papered over.

That reading matters, because the two readings lead to opposite actions. If the prompt broke something, you revert the prompt. If the prompt revealed something, you keep it and go fix retrieval. I would have taken the first action, on a stronger metric, and re-buried a real defect.

What I took from it

Segment on the dimension the change acts on. The overlay was about answer shape. Multi part and single part questions have different answer shapes. That was the split, and it was predictable from what the change was, not discovered by luck.

Name the segments before you run the change. This is the part I want to be careful about. If you go looking for a split after seeing a flat aggregate, you will find one, and you will find the one that flatters you. Writing down what you expect to move, and in which group, before you ship is the difference between a segment and a rationalisation. I got away with it here because the split was obvious. I would not always.

An average over two populations is not a measurement. It is a summary, and a summary is allowed to lose the thing you needed.

What I have not done

The fix is to retrieve per sub part instead of per question. It is obvious and I have deliberately not built it.

I want to know how many of those declines are genuinely unanswerable from the corpus first. If most of them are, per sub part retrieval is an expensive way to turn honest refusals into confident guesses, and I would be spending the retrieval budget to make the coverage report look better rather than to make the coverage better.

The caveat on all of this: one bank, one cohort, 809 questions, measured by me. It is not a benchmark. It is the only version of this I have run.

A feature that reports its own coverage honestly is worth more to a student the week before an exam than one that always says something. That is the whole reason the decline rate is a metric here at all, and it is also the reason the 28.7% is the number I care about rather than the one I wish I had published.