The control beat my clever version

The topic ranker in EaseMySemester predicts what is likely to come up in an exam. It had recency, gap and momentum signals sitting on top of a frequency count, and weights I had tuned by hand until the output looked right.

I finally graded it against the obvious alternative, which is counting, with none of that logic on top.

Counting scored 7.67. My hand tuned weights scored 6.00. The best weights the sweep could find for itself scored 7.33.

It ships as frequency only.

What the harness does

It grades a ranking against an exam year it has never seen. It tunes on every subject and cutoff with at least three prior years behind it, ten of those, and validates only on 2025, which the sweep never gets to look at.

That last clause is the whole design. A ranking tuned and scored on the same data will always look good, and I would have had no way to tell the difference between a ranker that works and a ranker that has memorised.

The rule was written before the run: a tuned result that cannot beat plain counting on the held out year is not an improvement. Writing it down first is the only reason it survived contact with a result I did not want. A rule you invent after seeing the number is not a rule.

The other signals are still computed and named in the code, zeroed out, so a later sweep can switch them back on if one ever earns it. I did not delete them, because the finding is that they did not earn their weight on this data, not that they are worthless.

Two things nearly let me believe otherwise

Frequency was unbounded while everything else was squeezed. Every other signal was normalised into a nought to one range. Frequency was left as a raw count. So it dominated by construction, and on one subject the tuned engine was already handing back the baseline's answer while reporting itself as tuned. Nothing on screen would have told me. The ranker and the control were the same thing, wearing different labels, and the comparison I thought I was running was not being run.

The cutoff leaked. Asking the harness for 2024 let 2024's own sittings count as evidence from before 2024. Ninety four occurrences, in every one of the ten cells I was tuning on. Not a rounding error, and not confined to one corner of the data.

A harness with a leak in it is worse than no harness, because it still produces a number, and the number has my name on it. Without the harness I would have known I was guessing. With a leaking harness I would have been confident.

Both of these were found by looking at the intermediate values rather than the score. The score looked plausible throughout. It is the only reason I check them now as a habit rather than when something seems off, because nothing seemed off.

What this is not

This is not the same measure as the 40% on the front of my site. That was 20 questions scored against the 2026 paper. This is topics, scored on recall at ten, against 2025. The question level control on that same paper is still unrun, and until I run it, the 40% has no baseline next to it either.

I would rather say that than let two different measurements borrow each other's credibility because they both came from the same project.

The finding that changed the product

Holding out 2025, 14% of its questions had been asked before. 89% of its topics had.

That is the real result, and I only have it because I stopped arguing about ranking rules and graded them. A question level ranker has a ceiling too low to carry this product. A topic level one does not. The ranking moved to topics because of that number, not because topic ranking was a nicer idea.

What I took from it

Write the control first. Not after, when it is a thing you are doing to confirm a result you already like. The dumb version is the product until something beats it on data neither of you has seen.

A tuned number and an honest number are different objects. I had spent real time on those weights. The sunk cost is not evidence.

Check whether the comparison is actually happening. My control and my treatment were briefly the same code path with different labels on it. That is a failure mode no score will report, because the score is fine.

The caveat, as always: one subject area, ten tuning cells, one held out year, and I built both the ranker and the thing that graded it. That is exactly why the held out year and the pre written rule matter. They are the only parts of this that I could not quietly adjust once I saw what the answer was going to be.