v2 topic ranking, backtested to frequency-only

See overview and v1's prediction engine, which this replaces for v2.

What it does

rankTopics scores a subject's topics as of a cutoff date, over pure occurrence rows: no database, no model call, no writes. Only occurrences strictly before the cutoff are used, and a topic first appearing at or after the cutoff is never ranked. The strong claim is topic-level recurrence, measured at roughly 89 percent; a separate, weaker question-level ordering within a topic caps out around 14 percent and is kept structurally apart so the weak signal never contaminates the strong one.

A ranked-topics screen shows the true sittings a topic was asked in ("asked in 11 of 23 sittings") rather than a computed score, verified against the live corpus (Hand & Wrist ranks first).

The backtest and its verdict

A harness (npm run backtest) grades the ranking against a held-out year and sweeps the weights, tuning on ten subject-cutoff cells (every cutoff with at least three prior exam years, so trend has a history to split) and validating only on 2025, which the sweep never sees. On 2025, frequency-only validated at 7.67 (topic recall@10) against 6.00 for the hand-tuned weights and 7.33 for the sweep's own tuning-best (which scored 5.84 on the tuning set itself). Per the harness's own rule, a tuning-best that does not beat frequency-only on the held-out year is not an improvement, so the verdict is: ship frequency-only. The other signals (recency, trend, volume, paper) remain computed and named in the code, zeroed in the shipped ranking, so a future re-sweep can turn them back on.

The objective is the mean of three per-subject means, not a flat average over the ten cells, since Anatomy supplies five of the ten tuning cells and Biochemistry two; a flat average would quietly optimise for Anatomy.

What the review caught first

An initial version left frequency as an unbounded weighted count while every other signal was bounded to [0,1], so the unbounded term dominated by construction: at a 2025 cutoff the engine's output was identical to a frequency-only baseline for Physiology's top 10. Normalising frequency to a bounded share of past sittings fixed the scale problem, but did not change the eventual verdict once backtested.

A separate cutoff-leak bug mapped a year-only cutoff's unspecified sitting to the supplementary slot, so {year: 2024} sorted 2024's own regular sittings as "before the cutoff" (94 occurrences leaked at that cutoff, and all ten tuning cells were affected). Fixed by mapping absent cutoff fields to the minimum rather than the maximum.

Why topics, not questions

Measured against the real bank on 2026-09-07, holding out 2025 and using only earlier evidence: 14 percent of 2025's questions had appeared before, 89 percent of its topics had, and 100 percent of its chapters had. Chapter is therefore useless as a discriminator, and a question-level ranker has a hard recall ceiling too low to be the product's headline claim. Topics rank first with questions ordered inside them as a tiebreaker, kept structurally separate so the weak question-level signal cannot contaminate the strong topic-level one. Near-duplicate question clustering was scoped out of this work (deferred to a later embeddings-based merge) after measuring that it moves the question-level rate only from 14.3 to at most 17.3 percent even at a reckless similarity threshold, well short of the topic-level number.

Why this settles an open question

v1's rule set was never backtested. v2's version was built, then graded against a held-out year the tuning never saw, and the honest answer was that the extra rules did not earn their place over counting.

Status

Shipped 2026-09-08 as part of v2's topic ranking and ranked-view screens.