Import integrity: edited questions and malformed tags

See overview and content contract.

Edited questions were silently stranded

Import was insert-or-ignore, so correcting a question's wording inserted a second, near-identical question with no occurrences, left the original in place, and stamped the file's content hash so it was never revisited. The near-duplicate detector already existed (pg_trgm at 0.9) but had no resolution path.

The fix refuses the whole file on an unresolved edit candidate, the same treatment already given to a parse skip, with --force to opt out and a rollback that undoes the hash stamp so the file stays re-importable. A new resolveQuestion command rewrites one body in place, keeping its topic, position and occurrence history; it is never auto-applied, because two genuinely different questions in a topic can score 0.95 similar and a wrong merge silently destroys history. A reportAbsentQuestions command lists bank questions missing from their source file without deleting them.

pg_trgm measured wrong on this corpus. Because the question bank is written from templates, trigram similarity measures the template rather than the question: the highest-scoring pair across the 809-question bank was a false positive, and a true duplicate scored lower than two false positives around it. No threshold separates them. Embeddings, already available from the ingest pipeline, compare meaning instead and were proposed as the better instrument, though the merge decision still needs a human either way. Measured effect on ranking: crediting near-duplicates as one question moves the 2025 question-level repeat rate from 14.3% to at most 17.3%, against an 89% topic-level ceiling, small enough that it did not block topic ranking.

Truncated appearance tags were undercounted

The tag detector required four tokens before it would treat a group as a tag attempt, so a truncated real tag like (AKU 2025), two tokens, was never flagged and stayed in the question body: present in the bank but invisible to ranking. The shorter the truncation, the more likely it was a real tag, which is exactly backwards from what the old check caught.

The fix keys on shape instead of field count: a group is a tag attempt when it starts with an all-caps institution code followed by a four-digit year. Sub-part markers and parentheticals like (i) or (see diagram) still pass through untouched.

Verified against the full corpus: parsing all six source files with the fixed detector reports zero skipped lines and 910 appearance tags, against 909 occurrences in production, the single difference being a duplicate tag collapsed by a uniqueness constraint on occurrences rather than a missed truncation.

Status

Both shipped, 2026-09-09 and 2026-09-10.