The measurement loop

Before any retrieval setting was tuned, the tool could save a question set and replay it through any model combination, recording every timing. The sequence is the point: decide what to measure, make it repeatable, make it comparable, make it observable while it runs.

What it records

Per question: timestamp, query, embedding model, LLM, reranker, result count, chunks served, semantic and keyword hit counts, retrieval time broken into embed/vector/BM25/rerank phases, time to first token, answer time, total time, token count, tokens per second, an ok flag, an error string, and a run id.

Why speed first

Answer quality scoring was explicitly deferred because scoring it needs ground truth that did not exist yet. Speed and latency are the honest, cheap first metric. This is a latency and throughput harness, not a quality harness, stated plainly rather than implied.

Scope decisions made before building it

Question	Decision
Where do test questions come from?	Editable, persisted question sets, not a hardcoded file
One model combination or a grid sweep?	Single combination per run, simpler and comparable across runs
Speed or quality?	Speed first, quality scoring deferred to roadmap
What happens when a question fails mid-run?	The error is recorded as a row and the run continues
How much control in the runner?	Models, result count, and rerank pool; fusion weights inherited from the tuning surface

Treating a failed question as a data row rather than an aborted run was the harder call to defend: a run that dies partway through teaches nothing.

Consequences

Grid sweeps and question-set versioning are both roadmap, not built. See known gaps and open questions.