Brief · 2026 · Product and engineering

Semantic Search

A local, offline hybrid search and RAG evaluation workbench over PDF textbooks: semantic plus BM25, fused with Reciprocal Rank Fusion, every run benchmarked and comparable.

Problem: No way to tell whether a retrieval change helped, which makes every setting a thing to argue about rather than a thing to compare.
User: One user, and that user is me. It is the bench the other two projects get judged on.
Decision: Measure first. The benchmark runner existed before a single retrieval setting had been tuned.
Number: 5 concurrency defects fixed in blast-radius order, deadlock first. The headline throughput fix went last on purpose, so the measurement could not flatter the fixes underneath it.
Refused: No sweeps across model combinations, no answer-quality scoring yet, and a locking issue filed rather than rushed into a change that had no room for it.

5 defectsfixed in blast-radius order
0 GPUsCPU-only, embeddings included

No demo for this one. It runs on my own hardware with my own books on it, and it is the bench I test the other two projects against rather than something built to be handed out.

The full case study is about 6 minutes on how it was decided, measured, and where it can still fool me.