Semantic Search

What it is

Search over my pdf textbooks, running on a CPU only server. Meaning search and keyword search at the same time, fused, optionally re-ranked, with a local model writing a cited answer from what came back. No GPU, no account, and nothing leaves the box unless I point it at a cloud model on purpose, to have something to measure the local ones against. Double-click a file and a browser opens.

It's also turned into the thing I didn't set out to build: a bench for measuring retrieval. 101 recorded runs, filterable by which embedding model, reranker and LLM produced them.

Two decisions on the other two projects are sitting on it right now. Whether Clarity's monthly narration can be served off my own hardware instead of a cloud API, and whether EaseMySemester can answer a student's question with a citation on a CPU-only box. Both are ship or don't-ship calls, and neither is one I can settle by reasoning about it. That is what this page is for.

I didn't build this to learn RAG

I built it because I couldn't find things in the textbooks.

I had a pile of PDFs and a real problem: I knew something was in there and keyword search wouldn't find it, because I remembered the idea and not the words. That's the whole origin. RAG is just what this arrangement is called. it wasn't the goal, it was the shape the answer took.

That distinction turned out to matter, because it meant I cared whether it actually worked rather than whether I'd built one.

So I built the measuring part first

Before I tuned a single retrieval setting, the thing could already save a set of questions and replay them through any combination of models, recording every timing.

That was deliberate. Four parts:

• Decide what to measure. Speed, to begin with, because it is honest and cheap. Answer quality left out on purpose, because scoring quality needs ground truth I didn't have.
• Make it repeatable. Saved question sets, persisted results, run ids.
• Make it comparable. Filter by model combination, average the set.
• Watch it while it runs. Live progress, per-question rows, a stop button.

Choosing speed first and writing down why quality was deferred is the difference between shipping a measurement and shipping a number that flatters you.

Being able to see why something surfaced

Every retrieval setting is exposed, and the page shows what each retriever actually found, rank and score per result, because retrieval you can't inspect is retrieval you're tuning by feel.

The settings are how many candidates each retriever contributes, the weight on each, the RRF constant, minimum similarity, which reranker and how many candidates it re-scores. The results are broken out per source, per result.

The screenshot above is the argument for doing it this way. The same passage sits at rank 2 for meaning search and rank 1 for keyword search, on scores that aren't even on the same scale. Either one alone puts it in the wrong place. The fusion is what sorts that out, and you can watch it happen instead of taking my word for it.

Questions I asked myself before building the runner

Before building the runner I wrote down five questions and let the answers decide its shape. The one I'd defend hardest is treating a failed question as a row rather than an abort: a run that dies on question 7 of 20 teaches you nothing.

Question	What I settled on
Where do the test questions come from?	Editable, saved question sets, not a file baked into the code
One model combination, or sweep them all?	One per run. Comparable, and simple to hold in my head
Speed or quality?	Speed. Quality deferred, on purpose
What if a question fails halfway through?	Record the failure and carry on, because a failed question is data
How much control in the runner?	Models, k, rerank pool. Fusion weights come from the Tuning tab

Five defects, one of them a deadlock

A review turned up five concurrency problems. I fixed them by blast radius, not by the order I found them.

The deadlock went first. Clicking Start twice took the whole server down, and it never came back on its own. Nothing else compares to that.

Then correctness and typing, before letting requests actually run at the same time. Making something faster before making it right just gets you the same bugs at higher speed.

The change anyone would actually notice, getting the blocking work off the event loop, went last. On its own it would have shown no speed-up at all, because everything was still queued behind the same lock. It would have looked like a failed optimisation and I'd probably have reverted it.

That last one is the bit I think about. I ordered the work partly so the measurement wouldn't lie to me. I also wrote the project's first tests, 16 of them, alongside the fix.

I reproduced the failure instead of guessing at it

Runs on the server started dying mid-answer with "peer closed connection".

Easy to guess from that. I pulled the same model down locally, ran the identical code path, and it worked, which told me the code was fine and the environment wasn't. The model runtime on the server was running out of memory. Then I added retries with backoff and gave myself the exact things to check next time: memory, container logs, evidence of the process being killed.

Separating "my code is wrong" from "the box is wrong", with evidence rather than a hunch, is most of what debugging something in production actually is.

The parts I left out

No sweeps across every model combination. One per run, so results stay comparable and the manual loop is proven before I automate it. No answer-quality scoring yet, with the path written down for when I add it. And a locking issue I found during the concurrency work is filed rather than rushed, because it needed more care than that change had room for. Each with the reason recorded at the time, so I can defend it later or change my mind honestly.

Scoring answers, not just speed

Answer quality is the thing I deliberately left out, and it is what I would measure next. The plan is ground-truth question sets with the book and page per question, scored on hit rate and ranking, then a model as judge as a cheaper approximation once the expensive version has told me what good looks like.

Where this can still fool me

I'd pin the question set and the model versions to every run. Right now a run records which models it used but not which revision of the questions, which means comparing runs across a long enough stretch of time is quietly untrustworthy. It's the one place where the thing I built to stop me fooling myself can still fool me.