Shared LLM client with retry, fallback, and a run budget

See overview, v2, and OpenRouter decision.

What it does

src/lib/llm.ts (issue #13) is a library, not a script: it takes an injected transport, so retry, fallback, and cost accounting are testable against a fake transport with no network call. The OpenRouter transport is the only place that does HTTP, replacing duplicated embed implementations that had lived separately in the ingest and eval scripts.

• A single retry with backoff on 429, 5xx, or a network error only; a 4xx fails immediately, names the model and status, and never burns a retry or the fallback.
• A fallback model fires once, after the primary's retry is spent, never on the first failure. The response records which model actually answered, so a silent fallback is not possible.
• RunBudget tracks cumulative tokens and estimated cost and throws at a configured ceiling, priced per caller (ingest passes its own per-million-token rate).
• Per-request timeouts (embedding 60s, completion 120s), so a stalled request cannot park a worker forever, which would have broken the resumable generation loop.
• A 200 response with no content, or no usage block, is treated as a protocol failure and throws, rather than being written as real content or silently costed as zero, which would have let a loop of empty responses defeat the run ceiling.
• Backoff honours a Retry-After header when present and otherwise jitters by plus or minus 25 percent, so concurrent workers hit by the same rate limit do not retry in lockstep.
• embed() batches internally (128 per call) and refuses to return a count that differs from the input, since a silent misalignment would attach the wrong embedding to the wrong chunk. Batching moved inside the client after the provider's 2048-element cap was hit live ingesting 2,502 Guyton chunks in one call.

Why a shared client

Retry, fallback, and cost logic had been duplicated across the ingest and eval scripts before this. Centralising it in one tested library, with the API key read only through a validated environment spec and never logged, in an error, or in a report, is what let issue #15's generation script inherit correct retry and budget behaviour for free.

Verified against live data, 2026-09-07

One real completion call (primary model, no fallback triggered). A full re-ingest of one book (Guyton, 2,502 chunks, 1,280,601 tokens) under a scratch config reproduced the original ingest's numbers exactly. npm run eval returned identical numbers across five consecutive runs after the client's embedding batching and retry changes landed.

Token budgets, revised 2026-09-10

Muse Spark 1.3, the model made default that day (model choice decision), reasons on every call, and those tokens count against max_tokens. The ask path first shipped with a 1200-token budget and had it consumed entirely by reasoning, returning content null; it now pins 8000 for one long answer. Generation now pins 10000, since it emits a Long Answer and its condensed form in one completion, so 8000 is the floor there, not the ceiling. Generation previously sent no max_tokens at all and took the provider default.

Status

Shipped 2026-09-07 (issue #13) across three commits: the initial client, a fix restoring per-caller retry counts and internal batching after a corpus rebuild broke reproducibility, and a final pass adding timeouts, malformed-response guards, and jittered Retry-After backoff. Token budgets revised 2026-09-10 for the Muse Spark default. Proven in production use by issue #15's generation script: 39 answers generated through complete(), primary model only, no fallback fired.