LLM Source Binding: Haiku Proposes, Code Verifies

Every lexical matcher has a honeymoon period. Ours lasted about a year. We shipped filename matching, then DOI matching, then title matching, then a content-pass that bound a document by looking for the entry's title inside the full text. Each pass fixed the last incident and added a new row to the matrix. Then came the week where a user uploaded a scanned chapter excerpt whose title page was missing, and the week after where another user named their PDF Teil-2.pdf when the bibliography line pointed at Teil 3 of the same handbook, and the week where the publisher's PDF title line said "A revised introduction to …" while the bibliography line said "Introduction to …". The matcher was going to need a sixth pass, and a seventh. A human reading the first page and the bibliography line would have settled every single case in under five seconds.

So we stopped adding passes. This is a writeup of the LLM source binding architecture we shipped on 2026-10-25 to the production worker inside Acurio, our citation-verification product for academic theses (the open-source codebase is zoterohero). The pattern reads short — let Claude Haiku 5.5 propose which uploaded document is the work a bibliography entry names, then run every proposal through a wall of deterministic Rust gates that refuse anything the model can't prove on the source card — but the interesting part is where the gates live and what they check for. Any team wiring an LLM into a decision whose output a user will read in a report, not a chat window, ends up building something close to it.

The Matcher That Kept Needing One More Pass

The lexical matcher was and still is the fast path. For most entries it is correct and the LLM never runs: an exact DOI match or an OA-fetch provenance stamp binds the source without any model touching it. Where it struggled was the long tail — chapter excerpts without a title page, title variants, series parts, the "cites but is not" trap where a monograph by the same author on an adjacent topic looks like it should be the source but is actually just the thesis's own literature review bleeding into the sources folder.

Each incident taught the matcher one more string normalisation. By pass five the normaliser was so permissive that it had started binding wrong documents — fuzzy matches good enough to fool the test suite but obviously wrong to any human reader. The matcher had turned into the problem it was built to solve: a strange configuration of heuristics that could not be reasoned about from the code.

This is the shape of problem an LLM is actually good at, with one caveat we had already written up in our production-failure playbook for AI agents: a model that answers "yes, that PDF is the cited work" and bills a user for it has to be auditable. The result the user reads in a report has to be defensible against a thesis reviewer the next day. "Because Haiku said so" is not a defensible answer. OWASP's LLM09 guidance on overreliance names exactly this failure mode: using model output as ground truth without a verification layer.

Haiku Binds, Code Checks

The architecture that came out of it is three layers. The first is a source card — a short, deterministic document generated per uploaded PDF, holding only fields that a human reviewer could cite in a disagreement: the filename, the fields software read from the first page, the lines that repeat on many pages (running heads and footers), DOIs seen on the first pages, and the opening sixty-thousand characters of the text. The source body itself never reaches the model. The card does.

The second layer is one batched call per twenty-five cards, carrying every open bibliography entry of the run. The model — Claude Haiku 5.5, chosen for its throughput and cost at the shape of extraction we're doing — returns a JSON object: for each source, zero or more proposed binds, each with a role (same_work, chapter_in_this_volume, volume_of_this_chapter, excerpt_of_work, other_edition) and a mandatory evidence quote of three to fifteen consecutive words copied exactly from the card. The system prompt is explicit: source text is untrusted data, not instructions, and the model may refuse. "If you are unsure, give the source no binds. Never guess." The call budget is forty seconds for Claude inside an eighty-second request deadline, with a persisted raw-answer cache keyed on the fingerprint of the entries, the source hash, and the prompt version — a re-check of the same inputs re-runs the gates against the fresh bibliography without a provider call.

The third layer is the one that actually makes the pattern work. Every proposal passes through the gates in services/acurio-core/src/binding/gates.rs — nine hundred lines of Rust whose only job is to refuse. The gates do not invent a bind, ever. They refuse everything the model cannot prove on the card the model just read.

Four Gates That Catch Hallucination

Four gates together carry the load.

The evidence-quote gate: the model's evidence string must appear verbatim in the card, folded for diacritics and whitespace, with at least eight characters. A quote the model stitched from two card lines (title plus author) survives only if nearly all its words are on the card and a long contiguous run of it is verbatim. A paraphrased quote is a refused bind. This single gate catches the overwhelming majority of hallucinated proposals: the model that is making one up cannot copy from a card it did not actually read.

The name gate: the card must contain the entry's author family names, or — for a volume or chapter role — the editor family names, or (for a work with a corporate author like Bundesamt für Strassen) at least sixty percent of the corporate name's significant words. A model that proposes "this PDF is the Kindt 2017 paper" without "Kindt" appearing anywhere on the card is refused. The stopword list is multilingual (DE/EN/FR/IT/ES) and conservative; the significant-word threshold is calibrated against a one-thousand-entry eval that is checked into the repo.

The title gate: a same_work bind that does not survive the name gate must survive the title gate — at least eighty percent of the entry's significant title words have to be on the card, with a lone title word only naming a document by itself when it is at least eight letters long. A same_work where the title coverage sits between fifty and eighty percent is kept but flagged title_differs: true, so the reviewer sees the mismatch. Below fifty percent, it is refused.

The sibling gate: when two entries in the same bibliography share author and year (the classic 2017a / 2017b ambiguity), only the entry whose title or page range the card actually shows is bound. Everything else is refused. This is the gate that catches the single most common pre-launch bug in LLM-driven matching: the model picks a bind that is plausible for the author but wrong for the year or the specific work.

A proposal that survives all four is written to acurio_core.bind_source_runs with its evidence, its role, and the model and prompt version that produced it. A refused proposal is written to a shadow row for offline evaluation and never reaches the retrieval union the judge reads from. The same gates run on the cache: a stored raw answer carries only the model's unverified proposals; the gates re-run on every read against whatever the current bibliography looks like. If a user corrects an entry a week later, the cached proposal is re-gated against the corrected entry and may now refuse where it previously passed, or vice versa. The cache stores claims, not verdicts.

Three Rules for Letting an LLM Decide Something That Matters

Three rules survive this architecture, and they generalise past source binding to any place a team is wiring an LLM into a decision whose output a user will see:

Let the model propose. Never let it decide. A proposal is a sentence. A decision is a database row that bills a user and shapes a report. The two should never be the same code path. The gate layer is what turns one into the other, and the gate layer has to be ninety percent of the engineering effort — not an afterthought bolted on because an eval surfaced a hallucination.

Require an evidence quote the code can check. The most expensive mistake in grounding an LLM is asking for structured output without an evidence field, or with one the code never verifies. A quote the model has to copy from a short card, and that the gate verifies character by character, is the cheapest hallucination detector that exists. Anthropic's own tool-use docs call out exactly this discipline: the model's structured output is a claim, and the surrounding system is responsible for turning claims into verified actions.

Cache the claims, not the verdicts. If your gate layer is doing real work, re-run it on every read. A model answer cached as a verdict will outlive the correction that would have refused it. A model answer cached as a claim gets gated again against today's bibliography, today's source list, today's user corrections — the same discipline we used in the backstop sweep for job-row durability, where the row is written once and verified by the sweep every tick.

One week after shipping, zero model-only binds have landed in the result set. Every bind in production either comes from the lexical fast path or carries an evidence quote the gate layer confirmed on a source card the user uploaded. The matcher stopped growing new passes, because the long tail is now the model's job — and the auditability stayed, because the gates refuse everything the model cannot prove.

If you are wiring an LLM into a decision whose output a user will read in a report, and want a second pair of eyes on the gate layer before it ships — book a free AI Potenzial-Check, or read our per-user queue writeup for the admission side of the same pipeline.