Evidence Ledgers: Copy-Constrained Reasoning Records for Medical Retrieval QA

When a language model answers a clinical question with retrieved literature in its prompt, nothing forces its reasoning to rest on that literature. It can recall an answer from memory, write a plausible justification afterward, and attach citations or guideline names it invented. It can also accept a false premise slipped in by the user. This proposal makes the model write its reasoning as an evidence ledger. The ledger is a small structured record in which every clinical claim either carries a verbatim quote copied from a retrieved passage, is explicitly tagged as something the user asserted, or is marked "unsupported." The copying is enforced during generation, so the model cannot emit a quote that is not in the retrieved text. The final answer is then derived from the ledger's supported claims, and the ledger's own contents (how many claims lack support, whether the answer depends on a user-asserted premise) feed a trained abstention module that decides whether to answer at all.

The bet is that this record does two things free-text reasoning cannot. First, it removes fabricated evidence by construction and separates the user's premises from the literature, which targets the "baiting" traps on which current models fail badly. Second, it turns hallucination risk into countable, checkable features, so abstention can be calibrated on something better than token probabilities. The question that decides whether the idea is worth anything is whether these benefits survive two cheaper alternatives: letting the model write freely and checking its quotes afterward, and paying a "structure tax" in reasoning accuracy. The experiments are designed around those two threats.

Core Claim

On adversarial clinical dialogue and on reasoning-heavy biomedical question answering, an open-weight 7B model that reasons through a copy-constrained evidence ledger over a large biomedical corpus will show three properties. Its hallucination rate on clinical traps will be lower than free-text thought-driven retrieval. Its answer accuracy will stay within about one point of that baseline, meaning no meaningful reasoning tax. Its abstention will keep more coverage at a fixed error rate than abstention based on probabilities or on free-text features. The novel ingredients are the input-dependent copy grammar, which makes quotes valid by construction; the premise-provenance slot, which separates user claims from evidence; and the use of ledger structure as the feature space for a learned rejector.

Where This Sits

Three medical works motivate the design. RAR2 shows that generating a free-text thought before retrieval, and using that thought as the search query over the 30.4M-chunk MedCorp corpus, beats querying with raw symptoms. It reports 60.46% average accuracy over six biomedical QA benchmarks for Qwen2.5-7B, against 57.97% without thoughts. Its thoughts and answers are unconstrained text, though, and it is not evaluated on adversarial traps. We keep its thought-driven query step and constrain what comes after retrieval. MedCite breaks generated answers into statements and re-retrieves evidence for each one against 24.6M PubMed documents. It finds a "RAG paradox": retrieval raises correctness but lowers citation quality unless each claim is verified, and citations recalled from model memory are fabricated. MedCite verifies after the fact. We move the claim-evidence binding into generation, and we use MedCite's attribution metrics and post-hoc pipeline as a competitor. MHB supplies the stress test: 14 trap subtypes in which frontier models still hallucinate, including invented guidelines and agreement with false assertions.

Two structured-reasoning works supply the method and frame the main tension. LiteCoST makes small models build a serialized table or graph before answering and reports large gains on long-document finance QA. That supports "structure first." The research file also summarizes a competing finding that juggling format and reasoning at once splits the model's capacity and hurts accuracy. LiteCoST used no retrieval, no medical setting, and no decoding-time guarantees, so the tension is unresolved here and we measure it directly. CoTAR asks the model to quote supporting spans before answering, which raises GPT-4's span citation score on QuoteSum from 81.15 to 89.88. Its quotes are prompted, not enforced, and it is not clinical.

On constrained decoding, the input-dependent grammars in Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning restrict outputs to candidates derived from the input, which raised entity disambiguation accuracy from 54.1% to 80.3%. Dialogue Ontology Relation Extraction via Constrained Chain-of-Thought Decoding restricts vocabulary to transcript terms. We extend this "only say what is in the source" idea from entity names to evidence quotes drawn from retrieved passages. Flexible and Efficient Grammar-Constrained Decoding reports per-token masking overhead of 5–32 ms, which makes per-query grammars practical.

Two works constrain the design. Reasoning or Retrieval? shows that reasoning models often recall an answer from memory and rationalize it afterward. A ledger does not stop this by itself, because a model can quote a real but irrelevant span to decorate a memorized answer. We therefore add an entailment check between each quote and its claim, plus a perturbation test borrowed from that paper. Learning to Reject with a Fixed Predictor trains a separate rejector on top of a frozen model using a convex surrogate loss. It keeps 90.6% coverage at halved error, against 64.4% for thresholding on the top token probability, on decontextualization. We reuse that rejector and change what it sees. Finally, Reducing hallucination in structured outputs via Retrieval-Augmented Generation shows that retrieval cuts hallucinated components in JSON workflow generation from up to 21% to under 7.5%. That work grounds the schema vocabulary, not the reasoning, and has no clinical content.

Research Program

The pipeline has four stages. In the first, the model writes a free-text thought in the style of RAR2 and uses it to query MedCorp with a hybrid lexical-plus-dense retriever and reranking, keeping the top passages. Leaving this stage unconstrained is deliberate: it is where open-ended clinical reasoning (mapping symptoms to mechanisms) pays off, and constraining it would invite the structure tax.

In the second stage, the model writes the ledger under a grammar compiled for each query. Each entry has a short claim in free text and a provenance field that takes one of three forms. The first form is a quote. The decoder may emit only token sequences that occur contiguously in one retrieved passage, implemented as a finite-state constraint over the retrieved passages' token sequences, plus a passage identifier drawn from the retrieved set. The second form is a user-asserted tag, which must copy from the user's turns. The third is an explicit unsupported marker. Because the grammar is rebuilt for every query, the cost of compiling it matters. The research file flags dynamic-grammar compilation as an open bottleneck, so we report compilation and masking time separately.

In the third stage, the answer field is free text but must reference ledger entries by index. A lightweight entailment model, a biomedical natural-language-inference model, scores each quote–claim pair.

In the fourth stage, a rejector is trained with the surrogate loss of Learning to Reject. Its features are the counts of supported, unsupported, and user-asserted entries the answer relies on, the minimum and mean entailment scores, and retrieval scores. When it fires, the system abstains and returns the ledger to the user so the gap is visible.

The model is Qwen2.5-7B, chosen because decoding-time constraints need access to the model's next-token probabilities. Training is light. A supervised warm start on ledgers written by a larger model and filtered by entailment teaches the format, so the grammar is not doing all the work. An optional second stage of GRPO, a reinforcement-learning method that compares groups of sampled outputs, uses separate rewards for answer correctness and ledger entailment, in the spirit of the decoupled rewards in Reading or Reasoning?. All main comparisons are run both with and without this second stage.

Baselines

The do-nothing reference is the same 7B model answering with chain-of-thought and no retrieval. It shows how much any retrieval helps on each benchmark and gives the raw MHB hallucination rate for this model. MHB reports that open-source models exceed 57.0%, with Llama-4-Scout at 59.0%. No figure is given for Qwen2.5-7B, so we measure it.

Standard retrieval-augmented generation queries MedCorp with the question text and puts the top passages in the prompt. RAR2 reports the corresponding MedRAG figure on MedXpertQA as 14.20%. We rerun it with our retriever and passage budget.

The strongest competitor on accuracy is RAR2's thought-driven retrieval with free-text reasoning and answer. Its reported numbers are 60.46% average over six benchmarks and 17.59% on MedXpertQA for the tuning-free variant. We reproduce it on the same model, corpus, retriever, and top-k, and give it the same training budget as our method: the same number of preference pairs or GRPO steps, with the same tuning sweeps. Beating it means two things. On MHB, the ledger system must cut the hallucination rate by at least a quarter in relative terms. On the six-benchmark average, it must land within one point of RAR2 or better.

The strongest competitor on attribution is MedCite's post-hoc pipeline applied to RAR2's outputs. It decomposes the answer into statements, re-retrieves evidence for each, and reranks. MedCite reports citation F1 of 84.36 on BioASQ and 86.48 on PubMedQA using GPT-4o. For a fair comparison we run the pipeline with the same 7B model doing the reranking and report its extra retrieval calls and tokens alongside ours.

The key ablation is a prompted ledger with no grammar. It uses the same schema and the same warm-start data, followed by an after-the-fact check that each quote appears verbatim in a passage, and it is allowed to resample until it passes. Retries are capped so total generated tokens match the constrained system. This isolates whether enforcing quotes during decoding matters beyond having the ledger at all. A second ablation removes the user-asserted provenance slot, isolating the premise separation that is supposed to target baiting traps. For abstention, the baselines are top-token-probability thresholding, verbalized confidence, and the same learned rejector trained on free-text features from RAR2 outputs: thought length, retrieval scores, and token probabilities. Beating them means higher coverage at an error rate half of the non-abstaining system's.

Why not just…?

Why not just let the model reason freely, then check its quotes and retry? This is the real threat, and it is why the prompted-ledger-with-retry ablation exists. After-the-fact string matching catches fabricated quotes as reliably as decoding-time constraints do. The case for the constraint is cost and coverage. In the GCD paper, unconstrained LLaMA produced valid structures only about 64% of the time, which implies many retries. Retries also tend to drop hard claims instead of grounding them. If the retry system matches our MHB rate and citation F1 at equal token budget, we will report that the ledger schema and the rejector are the contribution and the grammar is just engineering convenience. The proposal would then narrow to that claim.

Why not just add MedCite-style claim verification on top of RAR2? That checks evidence but does not change reasoning. A false user premise that the model accepted is still in the answer; verification only labels it unsupported after the fact. MedCite also does not separate user assertions from evidence. We test this directly by comparing hallucination rates on MHB's trap subtypes, where the premise slot should matter most. The comparison is also checked against chance: if post-hoc verification plus abstention on unsupported statements matches the ledger, the decoding-time binding adds nothing.

Why not just threshold confidence, or train the rejector on what the free-text system already exposes? On decontextualization, top-token-probability thresholding kept 64.4% coverage at halved error, against 90.6% for a learned rejector (Learning to Reject). That favors learned rejection but says nothing about which features it should use. Training the same rejector on free-text features separates "a learned rejector helps" from "ledger features help."

Won't the structure tax hurt reasoning? Possibly; this is the open tension between LiteCoST and the capacity-splitting view. We confine constraints to the stage after retrieval and leave the retrieval thought free. Accuracy within one point of RAR2 is a pass condition, not an aspiration.

Decisive Checks

We evaluate on MHB with retrieval enabled, reporting the overall hallucination rate and the rate for each of its five categories. For context, MHB reports 29.1% for Claude-4-Sonnet. We also report the six RAR2 benchmarks including MedXpertQA, BioASQ-Y/N and PubMedQA citation precision and recall using MedCite's attribution protocol, risk–coverage curves, and per-token and compilation overhead.

Three outcomes would kill the idea. The first is if the retry ablation matches the constrained ledger at equal tokens. The second is if accuracy falls more than a point below RAR2. The third is if ledger features do not beat free-text features for the rejector.

One more check targets shortcut rationalization. Following the perturbation logic of Reasoning or Retrieval?, we swap the retrieved passages for passages that support a different answer. A ledger-grounded system should change its answer or abstain. A system that rationalizes after the fact will keep its memorized answer and quote whatever it can. We compare the flip-or-abstain rate against RAR2 on this test. Physician review of 200 sampled ledgers, stratified by MHB subtype, checks that "supported" entries are clinically supporting and not just textually entailed.