Near-Miss Distractors in Long Patient Records, Trained Away with Checkable Evidence

Language models reading a patient's full hospital record often get the answer wrong for a simple reason. The record contains many facts that look like the right one but belong to the wrong time or the wrong context: last year's potassium value, a diagnosis that was later ruled out, a paragraph copied forward from an old progress note. We propose to build a benchmark made of exactly these "near-miss" distractors. Every question is generated from the record's timestamped structured data, such as lab tables, prescriptions and coded diagnoses. Because of this, a deterministic program can check whether the evidence a model cites is the true fact or the decoy. We then use that same checker as the training reward for a compact 8-billion-parameter clinical model.

The bet has two parts. First, the distraction that matters in clinical records is plausible, same-entity, wrong-time text, not random irrelevant text. Second, a model trained against a symbolic check of its cited evidence learns to resist that distraction better than one trained on lexical-overlap or language-model-judge signals. In two sentences: current anti-distraction training scores answers with metrics that cannot tell a correct fact from its decoy, so it cannot teach a model to tell them apart. We replace those metrics with a timeline checker, and we test whether that reward, rather than the extra data or the citation format, is what produces robustness on long records.

Core Claim

Robustness to distracting context in long clinical records is limited by what the training signal can see, not only by context length or model size. Evidence for the gap comes from two sides. Lexical and judge-based scoring miss real correctness: symbolic verification raised Llama-3.1-8B's measured CLadder accuracy from 0.66 under an LLM judge to 0.90 (Uncovering Hidden Correctness in LLM Causal Reasoning via Symbolic Verification). Statistically plausible clinical outputs can also be ontologically wrong (DataSynK). If the reward cannot separate "cited the March creatinine" from "cited the January creatinine", optimizing it cannot teach that separation.

We claim three things, each testable. A verifier-rewarded compact model will beat both supervised timestamp-citation tuning and judge-rewarded preference tuning on near-miss decoys, while matching them on clean inputs. The gain will concentrate on same-entity temporal decoys and copied-forward text, not on unrelated noise. The robustness training will not impose a larger specialization tax on general medical question answering than ordinary clinical instruction tuning does.

What Exists and What Is Missing

Three papers motivate the regime. ClinicBench evaluates 22 models on 17 clinical tasks and finds quality falls sharply past about 2,000 words. MedAlign builds clinician-written instructions over full records and shows GPT-4 losing 8.3% accuracy when its window drops from 32k to 2k tokens. It also notes that about 30% of its BM25-matched pairs carry irrelevant record context. TIMER diagnoses recency bias and lost-in-the-middle failures in records. It fixes them by instruction-tuning Llama-3.1-8B to cite visit timestamps, reporting a 9.2% gain on TIMER-Bench and 7.3% on MedAlign. TIMER is our closest neighbor. It supervises the citation format but never checks the cited fact against ground truth, and its evaluation is clinician ratings and lexical scores. Its distractors are whatever the record happens to contain, not controlled decoys.

Two papers supply methods we reuse. Ext2Gen trains an 8B model to extract key sentences and then answer, using preference pairs built under up to 25 distracting chunks. It reports an average score of 0.557 against 0.473 for the default Llama-3.1-8B-Instruct, and it beats separate pre-generation compressors. Its pairs are ranked by accuracy, ROUGE-L, BERTScore and an LLM judge, and its data are short open-domain passages. We keep its extract-then-answer output shape but swap the ranking signal for a symbolic check, and move it to long records. Execution-feedback reinforcement learning (AutoTriton, regex synthesis with unit tests) shows that a deterministic checker can drive a small specialized model. Our timeline checker plays the role of the unit test.

Two papers constrain the design. DOVERIFIER needs a clean causal graph, and free clinical narrative does not provide one. We therefore restrict verifiable claims to timestamped facts that the structured tables ground. This is narrower than causal verification, and we say so. What Characterizes Effective Reasoning? shows that failed reasoning branches left in context lower later correctness by 8–14%. That warns us that long self-generated traces are themselves a distractor. Our output format is therefore short evidence tuples rather than open-ended reasoning. Specializing Smaller Language Models towards Multi-Step Reasoning shows narrow specialization can wreck general reasoning, so we measure forgetting directly.

Research Program

The benchmark is built from MIMIC-III, which pairs notes with timestamped lab, medication and diagnosis tables, and whose admissions mostly fit within 16k tokens per ClinicalMamba. Questions are templated from structured events. Examples include "What was the most recent creatinine before the first dose of vancomycin?", "Was atrial fibrillation present at the second admission?" and "Did the potassium value rise or fall between the last two measurements?". Each question has a gold answer and a gold evidence tuple of the form (event type, value, time or admission).

We then control the context. We insert or relocate four decoy types at controlled positions in the note stream. A temporal near-miss is the same lab or drug at a different time. A negated or resolved mention is something like "r/o PE, CTA negative". A copied-forward block is an earlier note's assessment pasted into a later note. An unrelated clinical passage serves as a control. Each decoy carries its own tuple. A check against the structured tables confirms the decoy's tuple is false for the question while the gold tuple is true.

The model must answer and emit the evidence tuples it relied on. The checker parses the tuples and matches them against the structured tables with a tolerance on values. The reward is

$$r = \mathbb{1}[\text{answer correct}] \cdot \frac{v}{c} \;-\; \lambda\, \mathbb{1}[\text{a decoy tuple is cited}],$$

where $c$ is the number of tuples the model cites, $v$ is how many of them the checker verifies, and $\lambda$ is a penalty weight tuned on a validation split. We train Llama-3.1-8B-Instruct with group-relative policy optimization, starting from a short supervised warm-up on format.

Generalization to free-form clinical tasks is measured on MedAlign and TIMER-Bench, where the checker does not apply and their own metrics are used. Forgetting is measured on USMLE-style multiple-choice questions. As a secondary analysis, we run the decoy benchmark on ClinicalMamba-2.8B against a same-size transformer. The question is whether compressing history into a recurrent state makes near-miss decoys more or less harmful. No paper in the file measures this.

Baselines

The naive reference is Llama-3.1-8B-Instruct prompted zero-shot to answer and cite evidence, given the full record with decoys. It shows how much damage the decoys do. We report its accuracy on clean and decoy versions of each question.

The strongest existing in-domain method is TIMER-style supervised tuning. We fine-tune the same base model on the same question pool with gold timestamp citations as targets, matching example count, epochs and GPU hours to our run. TIMER is the strongest competitor because it is the only cited method that targets temporal distraction in records with an 8B model. Its reported numbers are a 9.2% gain on TIMER-Bench and 7.3% on MedAlign (TIMER). No reported result was found on Lacuna for TIMER or any method on a controlled near-miss decoy benchmark, since the benchmark is ours.

The strongest generic anti-distraction method is Ext2Gen. We rebuild its preference pairs on our records, ranked by its original signal (accuracy plus ROUGE-L, BERTScore and LLM-Eval), and train with direct preference optimization on the same budget. Its reported general-domain score is 0.557 against 0.473 for the default model (Ext2Gen).

The ablation that isolates our key ingredient keeps everything identical, including the reinforcement learning algorithm, data, output format and compute. It replaces only the symbolic check with an LLM-judge reward that sees the same record and is asked whether the cited evidence supports the answer. A frontier long-context model given the whole record is reported as a reference ceiling, not a compute-matched competitor.

Beating the field means two things. On decoy-bearing questions, our model must exceed TIMER-style tuning and the judge-reward ablation by a margin outside the confidence intervals of paired bootstrap tests over questions. It must also stay within noise on clean questions, on MedAlign and on TIMER-Bench.

Why not just…?

Why not just filter the record with BM25 or a reranker before the model reads it? Near-miss decoys share the query's entity names and vocabulary by construction. A lexical filter will keep them, and MedAlign's BM25 matching already left about 30% irrelevant pairs on easier matching. Ext2Gen also found that separate compressors lose query-dependent context. We still test this directly: a BM25 and a dense-retriever pre-filter feed the base model, and we report the decoy-retention rate of the filter itself.

Why not just do TIMER-style supervised tuning with timestamp citations? This is the real threat. If supervised imitation of gold citations already teaches the model to prefer the correct timestamp, the verifier reward adds nothing. The compute-matched baseline above settles it. Our prediction is specific: the supervised model learns to produce timestamps but still copies decoy timestamps when the decoy sits closer to the end of the record. The verifier-rewarded model should cut the decoy-citation rate on recency-placed decoys in particular. If it does not, we will report that the reward is unnecessary.

Why not just use a strong LLM judge as the reward? Judges are cheaper to build and apply to free text. But the CLadder result (0.66 judged versus 0.90 verified) and ClinicBench's observation that evaluators favor fluent formatting suggest judges miss exactly the fine distinctions at stake. The judge-reward ablation measures this. We also audit judge agreement with the checker on decoy citations specifically.

Why not just use a long-context frontier model? ClinicBench and MedAlign show frontier models degrade on long clinical text. In addition, ClinicalMamba-2.8B reached 91.8 ROC-AUC on n2c2 cohort selection, against a reported 84.8 F1 for zero-shot GPT-4, which suggests compact specialized models are competitive. We include the frontier model as a reference so readers can see the gap rather than take it on faith.

Decisive Experiments

The first check is the decoy curve. We plot accuracy and decoy-citation rate for every system across the four decoy types and five insertion positions at record lengths of 4k, 8k and 16k tokens. The hypothesis fails if near-miss decoys hurt no more than unrelated passages do. In that case the benchmark premise is wrong and generic noise training would suffice.

The second check is the reward ablation: verifier against judge against supervised citation tuning at matched compute. Only the verifier arm should reduce near-miss citation errors.

The third check is transfer and tax. Gains must not come with drops on MedAlign, TIMER-Bench or USMLE-style questions relative to TIMER-style tuning. We also test whether training on two decoy types transfers to the held-out two.

The fourth check targets self-generated noise. Following the failed-branch intervention in What Characterizes Effective Reasoning?, we append the model's own rejected hypotheses to the context. We test whether the verifier-trained model is also less swayed by them.

The fifth check is the architecture probe. We run ClinicalMamba against a matched transformer on the same decoy curves to test whether recurrent compression drops the low-frequency exact values that near-miss questions hinge on.