Do Clinical Summaries Know the Patient Was Treated?

When a language model summarizes an intensive care stay, it reads vital signs and lab values that were shaped by treatment. A blood pressure of 68 mmHg means one thing in an untreated patient and something very different in a patient on a high-dose vasopressor, a drug that raises blood pressure. Clinicians read the chart with the treatments in mind. We suspect current models often do not, and that they fail in two ways. They write "hemodynamically stable" about a patient whose stability is produced by drugs. They also write "started on broad-spectrum antibiotics" because septic patients usually are, not because this chart says so. The first failure leaves out the external cause behind an observed state. The second invents a treatment from medical background knowledge. Both are hallucinations about treatments, the outside inputs that clinicians apply to a patient's physiology. Existing faithfulness checks do not separate them from ordinary errors.

This note proposes a diagnostic, and a mitigation to test against it. The diagnostic intervenes only on the treatment records of a real ICU chart. It deletes them, and it pairs patients with matched physiology but different treatment. It then measures whether the summary's claims change the way a faithful summary must. The mitigation is a decoding rule that compares the model's output on the full chart with its output on the same chart with treatments removed. It down-weights words the model would have written anyway. The core claim is that treatment-related hallucination is a distinct and measurable failure in longitudinal clinical summarization. Global faithfulness scores hide it. The chart-minus-treatments condition is the right counterfactual for exposing it. Whole-context removal and generic domain priors are too coarse.

Why this gap, and why now

Three findings motivate the proposal. Boosting the interpretability of clinical risk scores with intervention predictions trains a multitask model on MIMIC-III that predicts mortality alongside 13 ICU interventions. The authors argue that a risk score which silently assumes future treatment will mislead clinicians, a "self-defeating prophecy." That work concerns prediction heads, not generated text. We carry its insight into summarization: a narrative that describes physiology without its treatment context makes the same error in prose. The Hallucination Dependence Index compares unsupported-claim rates with and without the patient evidence bundle on 119 TCGA-LUAD oncology cases. It finds a ranking inversion. Gemini-2.5-flash had the highest grounded support (93.2%) but a 91.5% ungrounded hallucination rate, while GPT-5.4-mini had an index of 0.336. High grounded scores can therefore hide prior recycling. That diagnostic removes the entire evidence bundle from static cases. We remove one causally meaningful part of a longitudinal record, the treatment stream, and keep everything else. Predictability of Popularity shows that apparent predictive success can come almost entirely from early momentum: temporal features carried about 97% of predictive power, and accuracy fell below 65% once items were matched on adoption time. We borrow its method of matching on the confounding trajectory before asking whether a model responds to the external driver.

Two decoding papers supply the mitigation's mechanism and also mark its limits. Mitigating Hallucination in Abstractive Summarization with Domain-Conditional Mutual Information penalizes tokens that are likely given only a keyword domain prompt. On XSUM it raised AlignScore from 60.02 to 60.78. The authors report that the method fails on numbers, dosages and entities, which are exactly the tokens that carry treatment claims. Characterizing Context Influence and Hallucination in Summarization adjusts how much the model relies on context versus its own memory. It reports that pushing context reliance raised verbatim regurgitation by 50%, a constraint we must measure rather than ignore. Our contrast replaces "keywords" or "no context" with "the same chart minus treatments." The penalty then falls on treatment claims the chart's treatment records do not support, not on medical vocabulary in general.

Core Claim

Let $c$ be a full chart, $c_{\setminus I}$ the same chart with all intervention records removed, and $y_t$ the next token given prefix $y_{<t}$. We score candidate tokens as

$$S(y_t) = \log P(y_t \mid c, y_{<t}) - \lambda \, \mathbb{1}[H_t > u] \, \log P(y_t \mid c_{\setminus I}, y_{<t}),$$

where $\lambda$ is a contrast strength, $H_t$ the model's predictive entropy at step $t$, and $u$ an entropy threshold, gating borrowed from the domain-conditional decoding work. A token that is just as likely without the treatment records is a guess from medical priors when it appears in a treatment claim. The diagnostic side defines, for treatment claims only, an intervention dependence index

$$IDI = \frac{f_{\setminus I} - f_{c}}{f_{\setminus I}},$$

where $f_{\setminus I}$ is the rate of specific treatment claims (drug class, start or stop time) made when treatments are deleted, and $f_c$ is the rate of such claims unsupported by the structured records in the full chart. Any specific treatment claim made under deletion is fabricated by construction. That property lets us check this index automatically against structured MIMIC tables, without human raters.

We predict three things. Models will differ more on IDI than on global faithfulness. Physiological-state claims about variables under active treatment will often omit the treatment. The chart-minus-treatments contrast will cut fabrication more than whole-context contrast at equal regurgitation. The second prediction concerns a failure that decoding alone may not fix, and we say so below.

Research Program

The data come from MIMIC-III and MIMIC-IV ICU stays. We reuse the 13 intervention categories of the intervention-prediction work (ventilation, vasopressors, sedation, antibiotics, anticoagulants and others), extracted from structured medication and procedure tables. Each stay is serialized into a time-ordered chart of vitals, labs, notes and intervention events over a 48–72 hour window. We then build three conditions per stay. The first is the full chart. The second deletes every intervention event and every note sentence naming one, using a medication lexicon plus manual audit of a sample. The third uses matched pairs: two stays whose trajectories of the treatment-affected variable agree closely over the window (mean arterial pressure, oxygen saturation, temperature), where only one received the relevant intervention. Pairing on the observed trajectory follows the time-matched design of the popularity work. A faithful summary must describe the two members of a pair differently even though their numbers look alike.

To measure omitted treatment context, we need to know which variables a treatment affects. We encode a small masking map: vasopressors with blood pressure, ventilation and supplemental oxygen with saturation and respiratory rate, sedation with mental status, antipyretics with temperature. It is curated by a clinician and cross-checked against standard pharmacology. This is the domain-knowledge component, kept small and explicit. The knowledge-gap literature shows that missing or noisy retrieved facts drive errors: in Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text, 44% of failures came from facts absent from both text and knowledge graph. A hand-checked map avoids that noise. From it we define the masked-state error rate. It is the fraction of summary claims about a mapped variable, during an active matching treatment, that describe the variable as normal or stable without naming the treatment.

Claims are extracted from summaries by sentence decomposition and checked in two ways. Treatment claims are checked against the structured tables. State claims are checked by matching to the time window and the masking map. A clinician audits 300 summaries to estimate the agreement of each automatic metric with expert judgment. Open-weight models, including MedGemma, which the file reports on clinical telemetry, and a Llama-family instruction model, run locally for both the diagnostic and the decoding experiments, because the contrast needs token probabilities. API models are evaluated on the diagnostic only, under the data-use terms that MIMIC credentialing permits.

Baselines

The naive reference point is greedy or beam decoding with a standard summarization prompt on the full chart. Every other condition is reported relative to it, on the same stays, models and output length limit.

The strongest whole-context competitor is context-influence decoding from Characterizing Context Influence and Hallucination in Summarization. It contrasts the full context against no context at all. It is the closest existing method because it also contrasts two conditionings at decoding time. It differs only in what it removes, so it is the ablation that isolates our key ingredient. We run it with the same entropy gate and tune $\lambda$ on the same development split, with the same grid size as ours. We count our method as beating it only if it lowers intervention fabrication at a matched or lower verbatim copy rate. The paper itself reports only a ROUGE-L gain of 10% on CNN-DM with 1.5 times more context influence; it gives no clinical result. Domain-conditional decoding from Mitigating Hallucination in Abstractive Summarization with Domain-Conditional Mutual Information is run with KeyBERT keywords from the chart as the domain prompt and the same tuning budget. Its reported result is AlignScore 60.02 to 60.78 on XSUM; no reported result was found on Lacuna for it on clinical charts.

The strongest non-decoding competitor is treatment-aware prompting. The model receives an explicit treatment table at the top of the prompt, plus the masking map and an instruction to name active treatments whenever it describes an affected variable. It costs nothing and needs no logit access. It gets the same context budget, and its instructions are tuned on the development split with the same number of attempts as $\lambda$. Two post-hoc verifiers round out the comparison. The first is the self-verification loop of Self-Verification Improves Few-Shot Clinical Information Extraction, which extracts, finds evidence spans, and prunes unsupported claims. It reported GPT-4 ICD-10 F1 rising from 0.487 to 0.533 on MIMIC-IV. The second is MEDAL from Better Late Than Never, which reports 6–7% faithfulness gains on Llama-2 summaries. For the specific quantities here, fabrication under deletion, masked-state error, and pair sensitivity, no reported result was found on Lacuna for any method. Establishing these numbers is part of the contribution.

Why not just…?

Why not just use the Hallucination Dependence Index as is? It removes all evidence. Under that condition every specific claim is unsupported, so it cannot tell a model that fabricates treatments from one that misreads vitals. The index's own oncology results show that grounded scores bunch together in an 81.9%–93.2% band. The treatment-only deletion is a finer cut. If IDI and the original index rank models identically across our models, the finer cut adds nothing, and we will report that.

Why not just prompt the model to state active treatments? This is the real threat. It may well remove most masked-state errors in capable models, since the failure may be inattention rather than missing knowledge. We treat it as the main baseline, not a straw man. If prompting matches the contrastive decoding on both fabrication and masked-state error, the decoding contribution fails. The diagnostic, the deletion and matched-pair harness, then stands alone as the result. We expect prompting to reduce omissions but not fabrication. Telling a model to "mention treatments" plausibly increases its tendency to mention typical treatments when none are recorded. The deletion condition measures that directly.

Why not just run a post-hoc verifier? Verifiers that check claims against source spans will catch fabricated treatments when the records are absent. They cannot flag an omission: "blood pressure stable" has a supporting span in the vitals. The Better Late Than Never authors themselves note that their corrector struggles with long multi-day trajectories. We therefore expect verifiers to compete on fabrication and to lose on masked-state error and pair sensitivity. Stacking a verifier on top of our decoding is also tested, since the two are complementary rather than exclusive.

Why not just give a frontier long-context model the whole chart? We include such models in the diagnostic. The MHB benchmark reports a 29.1% hallucination rate even for its best model, Claude-4-Sonnet. The HDI ranking inversion shows that strong grounded scores can hide prior recycling. Whether the treatment-specific failures persist at the frontier is an empirical question this harness answers. It is not something we assume.

Decisive Checks

The proposal rests on four outcomes. The first concerns the diagnostic itself. IDI and masked-state error must show clinician-audit agreement comparable to claim-level faithfulness metrics. They must also order models differently from a global unsupported-claim rate; otherwise they are redundant. The second is the key ablation. The chart-minus-treatments contrast must reduce fabrication under deletion more than whole-context contrast at equal $\lambda$-tuning effort. We report fabrication against verbatim copy rate across the full $\lambda$ sweep, so the context-influence paper's regurgitation tradeoff is visible rather than hidden. The third is a head-to-head against treatment-aware prompting on matched pairs. We measure pair sensitivity, the fraction of pairs whose acuity statements correctly differ. The decoding method earns its place only if it adds sensitivity on top of prompting, not merely instead of it. The fourth is a cost check. ROUGE against MIMIC-IV discharge-summary sections and a clinician usefulness rating must not drop. A summary that avoids treatment hallucination by saying nothing about treatment has not solved the problem. If the second and third checks fail while the first holds, the paper becomes a measurement study of a hallucination type no current benchmark isolates, which we would still consider worth publishing.