A common way to use large language models (LLMs) on patient records is to have them condense months of clinical notes into a short summary, then train an ordinary risk predictor on that summary. The summary also serves as the explanation a clinician reads. This creates a quiet failure mode. A summary can look accurate and still contain two kinds of content that make the predictor look better than it will be in practice. The first is anticipation: faithful restatements of decisions clinicians have already made about the outcome, such as a do-not-resuscitate order, a palliative-care consult, or a planned transfer. The second is forward hallucination: confident prognostic or diagnostic statements that nothing in the record supports and that the model filled in from what it learned during training. Both raise retrospective accuracy. Neither helps at the bedside, because an early warning that fires only after the care team has written the plan, or that rests on a claim nobody documented, tells clinicians nothing new.
We propose a claim-level temporal provenance audit. Every sentence-level claim in a summary is checked against timestamped evidence, using both the notes and the structured tables. Each claim is sorted into one of four groups: grounded physiological observation, grounded record of a clinician's anticipation of the outcome, ungrounded claim recycled from the model's prior knowledge, or ungrounded inference. We then measure how much of the downstream predictor's accuracy, and how much of its explanation, rests on each group. The decisive test is performance on unanticipated events: patients who deteriorated with no documented clinician anticipation before the prediction time. That subgroup is where an early-warning system actually earns its place.
Core Claim
Existing safeguards target the problem from one side at a time. Temporal firewalls remove data by time: On the Importance of Clinical Notes in Multi-modal Learning for EHR Data masks the final note before the event, and Clinical XLNet restricts inputs to the first 48 hours of ventilation. Hallucination diagnostics check whether claims are supported by evidence. Our claim is that in summary-to-predictor pipelines these two problems meet at the level of individual claims, and that neither safeguard handles them alone:
- A strict time cutoff still lets anticipation claims through, because earlier notes record decisions already made. Husmann et al. themselves show the model spiking on "DNR".
- A support check counts anticipation claims as perfectly faithful.
- A support check also misses claims that the model would have produced without reading the record. The Hallucination Dependence Index showed this on TCGA lung cancer cases: grounded support rates compressed into 81.9–93.2% while ungrounded hallucination ranged from 5.9% to 91.5%.
What would be new is a measurement that separates these components per claim, ties them to the predictor's attributions, and reports how much of the predictor's accuracy survives restricting it to admissible observations.
Research Program
Building the summary pipelines
We build summary-then-predict pipelines on MIMIC-III for three tasks:
- Decompensation and in-hospital mortality, following the cohort and note-masking setup of Husmann et al.
- Prolonged mechanical ventilation, following the 48-hour Clinical XLNet cohort.
Summaries are produced in the incremental style of CLIN-SUMM: an initial summary, then update prompts that record changes over time. CLIN-SUMM's own clinician audit found under 4% diagnostic interpolation, but it never checked temporal placement or prior recycling.
The summarizers are open-weight models run locally, as required for credentialed data. We use Qwen3 and MedGemma-27B, the families evaluated on EHR-ReasonCon. Each summarizer runs in three conditions:
- Strict. The model sees only notes timestamped before the prediction time.
- Leaky. The model sees the full stay and is instructed to summarize only up to the prediction time. This mirrors the common shortcut of summarizing a stay once and slicing it afterward.
- Ungrounded. The model receives only the admission diagnosis, demographics, and task. This is the HDI baseline, adapted so we can detect claims the model would write without the chart.
The audit
The audit splits each summary into atomic claims. It checks each claim for entailment against evidence restricted to timestamps at or before the prediction time, drawing on both notes and structured tables (chart events, orders, procedures, code status).
- Anticipation is labeled from structured tables rather than by an LLM. A claim counts as anticipation when its supporting evidence refers to an order, code-status change, consult, or scheduled procedure that is clinically tied to the target outcome. The vocabulary is fixed in advance with clinicians.
- Unsupported claims are split into two groups. Prior-recycled claims semantically overlap the ungrounded-condition output for the same patient, following HDI's overlap probe. The remainder are inferred claims.
- Validation. The claim checker is validated on EHR-ReasonCon, which tests the same note-versus-table verification problem and on which LLMs reached F1 only in the 50s. It is also validated on a clinician-adjudicated sample from our own summaries.
Linking the audit to the predictor and its explanation
Downstream, we train two predictors on summaries. One is a Clinical-ModernBERT classifier, as in CLIN-SUMM. The other is a sparse logistic model over claim embeddings, whose attributions we can read off directly. We attribute each prediction to claims and report the anticipation share and ungrounded share of total attribution.
Finally, we retrain and evaluate on summaries restricted to admissible observation claims. The gap between the full and restricted pipelines on unanticipated events is the quantity of interest.
Baselines
Naive reference. A summary-based predictor trained on the leaky condition with no audit. This is the pipeline people build by default. It sets the inflated ceiling that everything else is compared against.
Strongest temporal-firewall baseline. The masking and windowing protocol of Husmann et al., with the final note masked and the cross-modal transformer over 42 vitals and ClinicalBERT notes. That paper reports decompensation AUPRC of 39.7, against 31.7 for an EHR-only transformer and 33.3 for an LSTM. For ventilation, the reference is Clinical XLNet, which reports AUROC 0.663 ± 0.011 for prolonged ventilation and 0.779 ± 0.006 for 90-day mortality. We re-run both on our cohort splits with the same tuning budget, rather than quoting their figures across cohorts. The strict-condition summary pipeline is our version of the same firewall applied to summaries.
Strongest hallucination-side baseline. HDI-style output diagnostics, plus a standard LLM-judge supported-claim rate. On the open models we also run ReDeEP, which scores how much a model copies from context versus injects from its own stored knowledge. ReDeEP reported chunk-level AUC 0.7458 on RAGTruth with LLaMA2-7B, against 0.7290 for RAGAS. We ask whether these detectors flag the claims our audit labels prior-recycled. Because anticipation claims are faithful by construction, we expect none of these detectors to flag them.
Ablations. - Timestamp-blind audit. Entailment is checked against the whole stay. This isolates whether time-restricted provenance matters. - Structured-only anticipation labels versus an LLM-judged anticipation label. This isolates whether grounding the anticipation definition in tables matters.
What counts as beating the baselines. The proposal succeeds if it does any of the following, at matched compute and data:
- explains a larger share of the gap between all-event and unanticipated-event performance than the firewall baselines remove;
- identifies prior-recycled claims that support-rate judges score as supported;
- yields a restricted predictor whose unanticipated-event AUROC matches or exceeds the full pipeline's, with confidence intervals from patient-level bootstrap.
Why not just…?
Why not just enforce a strict time cutoff and mask the last note? This removes literal future data. It does nothing about notes written days earlier that already record the decision, and the open problems in the file explicitly flag forward-looking conditional phrasing as an unsolved leakage path. Husmann et al.'s own attention analysis shows "DNR" driving risk spikes under their masking protocol. Our anticipated-versus-unanticipated stratification of the strict pipeline measures directly how much of its accuracy rests on such claims. If that share turns out to be negligible, the firewall is sufficient and we will say so.
Why not just filter keywords such as DNR, comfort care, and palliative? This is a real threat. It may capture most anticipation with no LLM checker at all. We include a clinician-curated keyword filter, applied to both raw notes and summaries, as a baseline. Our expectation is that summarization paraphrases decisions ("goals of care revisited; family aligned with focus on comfort") so that keywords miss them. The table-grounded labels give an independent count of misses. If the keyword filter recovers nearly all table-confirmed anticipation, the contribution shrinks to the prior-recycling component and the measurement protocol, and the paper will be framed that way.
Why not just skip summaries and encode raw notes? Raw-note encoders such as Clinical XLNet avoid generated hallucinations, but they inherit anticipation leakage unchanged and give no discrete claims a clinician can audit. We run the anticipation analysis on raw-note models too, using span-level tagging of the same table-confirmed events, so the comparison isolates what summarization adds or removes.
Why not just use an LLM-judge faithfulness score? HDI shows that grounded support rates compress across models and can invert rankings. Gemini-2.5-flash had the highest grounded support at 93.2% despite heavy ungrounded confabulation. A judge cannot flag anticipation, because those claims are true. Our timestamp-blind ablation and the detector comparison test this directly.
Proposed Experiments
Experiment 1: how much accuracy comes from anticipation and recycling. For each task and summarizer, we report AUROC and AUPRC for all events, for unanticipated events, and as a function of hours before the event. These are computed for the leaky, strict, and admissible-only pipelines, alongside the re-run firewall baselines.
We expect three things:
- the leaky condition to place post-cutoff content into pre-cutoff summaries at a measurable rate, consistent with the weak temporal ordering CRAFT documents on MedTempo, where the best exact match was 37.14%;
- anticipation claims to carry a disproportionate share of attribution in the strict pipeline;
- the strict pipeline's advantage over the admissible-only pipeline to concentrate in the hours just before the event and largely vanish on unanticipated events.
The hypothesis is falsified if the advantage persists on unanticipated events.
Experiment 2: validity of the audit itself. We measure:
- claim-checker accuracy on EHR-ReasonCon and on clinician-adjudicated claims from our own summaries;
- agreement between table-derived anticipation labels and clinician judgment;
- the overlap between our prior-recycled claims and claims flagged by the HDI-style, LLM-judge, and ReDeEP detectors.
Experiment 3: clinician reading test. Clinicians read the explanations, full versus admissible-only, on held-out cases. They rate whether each explanation tells them something they did not already know from the chart.
Two caveats bound the results. First, everything is single-center MIMIC-III, so external validity is limited. Second, MIMIC-III's per-patient date shifting rules out a true calendar-based temporal split. The unanticipated-event subgroup is therefore our stand-in for prospective deployment, and we state that as its limit.
References
- On the Importance of Clinical Notes in Multi-modal Learning for EHR Data
- Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation
- The Hallucination Dependence Index: A Cross-Condition Diagnostic for Clinical-LLM Faithfulness
- CLIN-SUMM: Temporal Summarization of Longitudinal Clinical Notes
- Towards Error-Free EHRs: Reasoning-Intensive Consistency Verification Between Clinical Notes and Structured Tables in Electronic Health Records
- ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability
- CRAFT: LLM-Based Iterative Refinement for Temporal Reasoning over Clinical Narratives