Language models that predict in-hospital death from clinical notes report high discrimination. On MIMIC-IV, a large public intensive care database, fine-tuned models reach AU-ROC around 0.87–0.88. Many of those deaths, however, follow a documented human decision to limit or withdraw treatment. In severe traumatic brain injury, 86% of ICU deaths follow withdrawal of life-sustaining measures, and half of those withdrawals happen within the first 72 hours (CENTER-TBI). Clinicians write those decisions into the notes as do-not-resuscitate orders, "comfort measures only," and "family meeting, goals of care." A model that reads such a note is partly reading the decision that produces the death. This is a self-fulfilling prophecy, meaning a prediction that comes true because people act on it. It is also a form of temporal leakage, meaning the model sees information from after the moment a real prediction would have to be made. Benchmarks currently score it as predictive skill.
We propose an evaluation that resets each patient's prediction cutoff to the point just before the first documented limitation of care, rather than a fixed number of hours after admission. Under this cutoff we measure how much of a note model's discrimination survives. We also test whether simple fixes, such as redacting keywords or truncating notes early, actually remove the shortcut. Subword tokenization is the reason we doubt keyword redaction. Notes express these decisions through abbreviations, misspellings, and paraphrases. Subword tokenizers break such strings into fragments, and a model can still pick up those fragments after the exact keywords are deleted.
Core Claim
Fixed-window, note-based mortality benchmarks overstate how well models predict the patient's biological trajectory. They do so because they let the model read documented decisions to limit care. A decision-anchored cutoff exposes this gap, and so does reporting discrimination separately for deaths with and without a preceding limitation of care. Lexical redaction is not a sufficient substitute, because the decision signal leaks through surface forms that the subword vocabulary breaks into fragments.
The work is an audit and an evaluation protocol. It is not a counterfactual risk estimator. We do not claim to recover each patient's risk under full treatment, because the drivers of withdrawal decisions are largely unrecorded.
Where This Sits
Two kinds of prior work motivate the proposal. Revisiting Clinical Outcome Prediction for MIMIC-IV benchmarks clinical encoders on note-based outcome tasks, including mortality, and finds that PubMedBERT's domain vocabulary helps. Revisiting the MIMIC-IV Benchmark converts structured EHR data into text for mortality prediction. Both treat death as an unconfounded label and evaluate on static splits. We keep their tasks but change when the input is cut off and how the outcome is broken down.
Three theory and methods papers constrain the design. When accurate prediction models yield harmful self-fulfilling prophecies proves that a model's AUC can stay level or rise while its use harms patients, so AUC alone cannot certify the model. Causal domain shift for decision support recovers baseline risk through a "domain pivot," a set of intended-action variables that makes the outcome independent of whether the system was deployed. Causal monitoring shows that naive monitoring misses risk shifts that causally adjusted calibration catches. Both of these require the clinical action to be observed accurately. We supply a measured action, the limitation-of-care timestamp, taken from structured code-status entries where they exist and checked against the notes. We use it to define the cutoff, not to claim identification.
Two further papers supply the method. EMDOT introduced chronological backtesting and showed that random splits are optimistic on MIMIC-IV. It anchors cutoffs to calendar time. We anchor them to each patient's care decisions, and we keep EMDOT's chronological split so the two kinds of leakage can be separated. The Paris neuro-ICU study handled the self-fulfilling loop by excluding patients who had treatment withdrawn. We test that design as a baseline.
On tokenization, Tokenization Falling Short shows that subword representations correspond poorly to whole words, with a cosine similarity of 0.21 for "assignment" against "assign"+"ment." It also shows that BPE-dropout, which randomly varies how a word is split during training, changes what models are sensitive to. Specialised or Generic? shows that domain vocabularies split clinical words far less, about 1.0 tokens per word against about 2.7. This raises a concrete question: does a domain vocabulary make abbreviations such as "DNR/DNI" or "CMO" into single, easily learned tokens, and so strengthen the shortcut?
Research Program
First, we build decision timestamps for MIMIC-IV ICU stays. The primary source is charted code-status changes. A secondary source is a note-level labeler for limitation-of-care language, validated by clinician review of a sample. We report agreement between the two sources rather than assume either is correct.
Second, for every stay we define two cutoffs:
- A fixed cutoff at 24 or 48 hours, as in prior work.
- A decision-anchored cutoff: the earlier of the fixed cutoff and a margin before the first limitation decision.
Patients with no such decision keep the fixed cutoff. All notes timestamped after the cutoff are excluded. We apply the same exclusion to note text that is charted later but refers to earlier events.
Third, we train note encoders under both protocols with identical hyperparameter budgets:
- A generic-BPE encoder.
- PubMedBERT, with a domain WordPiece vocabulary.
- CANINE (tokenization-free; Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation), which reads raw characters.
- A BPE-dropout variant.
All models are trained on EMDOT-style chronological splits over MIMIC-IV's year groups.
Fourth, we probe each trained model's representations for the limitation decision. We also run a redaction stress test. We remove a curated keyword lexicon, then measure how well a probe can still recover the decision from what remains. We then test variants with spelling noise and abbreviations. This shows whether recovery runs through subword fragments and whether it differs by tokenizer.
Baselines
The naive reference point is the standard fixed-window, random-split protocol of the two MIMIC-IV language-model papers. For fine-tuned models on mortality, Revisiting the MIMIC-IV Benchmark reports AU-ROC of about 0.87–0.88 and AU-PRC of about 0.43–0.46. That task uses templated structured data rather than free notes. We found no reported result on Lacuna for note-based mortality under a decision-anchored cutoff. We will re-run this baseline on our cohort, splits, and compute, so that its number is directly comparable to ours.
The strongest existing competitor for the question "does the model predict death from physiology?" is a structured-only logistic regression on vital signs and labs at the same cutoff. EMDOT found linear models to be the most stable over time, and these models cannot read notes. If note models under the decision-anchored cutoff do not beat this regression on AUROC and AUPRC, then the notes carried mostly decision signal.
Three further baselines are strong simple fixes:
- Keyword redaction under the fixed cutoff.
- Fixed truncation at 24 hours.
- Exclusion of patients with any limitation decision, following the Paris neuro-ICU design.
Each gets the same tuning budget. We also report true positive rate at a false positive rate of at most 0.05, the PhysioNet 2023 coma-prognosis metric, where the winning team scored 0.792 on a different task (PhysioNet 2023).
The key ablation isolates the anchoring. We hold the tokenizer and model fixed and change only the cutoff rule.
Why not just…?
Why not just redact DNR, comfort-care, and goals-of-care keywords? This is the cheapest fix, and if it works the tokenization part of the proposal is unnecessary. Our redaction stress test settles the question. If probes cannot recover the decision after redaction, and redacted models match decision-anchored models in AUROC, we recommend redaction and drop the tokenizer claim. We expect partial failure. Abbreviations, misspellings, and paraphrases such as "family wishes to focus on comfort" escape the lexicon. Subword models may also pick up fragments of these phrases, given how weakly subword embeddings track the words they come from (Tokenization Falling Short).
Why not just truncate notes at 24 hours? This is a real threat. Truncation is simple, and it may remove most decisions. The CENTER-TBI figures argue otherwise for some subgroups: among early withdrawals, the median time to withdrawal was 24 hours. We will report the fraction of decisions that fall before each fixed cutoff. If that fraction is small and truncated models match anchored ones, truncation is the answer and we will say so.
Why not just exclude patients whose treatment was limited? Exclusion removes the patients for whom the prognosis question matters most. It also changes the population the model is evaluated on. We report it, but we prefer stratified evaluation: we keep everyone and report AUROC separately for deaths with and without a preceding decision.
Why not use a causal estimator such as the domain pivot directly? Clinicians who trust a prognosis act almost deterministically. Under near-deterministic treatment the variance of propensity-weighted estimates explodes (causal monitoring), and positivity fails. We therefore treat causal correction as a later step, not as the first measurement.
Decisive Checks
Three results will be taken as evidence for the main claim:
- The gap in AUROC and AUPRC between the fixed and decision-anchored protocols is substantially larger for note models than for the structured-only regression.
- Discrimination for deaths with a preceding limitation decision clearly exceeds discrimination for deaths without one.
- After keyword redaction, the decision can still be recovered from the representations above chance, and recovery differs across the BPE, WordPiece, and character models.
Two results would refute the claim: a small gap between the protocols, or a gap that redaction or truncation alone closes. Either outcome is still useful, because it would establish that current note benchmarks are measuring biology rather than decisions.
References
- Occurrence and timing of withdrawal of life-sustaining measures in traumatic brain injury patients: a CENTER-TBI study
- Revisiting Clinical Outcome Prediction for MIMIC-IV
- Revisiting the MIMIC-IV Benchmark: Experiments Using Language Models for Electronic Health Records
- When accurate prediction models yield harmful self-fulfilling prophecies
- Evaluating and Correcting Performative Effects of Decision Support Systems via Causal Domain Shift
- Designing monitoring strategies for deployed machine learning algorithms: navigating performativity through a causal lens
- Evaluating Model Performance in Medical Datasets Over Time
- Multimodal assessment improves neuroprognosis performance in clinically unresponsive critical-care patients with brain injury
- Tokenization Falling Short: On Subword Robustness in Large Language Models
- Specialised or Generic? Tokenization Choices for Radiology Language Models
- Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
- Predicting Neurological Recovery from Coma After Cardiac Arrest: The George B. Moody PhysioNet Challenge 2023