Many published ICU models that predict death from clinical notes are built on a filtered population. They include only patients who stayed in the ICU for at least 48 hours and had at least one note written. A patient who died on the first night, or was stable enough to leave by morning, never enters either training or evaluation. This is survivorship bias: the dataset keeps only the entities that lasted long enough to be recorded. Finance has long documented the same problem. Backtests that drop bankrupt firms make mean reversion look stronger than it is (Borodin et al.), and backtests run only on surviving constituents misstate portfolio risk (Boyd et al.). In the ICU, the analogue of the delisted stock is the patient who deteriorated fastest, which is exactly the patient a risk alert exists to catch.
This note proposes to measure how much of the reported value of clinical text for ICU prognosis depends on this filter, and to replace the filter with a simple protocol that keeps everyone. We rebuild cohorts so that every patient still in the ICU at each hour is scored, whether or not anyone has written a note yet. We then ask two questions: does the text advantage survive, and how much of it comes from the timing and frequency of charting rather than from what the notes say?
Core Claim
Our hypothesis is that filtering cohorts on survival and documentation conditions on something tied to the outcome itself. This filtering makes note-based models look better calibrated and more useful than they are at the moment of deployment, when there is no filter. The proposed fix is a landmark protocol, borrowed from survival analysis. At each prediction hour, the landmark, the population at risk is every patient still in the ICU. Each patient's input is whatever was recorded before that hour, including the fact that nothing has been written yet. Discharge alive is treated as a competing event rather than silently dropped.
The claim is falsifiable in both directions. If text gains measured on filtered cohorts hold up under the landmark protocol, that strengthens the case for text-based stratification. If they shrink or move to different patients, a decade of benchmark numbers needs re-reading. A second, sharper claim follows DeepJoint, which showed that when clinicians order lab tests carries hazard information. We expect a large share of the text signal to be "textual clinical presence": how often and when notes are written. A model that sees only note timestamps and note types, not words, should recover much of the gain.
Prior Work and the Gap
Three papers motivate the gap directly, because each conditions on survival.
- Using Clinical Notes with Time Series Data for ICU Management fuses an LSTM over vital signs with a CNN over notes. It restricts MIMIC-III to patients with at least 48 hours of data and at least one note, which leaves 11,579 patients for the mortality task.
- Clinical XLNet models the sequence of nursing and respiratory notes within a fixed 48-hour ventilation window. It therefore never scores patients who died or were extubated before that window closed.
- FTL-Trans handles irregular gaps between notes with a learned time-decay. It treats the gaps as external facts about time rather than as decisions clinicians make because a patient looks unstable.
None of the three reports performance on an unfiltered intake cohort.
Two papers supply methods we reuse.
- DeepJoint jointly models survival alongside which lab tests are ordered and when. It does this only for 17 structured lab variables over the first 24 hours. We carry its idea of observation-as-signal over to notes, without its multi-task machinery unless the simpler version proves insufficient.
- CRL-MMNAR shows that note absence is informative in MIMIC-IV, where 24.5% of patients lack discharge summaries and documented patients are sicker. It encodes that absence as a static per-patient gate. We differ by making documentation state change over time within the stay, and by asking a population question (who is in the cohort) rather than a representation question.
Two further papers constrain the design.
- Instability in clinical risk stratification models using deep learning shows that the list of highest-risk patients can churn across random seeds (top-K Jaccard below 0.60 for diabetes) while ROC-AUC stays near 0.91. So we report the stability of the flagged list, not only aggregate discrimination.
- The scoping review of EHR bias names referral and admission bias as the most common category (35%) and flags an empirical evidence gap. This project is one concrete measurement inside that gap.
The finance work on retaining failed entities, as in FinRL-Meta, is the methodological analogy: keep the dropouts in the evaluation universe by construction.
Research Program
Cohort construction. We build the landmark cohort from MIMIC-III, and replicate it on MIMIC-IV.
- Every adult ICU stay enters at admission.
- At landmarks of 4, 8, 12, 24, 48 and 72 hours, each patient still in the ICU becomes one prediction instance.
- Each instance uses vitals, labs and notes with timestamps strictly before the landmark.
- The label is death within a fixed horizon. Discharge alive and transfer are competing events.
- Discharge summaries and any note without a reliable timestamp are excluded to prevent leakage. Many MIMIC-III notes carry only a date, so the leakage audit is a first-class deliverable, not a footnote.
Patient strata. Each patient is tagged as "would have survived the published filter" or "would have been excluded." The excluded group is split further into early death, early discharge, and never-documented.
Encoders. Three text encoders span the range from cheap to expensive:
- bag-of-words with negation handling, following the ED sepsis trigger;
- a ClinicalBERT encoder with time decay in the style of FTL-Trans;
- the CNN-plus-LSTM fusion of Khadanga et al.
Each is trained under two regimes: on the filtered cohort, as in the original papers, and on all landmarks.
Presence-only model. This model receives counts of notes by type, time since the last note, and a "no note yet" indicator, but no words. It isolates how much of the text gain comes from the act of documenting rather than from its content.
Baselines
Naive reference. A structured-only logistic regression on vitals, labs and demographics, trained and evaluated on the full landmark cohort. It represents what one gets with no text and no filter. It also matches the stability advantage that Instability in clinical risk stratification models using deep learning attributes to linear models.
Strongest existing text-plus-time-series method on this task: the fusion model of Using Clinical Notes with Time Series Data for ICU Management. It reports in-hospital mortality AUCPR rising from 0.487 to 0.525 when notes are added, on its filtered MIMIC-III cohort.
Strongest text-only sequence model: FTL-Trans. It reports 95.00% AUROC for in-hospital mortality on its own cohort of 10,574 patients.
Clinical presence on structured data: DeepJoint. It reports a 1-day C-index of 0.878 using 17 labs.
These published numbers come from different cohorts and are not directly comparable to each other or to ours. We therefore re-implement each model under identical splits by patient and identical inputs, with the same hyperparameter search budget and five seeds per configuration. We compare against our own reproductions, and also check that our filtered-cohort reproduction lands near the published figure.
Ablations. Three ablations isolate the key ingredients:
- Filtered versus full training: the same model trained on the filtered cohort versus all landmarks, both evaluated on all landmarks.
- Presence only: the presence-only model described above.
- Content scrambled, timing kept: note text shuffled across patients while each note's timestamp and type are preserved.
What counts as success. We would count the reframing as validated under either of two outcomes:
- The filtered-trained model loses at least several points of landmark AUPRC, or shows clearly worse calibration slope, on the excluded strata relative to the all-landmark model.
- The presence-only model recovers most of the text-over-structured gain.
Why not just…?
Why not just drop the 48-hour filter and add a missing-note indicator? That is essentially the core of our method, and we chose it deliberately because a reviewer would suggest it. The research contribution is not a clever architecture. It has three parts: the measurement of what the filter hid, the strata breakdown of where the error lives, and the test of whether text content matters once timing is included. If this simple version matches anything more elaborate, that is the expected result, and we report it as such.
Why not just reweight the filtered cohort by the inverse probability of surviving the filter? Reweighting can correct population averages for patients who resemble survivors. It cannot produce a usable prediction for a patient at hour 6 who has no notes and may die before hour 48. That patient is outside the filtered model's input space altogether. We still run the reweighted filtered model as a comparison. If it matches all-landmark training on the strata that both can score, then the bias is purely a population shift, which is itself informative.
Why not just use structured data, since text may only help survivors? This is a real threat. The ED sepsis trigger saw AUC rise from 0.67 to 0.86 with free text, and Khadanga et al. found notes improved AUCPR. Both effects were measured on populations where notes existed. It is plausible that in the first hours, before notes accumulate, text adds nothing. The landmark-by-landmark comparison against the structured-only baseline settles this directly. A null result at early landmarks would be a useful negative finding for deployment.
Why not just fit a Cox model with time-varying covariates? Landmarking with a fixed horizon is a close cousin of that approach. We include a Cox model with time-varying covariates as an extra structured baseline. The question this project asks is about cohort definition, not survival model family. Anaya et al. show that Cox can miss non-monotonic risk, which is why we do not rely on it alone.
Decisive Checks
Primary measurement. We report landmark-specific AUROC, AUPRC, and calibration slope for every model, broken down by the four strata. The single decisive figure is the gap between filtered-trained and landmark-trained models on early deaths at the 4-, 8- and 12-hour landmarks. This is the group no published note-based ICU model has been scored on.
Secondary checks.
- Recall of hyperacute deaths: the fraction of deaths within 48 hours that are captured in the top decile of risk.
- Presence versus content: text-only, presence-only, and content-scrambled comparisons. If the scrambled model performs within noise of the real-text model, the "extraction" value of notes in this setting is mostly charting behavior.
- Seed stability: top-K Jaccard and the range of demographic composition across seeds, following the instability study. The question is whether text encoders amplify churn in who gets flagged.
Replication. We repeat the audit on MIMIC-IV, using the note-missingness structure documented by CRL-MMNAR. This tests whether the effect depends on one database's documentation habits.
References
- Can We Learn to Beat the Best Stock
- Markowitz Portfolio Construction at Seventy
- DeepJoint: Robust Survival Modelling Under Clinical Presence Shift
- Using Clinical Notes with Time Series Data for ICU Management
- Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation
- Time-Aware Transformer-based Network for Clinical Notes Series Prediction
- Causal Representation Learning from Multimodal EHRs under Non-Random Modality Missingness
- Instability in clinical risk stratification models using deep learning
- Inherent Bias in Electronic Health Records: A Scoping Review of Sources of Bias
- FinRL-Meta: Market Environments and Benchmarks for Data-Driven Financial Reinforcement Learning
- Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning
- Characterization of Non-Monotonic Relationships between Tumor Mutational Burden and Clinical Outcomes