Hospitals increasingly build risk models that read doctors' and nurses' free-text notes. They work in two stages: a language model first pulls out clinical facts, such as symptoms, suspected infection source, or mental status, and a second model reasons over those facts across a patient's stay. Such systems degrade as years pass, because disease definitions change, coding rules change, and people write notes differently. The standard remedy is cheap. It keeps the model and re-fits only the mapping from score to probability, which is called recalibration. The literature supports this remedy for structured data: temporal drift mostly breaks calibration, not patient ranking (Systematic Review of Approaches to Preserve ML Performance under Temporal Dataset Shift). Nobody has asked whether this holds when the drift enters through the text-reading stage. It also has not been tested in the unit that matters clinically: net benefit, meaning the true positives gained minus a penalty for false positives, weighted by how risk-averse the clinician is at the decision threshold.
We propose to backtest note-based sepsis and mortality pipelines over a decade of MIMIC data, one calendar period at a time. The net benefit lost in each period will be split into two parts: the part a threshold-local recalibration recovers, and the part it cannot recover because patients are now ranked wrongly. We will then trace the unrecoverable part to either the extraction stage or the reasoning stage. The hypothesis is sharp. Drift that originates in documentation shows up mainly as ranking loss, and a label-free signal from the extractor's own evidence-checking can flag it before outcome labels arrive. If recalibration recovers nearly all the lost benefit even for text pipelines, that is also a useful result: it tells hospitals they do not need to retrain their extractors.
Core Claim
The claim has two parts. First, net benefit loss under temporal drift can be decomposed on a clinically chosen threshold window into a recalibration-recoverable term and a residual ranking term, and the size of each term can be measured on real calendar-time backtests. Second, for extract-then-reason pipelines over clinical notes, the residual term is driven disproportionately by extraction drift, and the extractor's rate of failing to ground its outputs in evidence spans tracks that residual better than feature-distribution or AUROC monitors do.
Let $NB(p)$ be net benefit at threshold probability $p$: true positives per patient minus false positives per patient times $p/(1-p)$. Let $[p_{\min}, p_{\max}]$ be the threshold range clinicians consider plausible for acting. We define windowed standardized net benefit as
$$\overline{NB}(f, D) = \frac{1}{\pi_D (p_{\max}-p_{\min})}\int_{p_{\min}}^{p_{\max}} NB_{f,D}(p)\,dp,$$
where $f$ is the model, $D$ is a period's data, and $\pi_D$ is that period's outcome prevalence. Dividing by prevalence keeps changes in base rate from masquerading as model decay. This integral is, up to a constant, the bounded-threshold Brier regret of A Consequentialist Critique of Binary Classification Evaluation Practices, so we can compute it with their briertools package. Let $D_0$ be the training period and $D_t$ a later period, and let $g_t$ be a monotone recalibration map fit on a small labeled slice of $D_t$ using only this window's loss. The drop then decomposes as
$$\overline{NB}(f,D_0)-\overline{NB}(f,D_t) = \underbrace{\big[\overline{NB}(f,D_0)-\overline{NB}(g_t\circ f,D_t)\big]}_{\text{ranking loss}} + \underbrace{\big[\overline{NB}(g_t\circ f,D_t)-\overline{NB}(f,D_t)\big]}_{\text{recoverable by recalibration}}.$$
A monotone map cannot reorder patients, so the first term is exactly the loss that only a model change can fix. This makes precise a tension the literature leaves open. Calibration is the main determinant of net benefit (Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness). Yet some clinical drift destroys ranking outright, as with sepsis models losing up to 0.090 AUROC over 2008–2019 (Evaluation of domain generalization and adaptation … temporal dataset shift). No paper says how much of that ranking loss lands inside the window where decisions are actually made.
Positioning
Three strands of prior work motivate the gap without closing it. Evaluating Model Performance in Medical Datasets Over Time built EMDOT, a rolling "train on the past, test on the future" backtester. It showed that random splits overstate deployment performance and that complex models decay faster than logistic regression. However, it measures AUROC only, and it explicitly leaves free-text notes to future work. We reuse its backtesting protocol and extend it to notes and to net benefit. The MIMIC-IV domain generalization study above supplies the negative result that constrains our design. Invariance-seeking training did not beat plain empirical risk minimization, meaning ordinary loss minimization on pooled data. We therefore do not propose a drift-robust training method; we propose measurement and triage. Decision-Focused Evaluation of Worst-Case Distribution Shift showed that accuracy-oriented shift analysis captures as little as 14% of decision loss, but it worked on census data with top-$k$ allocation. We carry its message to threshold-based clinical decisions and to real, not adversarial, calendar drift.
On the modeling side, three works supply methods but were all evaluated on static splits and with AUROC or F1: the self-verification extractor of Self-Verification Improves Few-Shot Clinical Information Extraction, the long-context note model ClinicalMamba, and the sequential note model Clinical XLNet. The self-verification extractor runs four steps: extract, detect omissions, ground each item in an evidence span, then prune. Its grounding step is what we turn into a monitor. Dementia-R1 is the closest two-stage precedent: an extractor produces intermediate scores that a longitudinal reasoner consumes. Its authors acknowledge that extraction errors propagate downstream, but they never measure this over time. The monitoring side draws on Clinical AI quality improvement, which proposes control charts that flag sustained shifts in a running statistic. We adopt its CUSUM charts and apply them to extractor self-consistency statistics rather than to inputs or outcomes.
Research Program
We will build three model families on the same MIMIC-IV cohorts, split into its four roughly three-year anchor-year groups spanning 2008–2019. Because MIMIC shifts dates for de-identification, periods are approximate. We will state this limitation and treat period boundaries as coarse. The outcomes are sepsis and in-hospital mortality, predicted from the first 24 hours. Sepsis was chosen because prior work found it decays most, and it spans a known definition change.
The first family is a structured-only model built on the tabular variables used by the domain generalization study. The second is an extract-then-reason pipeline. A self-verification LLM extractor reads the early notes and fills a fixed schema of clinical concepts, and a reasoner consumes the extracted concepts alongside the structured data and their within-stay trajectory. We test two reasoners: logistic regression, and a small sequence model over time-stamped extractions. The third family is an end-to-end long-context note model: ClinicalMamba, fine-tuned on the concatenated notes.
For each family, trained on the earliest period, we compute the decomposition above in every later period. We also produce full decision curves against treat-all and treat-none, since a model is only useful where it beats both.
To attribute ranking loss to a stage, we intervene on the extractor while holding the reasoner fixed. Some schema concepts have structured counterparts, such as antibiotics from the medication tables or mental-status items from charted scores. For those concepts we swap extracted values for the structured ones. For all concepts we alternatively re-run the extractor with few-shot examples and prompts refreshed from the target period's notes. The fraction of ranking loss each intervention recovers is the extraction-attributable share.
In parallel, we log label-free statistics from each deployment period: the extractor's omission-detection rate, its evidence-grounding failure rate, and the fraction of extracted items pruned. We track them with CUSUM charts. The monitor question is whether these statistics, available the day notes are written, predict the period's ranking loss better than input-distribution drift or AUROC computed after labels arrive.
The threshold windows will be set with clinical collaborators before any results are seen: the risk range at which a sepsis alert would trigger a bundle, and the range at which a mortality estimate would prompt a goals-of-care conversation. We will report sensitivity to wider and narrower windows. The ranking-reversal case on the Breast Cancer Surveillance Consortium data in the consequentialist critique warns that conclusions can flip with the window.
Baselines
The naive reference point is the earliest-period model applied unchanged to every later period, scored against treat-all and treat-none on the same window. This is what a hospital running a frozen model actually gets. For the structured-only family, the strongest existing method is empirical risk minimization on pooled historical data. The domain generalization study found it beat both invariance-seeking and unsupervised-adaptation methods on MIMIC-IV. The reported number we anchor to is a maximum sepsis AUROC drop of 0.090 across 2008–2019 for these models, from Evaluation of domain generalization and adaptation … temporal dataset shift. We found no reported net benefit result on Lacuna for any temporally backtested clinical model, structured or text-based, so our structured baseline's windowed net benefit will be the first such number.
The second strong competitor is sliding-window retraining, meaning a refit on only the most recent period. EMDOT found it beats all-historical training on rapidly shifting COVID-19 data, though the file quotes no numeric value for this. We will run it with the same model classes, the same hyperparameter search budget, and the same labeled data that the recalibration map receives, plus the full prior period.
For the note models, the reference points are ClinicalMamba and Clinical XLNet as end-to-end readers. Their reported results are static and on other tasks: ClinicalMamba reports 91.8 ROCAUC on n2c2 cohort selection, and Clinical XLNet reports 0.779 AUROC for 90-day mortality on MIMIC-III. These numbers are not comparable to our cohorts. We cite them only to show the models are competitive readers. For the extractor, the reference is the self-verification pipeline's reported MIMIC-IV ICD-10 F1 improvement from 0.487 to 0.533 over single-pass extraction. Our ablation removes self-verification and extracts in a single pass, to test whether verification matters for drift and not only for static accuracy.
The key ablation for the monitor keeps everything fixed and substitutes the grounding-failure statistic with a standard input-drift statistic computed on note embeddings. All arms see identical patients, periods, labeled-slice sizes, and tuning budgets. Every net benefit difference is reported with patient-level bootstrap intervals.
The proposal "beats" the baselines in a specific sense, since it is not a new predictor. The decomposition must reveal ranking loss that AUROC-based evaluation misranks. Concretely, some pair of period and model should have a larger AUROC drop but a smaller in-window ranking loss, or the reverse. In addition, the grounding-based monitor must correlate with period-level ranking loss significantly better than the embedding-drift monitor.
Why not just…?
Why not just recalibrate every period and stop worrying? This is the real threat and the null hypothesis of the study. The systematic review supports it for structured models: calibration broke in 11 of 15 studies and recalibration restored it. If the second term of the decomposition dominates for all three families, we will report exactly that, together with the labeled-slice size needed, and the extractor never needs refreshing. What makes the null doubtful for text pipelines is the same review's finding that mitigation often fails to recover lost discrimination. Documentation changes plausibly alter which patients the extractor flags, not just how confident it is. The decomposition measures this directly rather than arguing it.
Why not just retrain on a recent sliding window? Retraining fixes ranking loss but hides its source. It needs outcome labels for a whole period, and for LLM extractors it is expensive and needs fresh annotations. We run it as a baseline. If it matches refreshed-extractor recovery at similar cost, the practical recommendation becomes "retrain the reasoner." The stage attribution still tells a hospital whether retraining the cheap stage is enough.
Why not just monitor AUROC or input drift with control charts? AUROC needs labels, so it arrives late, and it measures ranking across all thresholds rather than inside the decision window. The decision-focused shift paper found accuracy-oriented analysis captures as little as 14% of decision loss. Embedding-drift monitors fire on harmless stylistic change. Our monitor ablation compares these statistics head to head against realized ranking loss.
Why not skip extraction and read notes end-to-end? We include ClinicalMamba as an arm for exactly this reason. It may drift less, because it is not tied to a fixed schema, or more, because it has no intermediate check. If it dominates the pipelines in windowed net benefit across periods, that is a finding against the two-stage design. The pipeline's remaining advantage would then be only its label-free monitor, which we can test separately.
Proposed Experiments and Decisive Checks
The main experiment runs the rolling EMDOT protocol on MIMIC-IV across four periods, for two outcomes and three model families. It reports full decision curves, windowed standardized net benefit, and its two-term decomposition. A secondary replication on MIMIC-CXR reports from 2011–2020, using report-extracted findings to predict a downstream label, tests whether the pattern holds for a different document type.
Three checks are decisive.
The first check asks whether recalibration suffices. The hypothesis is supported if, for the extract-then-reason pipeline, the ranking term is at least a substantial share of total loss in some later period while staying small for the structured-only model. It is refuted if recalibration recovers nearly all loss everywhere.
The second check asks whether ranking loss traces to extraction. It is supported if swapping in structured values, or using the period-refreshed extractor, recovers most of the pipeline's ranking loss. It is refuted if the ranking loss persists with oracle extraction, which would place the drift in the underlying relationship between clinical state and outcome.
The third check asks whether the label-free monitor works. It is supported if the CUSUM on grounding failures signals in the periods with large ranking loss and stays quiet where recalibration suffices, and outperforms the embedding-drift ablation.
A negative answer on all three is still a clean result: text pipelines drift like tabular ones, and cheap recalibration is the right policy.
References
- Systematic Review of Approaches to Preserve Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine
- A Consequentialist Critique of Binary Classification Evaluation Practices
- Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare
- Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine
- Evaluating Model Performance in Medical Datasets Over Time
- Decision-Focused Evaluation of Worst-Case Distribution Shift
- Self-verification improves few-shot clinical information extraction
- ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notes
- Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation
- Dementia-R1: Reinforced Pretraining and Reasoning from Unstructured Clinical Notes for Real-World Dementia Prognosis
- Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare