Clinical early-warning models, such as those that flag sepsis or kidney failure in the intensive care unit, are almost always scored on historical data in which every lab value and vital sign sits at the moment it was measured. At the bedside, a value is usable only once the lab has run the sample, a nurse has charted the reading, and the hospital's data feed has delivered it. Sometimes the feed does not deliver it at all. The machine-learning literature on real-time inference treats latency as a compute problem: make the network finish before the deadline, or make it return a coarse answer when it is interrupted. In hospitals, the larger delay is usually in the data rather than in the arithmetic.
This note proposes to treat both kinds of delay as one budget. The first piece is an evaluation harness that replays historical patient records in the order and at the times their values would actually have become available, with realistic feed outages added in. The second piece is a "data-anytime" risk model: one network trained to give a calibrated prediction from whatever subset of inputs has arrived by the decision deadline. Together they test a specific claim. Much of the well-documented drop between retrospective and prospective performance comes from evaluating models as if data latency were zero. If that is right, a replay harness should predict prospective performance better than standard retrospective splits, and models trained for partial availability should lose less when deployed.
Core Claim
The strongest evidence for this framing comes from Mind the Performance Gap, which deployed a C. difficile risk model prospectively on 26,864 encounters. It found a retrospective-to-prospective AUROC (area under the ROC curve) drop from 0.778 to 0.767. Infrastructure shift accounted for a loss of 0.016, and temporal population drift contributed +0.005. Infrastructure shift there means differences in how and when data reach the model. The study also found that research warehouses overwrite fields with later values, and that the live pipeline failed on 10 of 356 days.
That paper measured the gap but treated the infrastructure as a black box and did not change the model. Separately, the real-time literature shows how to build predictors that degrade gracefully when computation is cut short. Anytime Depth Estimation adds early-exit paths that return a usable depth map when interrupted. LayerSkip and CEEBERT do the same for transformer layers. These works supply the method in spirit, but they are interrupted on compute, not on data.
Our claim is twofold. First, replaying records at availability time, with outage injection calibrated to the rates observed in that deployment, recovers the sign and most of the size of the prospective gap that ordinary retrospective evaluation misses. Second, a single model trained under randomized availability gives better-calibrated predictions under unseen delay patterns and outages than models trained on fully observed or fixed-shifted data.
Research Program
The first component is the replay harness. Each input carries two timestamps: when it was measured, and when it became visible to a downstream system. Given these, the harness reconstructs at each decision time $t$ the set $A(t)$ of inputs whose availability time is at most $t$. The model sees only $A(t)$ and must answer by a deadline. The harness also records the end-to-end delay of each prediction as
$$ L = L_{\text{data}} + L_{\text{pipeline}} + L_{\text{compute}}, $$
where $L_{\text{data}}$ is the lag from measurement to result availability, $L_{\text{pipeline}}$ is the extract-and-transform lag of the feed, and $L_{\text{compute}}$ is model execution time on stated hardware. Pipeline outages are injected as whole-day or whole-feed blackouts at rates matched to the 10-in-356 figure from Ötles et al. They can be re-tuned when a partner site supplies its own logs.
On MIMIC-IV, we will use recorded result-storage times where the tables provide them, and we will verify field by field that these reflect availability rather than later edits. Where true availability times are absent, as in the hourly-binned PhysioNet/CinC 2019 sepsis data, we fit per-variable delay distributions from MIMIC-IV and impose them. These results are clearly labelled as simulated.
The second component is the data-anytime model. It is a sequence model over time-stamped events that is trained on many replays of each patient. In each replay, availability times are resampled from the per-variable delay distributions, and random feed blackouts are added. The loss is averaged over decision times and resampled availability, so the same weights must be accurate and calibrated across every plausible pattern of missing inputs. This is the anytime idea moved from layers to inputs: information is withheld instead of layers being skipped.
The third component is a deferral wrapper. We reuse stratified conformal prediction from Act or Defer, which guarantees class-specific error rates by splitting cases into "act" and "defer to a clinician." That work calibrates once, offline, on fully observed EHRSHOT data. We instead calibrate within strata of availability, for example "lactate not yet resulted." The error guarantee then holds for the information state the model is actually in, not for the average over complete records.
Compute latency enters through measurement, not assumption. We profile the model, a gradient-boosted baseline, and a structured-EHR foundation model on the same hardware. For the foundation model we use CLMBR-T-base from EHRSHOT. The question is whether $L_{\text{compute}}$ is ever a non-negligible share of $L$ for hourly early warning. If it is, layer-level early exits become a follow-on. If it is not, that is itself a useful negative result for the field's compute-centric framing.
Baselines
The naive reference is the standard retrospective pipeline. It trains on warehouse data with every value at measurement time, fills missing values by carrying the last value forward, and evaluates on a random or temporal split. The proposal's first claim is that this reference overstates deployed performance. Its retrospective score is the number every other result is compared against.
The strongest existing modeling competitor is gradient boosting on hourly features, as used for the hourly dynamicEWS in the renal replacement therapy early-warning system. That system reports AUROC 0.97 for predicting RRT within 24 hours, evaluated retrospectively on MIMIC-III, MIMIC-IV and eICU, with about 1.6 false alarms per true positive. A second strong reference for sepsis is the PhysioNet 2019 challenge, where winning teams reached utility between 0.33 and 0.36 on the hidden test hospital (Reyna et al.). The HEA-LSTM of Liu et al. reports AUC 0.8410 locally and utility 0.321 on the official test set.
We will reimplement gradient boosting on our task definitions under the same splits, the same feature set, and an equal hyperparameter-search budget of trials. We will report both its standard retrospective score and its replay score. Beating it means matching its replay AUROC within 0.01 while reducing calibration error and the utility loss caused by outages. We do not expect to beat its retrospective AUROC, and we do not claim we will.
The most important baseline is gradient boosting trained on availability-shifted features. Each variable is lagged by its median turnaround, and one model is trained per decision horizon. This is the cheapest version of our idea. The comparison between it and the data-anytime model isolates what the randomized-availability training adds beyond simply knowing the median delays.
Two ablations complete the set. The first trains our sequence model on fully observed data and evaluates it under replay, which isolates the training procedure. The second applies the conformal wrapper with a single global calibration instead of availability strata, which isolates whether stratification is needed for the error guarantee to hold.
Why not just…?
Why not just lag each feature by its typical turnaround and retrain XGBoost? This is a real threat. It may capture most of the benefit for the average patient. It is fixed to median delays, however, so it cannot respond when a specific lab is unusually late for a specific patient. It also has no principled behaviour on outage days, when whole feeds vanish.
The experiment that settles this compares the two models on replay days with delays in the upper tail and on injected blackout days. If the shifted XGBoost matches the data-anytime model there too, the modeling contribution collapses. The proposal then reduces to the harness and the stratified deferral wrapper, and we would report it that way.
Why not just make the model faster with early exits or a fast/slow split? Dual-AEB pairs a rule-based braking reflex with a multimodal language model that still needs 1.21 s on a Jetson Orin. VLMLight routes rare emergencies to a slow language-model team. Both target sub-second control loops. For hourly ward or ICU risk scores, lab turnaround is plausibly measured in tens of minutes or hours, which would make a second of compute negligible. We do not assume this. The $L$ decomposition measures it directly, and a fast/slow design is pursued only if $L_{\text{compute}}$ turns out to matter.
Why not just run a prospective silent trial and skip the harness? A silent trial is the ground truth, and we want one. But the Ötles et al. evaluation required a year of live deployment for one model, so it cannot be repeated for every design iteration. The harness is worth building only if it predicts what a silent trial would show. That is why the decisive experiment below tests the harness against a prospective measurement instead of trusting it.
Why not just use established severity scores? Rule-based scores have no compute cost, but they suffer the same data delays, since SOFA needs lab results. They also discriminate poorly at triage, where vital-sign rules give AUROC 0.67 in Horng et al.. We include a SOFA-style rule under replay as a floor, not as the main competitor.
Proposed Experiments
The first experiment asks how much of the retrospective score disappears once data latency is honoured. On MIMIC-IV, for sepsis onset and for RRT within 24 hours, and on PhysioNet 2019 with simulated delays, we report AUROC, precision-recall AUC, expected calibration error, challenge utility, and median lead time before onset. Each metric is reported under standard evaluation and under replay, for every baseline.
Precision-recall AUC matters because COP-IV showed AUROC near 0.87 coexisting with PR-AUC near 0.17 on rare outcomes. The prediction is that replay lowers utility and lead time much more than it lowers AUROC. This mirrors the Ötles finding that infrastructure effects hit discrimination while temporal effects hit calibration.
The second experiment tests robustness and the stratified deferral wrapper. Under injected outages and upper-tail delays, we compare calibration and utility across the data-anytime model, shifted XGBoost, and the naive pipeline. We also check whether availability-stratified conformal calibration keeps the target per-class error, for example α = 0.10, inside each information state while global calibration violates it.
The decisive check needs a deployment partner. We will seek a hospital already running a silent early-warning feed, mirroring the $D_{pro}$ versus $D'_{ret}$ design of Ötles et al. There we compute three numbers: the standard retrospective estimate, the replay estimate, and the observed prospective performance. The harness succeeds if the replay estimate's error against prospective AUROC and utility is at most half that of the standard estimate.
If no partner can be secured, the falsification test is weaker but still informative. We apply the harness retrospectively to the published CDI setting and check whether replay reproduces a drop of the reported 0.011 AUROC order. Any outcome claims beyond predictive performance would follow the warning of TREWS or FALSE?: observational alert-response comparisons are confounded by care processes such as lactate ordering. This project therefore makes no mortality claims without randomization.
References
- Mind the Performance Gap: Examining Dataset Shift During Prospective Validation
- Anytime Depth Estimation with Limited Sensing and Computation Capabilities on Mobile Devices
- LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
- CEEBERT: Cross-Domain Inference in Early Exit BERT
- Early Prediction of Sepsis from Clinical Data: the PhysioNet/Computing in Cardiology Challenge 2019
- Act or Defer: Error-Controlled Decision Policies for EHR Foundation Models
- EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models
- A deep learning-based early warning system for renal replacement therapy in the intensive care unit
- Early Prediction of Sepsis From Clinical Datavia Heterogeneous Event Aggregation
- Dual-AEB: Synergizing Rule-Based and Multimodal Large Language Models for Effective Emergency Braking
- VLMLight: Safety-Critical Traffic Signal Control via Vision-Language Meta-Control and Dual-Branch Reasoning Architecture
- Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning
- Revisiting Clinical Outcome Prediction for MIMIC-IV
- Randomized Clinical Trials or Convenient Controls: TREWS or FALSE?