Hospital early-warning models update a patient's risk every hour, but they learn from labels that are clinicians' decisions, not measured physiology. A model that predicts "renal replacement therapy (dialysis in the ICU) will be started within 24 hours" is trained on the timestamp at which some intensivist chose to start it. That choice depends on the patient's kidneys and on the doctor: how aggressive the hospital is, the era, the shift, the local protocol. A model trained on these timestamps with ordinary cross-entropy learns both at once. Its risk score is partly "how sick is this patient" and partly "how quickly would clinicians at this site act." That is acceptable inside one hospital. It is harmful when the model moves to a hospital with different habits, or when anyone reads the score as physiological severity.
We propose to model the decision label explicitly as a latent patient-risk trajectory that crosses an observer-specific, time-varying action threshold. One shared sequence model reads the physiology and outputs a risk curve. A small separate model represents how willing each hospital, era, and shift is to act. The two are fitted jointly by a discrete-time hazard on the observed decision times, and only the risk curve is used at deployment. The claim is that this split produces a risk estimate that transfers across hospitals better than a decision-trained model, that tracks objective deterioration better, and that is honest about how much of a "missed" alert is disagreement between practices rather than error.
Core Claim
Inter-observer variability in dynamic risk prediction mostly does not look like random label flips. It is a structured shift in when an observer acts. A threshold-crossing model is the right inductive bias for it. Let $r_\theta(x_{\le t})$ be the patient's latent risk at hour $t$ given physiology up to $t$, produced by a shared network with parameters $\theta$. Let $\tau_\phi(h, t)$ be the action threshold of hospital $h$ at time $t$, built from a hospital offset plus smooth functions of shift, weekday, and calendar era, with parameters $\phi$. We model the hourly probability that therapy is initiated, given that it has not been yet, as
$$ P(\text{initiate at } t \mid \text{not before}) = \sigma\big(a\,[\,r_\theta(x_{\le t}) - \tau_\phi(h,t)\,]\big), $$
where $\sigma$ is the logistic function and $a>0$ is a learned sharpness. Hospital offsets are constrained to average zero, so the risk scale is anchored to a pooled reference practice.
The key assumption must be stated up front. Any site-level difference in initiation that physiology does not explain gets assigned to the threshold. If one hospital's patients are sicker in ways the recorded variables miss, the model will wrongly call that hospital "aggressive." The experiments are built to test how much this assumption costs.
Relation to Prior Work
Two papers motivate the gap. Mamandipoor et al. build an hourly early-warning system for renal replacement therapy on MIMIC-III, MIMIC-IV, and eICU. They state that the initiation decision is "highly complex and subjective," yet train on it as hard ground truth. Their per-hospital AUROC spread of 0.86 to 0.91 across eight eICU facilities is the kind of site dependence we aim to explain. Learning Under Treatment-Induced Label Indeterminacy shows that clinician decisions can censor outcomes. They correct for it with expert counterfactual annotations, but only on a static admission snapshot. We work on hourly trajectories and replace their costly multi-expert annotation with structure learned from routine metadata (hospital, time).
The closest method is Learning under Temporal Label Noise (arXiv version). It corrects predictions with a time-varying flip matrix $Q(t)$ and cuts test error on moving from about 14.9% (static noise baselines) to about 4.2%. It assumes one global $Q(t)$ shared by all series, and the authors name that uniformity as their main limitation. We differ in two ways. Our noise is observer-specific. It is also directional: a threshold model says a cautious site starts therapy late, which produces late positives rather than symmetric flips.
The Latent Doctor Model supplies the conceptual move. It encodes each rater's style as a latent variable, using a conditional variational autoencoder (a model that learns a compressed probabilistic code) and reaching a KL divergence of 0.461 to the Gleason2019 expert panel. It needs doctor IDs and support sets, and it handles static images only. We use a much lighter observer model, a scalar threshold function, because ICU databases hold hospital and time metadata but rarely name the decision-maker.
Characterizing the Progression of Pulmonary Edema Severity is a constraint on our design. It shows that clinicians disagree on absolute severity grades but agree on direction of change. A threshold model inherits this property: rankings within a site are preserved even when thresholds differ. The HiRID-ICU-Benchmark is a second constraint. LightGBM on summary statistics beats deep sequence models there, reaching AUPRC 38.9 on circulatory failure. So our risk network will be gradient-boosted trees on rolling summaries, not a recurrent model, unless the deep version clearly wins.
Research Program
The work proceeds in three stages.
The first stage is the estimator. The hazard above is fitted by maximum likelihood over person-hours until initiation, discharge, or death, with death and transfer treated as censoring. The risk component is a gradient-boosted model on hourly rolling summaries of creatinine, urine output, potassium, bicarbonate, fluid balance, and vasopressor use, following the dynamicEWS feature set. It is fitted by alternating boosting steps on $r$ with gradient steps on the low-dimensional threshold model. A neural variant is kept for the trees-versus-deep question.
The second stage is a semi-synthetic testbed with known truth. We take real eICU and MIMIC-IV trajectories and define a "true" latent risk from an objective renal-failure rule built on measured creatinine and urine output. We then simulate initiation times by thresholding that risk with hospital offsets and shift drift of controlled size, including drift that is correlated with case-mix, which breaks our identifying assumption on purpose. This tells us when threshold recovery works and when it fails.
The third stage applies the method to real initiation labels across eICU hospitals and the two MIMIC eras. The outputs are an estimated practice-variation map, together with a deployable risk score whose cross-site behaviour we can test.
Baselines
The naive reference is a static admission score. We use APACHE IV, which Mamandipoor et al. report their system outperforms, though no numeric APACHE IV value is reported there. This gives the floor that any dynamic method must clear.
The strongest existing method is the dynamicEWS rolling XGBoost from Mamandipoor et al.. It reports AUROC 0.97 for 24-hour initiation prediction, 80% of events captured at 1.6 false alarms per true positive, and per-hospital screening AUROC of 0.86 to 0.91. We will reimplement it on identical cohorts, features, and splits, with the same hyperparameter search budget as our risk component. On in-distribution decision prediction we expect to match it, not beat it, because it is optimized for exactly that target. Beating it means a higher AUROC under leave-one-hospital-out evaluation and higher discrimination for objective renal deterioration, while staying within 0.01 AUROC in-distribution.
The temporal noise competitor is the forward temporal loss with continuous $Q(t)$ estimation from Learning under Temporal Label Noise, applied to the same tree or neural backbone. No reported result was found on Lacuna for this method on ICU intervention prediction. Its reported numbers come only from sensor datasets, such as 25.5% versus 31.1% error on Goodday. It gets the same tuning budget, and we also run a per-hospital $Q_h(t)$ variant so the comparison is about directional thresholds, not merely about adding a site index. The per-sample trust approach of LiNC is run as an ablation of "noise without observer structure."
The ablation that isolates our key ingredient is the same hazard model with $\tau$ fixed to a single global constant, which removes observer and time dependence. A second ablation keeps hospital offsets but drops time drift.
Why not just…?
Why not just add hospital ID and time of day as input features to XGBoost? This is the most serious threat. A flexible model with site features can absorb site effects. At deployment, though, it has no principled "reference site" setting. Its interactions between site and physiology are unconstrained, so setting the site feature to a pooled value need not give a site-free risk, and an unseen hospital has no valid input value at all. We run this baseline anyway, both with site set to the modal hospital and with predictions averaged over sites. If it matches our method on leave-one-hospital-out and on objective-outcome tests, then the explicit threshold structure buys only interpretability, and we will report that.
Why not just train on an objective label such as creatinine-defined severe kidney failure instead of the decision? This is also a real alternative. Objective criteria leave out the clinical reasons dialysis is started, such as refractory acidosis, fluid overload, or toxin removal, and the actionable question for clinicians is about need for therapy. We include an objective-label model as a baseline. Our method should beat it at predicting initiation and should not lose to it on predicting objective deterioration. If it loses on the objective test, the threshold model is not extracting physiology better than direct supervision does.
Why not just apply a temporal or static noise-robust loss? The evidence in Learning from Time Series under Temporal Label Noise shows that static corrections fail when noise varies over time. Their global $Q(t)$, however, cannot represent hospitals that disagree in a consistent direction. The semi-synthetic experiments with heterogeneous site thresholds will show directly whether a per-site flip matrix recovers the latent risk as well as a threshold does.
Why not just collect consensus labels from expert panels? Panel labelling of hourly trajectories across eICU-scale cohorts is not feasible at that scale. The Latent Doctor Model also shows that majority votes discard the ambiguity information that is the object of study here.
Proposed Experiments
The first decisive check is the semi-synthetic recovery test. We measure rank correlation between estimated and true latent risk, and error in recovered hospital offsets, as threshold heterogeneity grows and as case-mix confounding is introduced. The method should beat the global-threshold ablation and the per-site $Q_h(t)$ variant once threshold spread is non-trivial. We also chart how far it degrades under confounding.
The second check is leave-one-hospital-out evaluation on eICU against dynamicEWS, scored by AUROC, AUPRC, and calibration slope. Because initiation at the held-out site still reflects that site's own threshold, we report discrimination, which is invariant to a constant offset, and treat calibration as secondary.
The third check is objective validity on MIMIC-IV. We compute discrimination for subsequent creatinine-defined severe kidney failure and for 90-day mortality among patients never started on therapy, the group where decision-trained models are most likely to under-call risk. We also compare the MIMIC-III and MIMIC-IV eras to see whether estimated threshold drift follows known changes in practice.
The final check is face validity. Estimated hospital offsets should correlate with severity-adjusted initiation rates computed independently by a standard stratified analysis. We will report HiRID circulatory-failure results with the same machinery only as an exploratory transfer to a second decision-influenced endpoint.
References
- A deep learning-based early warning system for renal replacement therapy in the intensive care unit
- Learning Under Treatment-Induced Label Indeterminacy with Expert Annotations of Counterfactual Outcomes: A Case Study in Neurological Prognostication
- Learning under Temporal Label Noise
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Characterizing the Progression of Pulmonary Edema Severity: Can Pairwise Comparisons in Radiology Reports Help?
- HiRID-ICU-Benchmark - A Comprehensive Machine Learning Benchmark on High-resolution ICU Data
- LiNC: Lightweight Noise Correction via Adaptive Label Refinement