Early-warning models for sepsis and other forms of deterioration are built from hourly streams of vital signs and lab values. Their scores during development consistently overstate how well they do at a new hospital. The PhysioNet/Computing in Cardiology 2019 sepsis challenge makes this gap visible. One team's LSTM-plus-boosting ensemble scored a normalized clinical utility of 0.401 in internal cross-validation but 0.313 on the hidden test set, which came from a hospital it never trained on (He et al.). An attention-based recurrent model reported a local utility of 0.4126 and a test utility of 0.321 (HEA-LSTM). Automated tuning makes this worse. A Bayesian optimizer, which picks the next settings to try by fitting a cheap statistical model to the scores of settings already tried, will faithfully maximize whatever number it is given. If that number is a within-hospital cross-validation score, the optimizer rewards models that exploit hospital-specific habits, such as how often a lab is ordered.
This note proposes building a stacked ensemble so that its members and its combiner are chosen together, and scored only on a hospital held out of training. A stacked ensemble is one where a second-level "meta-learner" is trained on the base models' held-out predictions to decide how much to trust each one. The bet is that what makes an ensemble robust at a new site is members whose mistakes there are different, and that neither tuning members one at a time nor changing the cross-validation split alone selects for that.
Core Claim
Joint, ensemble-aware Bayesian optimization of a stacked early-warning model, with its objective computed as the utility of the whole refitted stack on a held-out hospital, will shrink the gap between development and deployment scores. It will also improve held-out-site utility over two alternatives given the same training budget. The first alternative tunes members independently and then stacks or averages them. The second is a single well-tuned gradient boosted tree. The key prediction concerns how the two ingredients combine. Switching to a leave-one-hospital-out split helps every method, but the gain from optimizing members jointly should appear mainly under that split, because only there does the optimizer see which members fail differently on a new site.
Where this sits relative to prior work
The optimization method comes from Bayesian Hyperparameter Optimization for Ensemble Learning. That paper keeps a fixed-size ensemble and replaces one member at a time in rotation. It scores each candidate by the validation loss of the updated ensemble rather than the candidate alone, and it reached an average rank of 1.89 against 3.39 for the best single tuned model. It used plain averaging, a smooth margin loss, and static benchmarks. We keep its slot-replacement scheme but change three things: a fitted meta-learner replaces the average, a clinical utility on a held-out site replaces the margin loss, and hourly patient streams with site shift replace static data.
AutoPrognosis shows that Bayesian pipeline search pays off in clinical prediction, with an area under the ROC curve of 0.78 versus 0.70 for the MAGGIC clinical score. However, it works on static tables and builds a weighted ensemble from separately scored pipelines, so it is our "tune separately, combine later" competitor rather than our method.
Three PhysioNet entries define the empirical tension we test. The highest unofficial challenge score, 0.364 utility, came from a single family of XGBoost models tuned with a tree-structured Parzen estimator, a common Bayesian optimizer that handles conditional settings well (SailOcean). That team averaged five fold-models and tuned the decision threshold on utility. Notably, 62 of its 168 features encode measurement frequency, which is exactly the kind of signal we expect to depend on the hospital. He et al. and HEA-LSTM instead argue for deep temporal representations. All three show large drops from internal to external scores, which is the evidence motivating this proposal. The challenge description itself attributes such drops to differences between hospitals (Reyna et al.).
On the stacking side, two papers constrain the design. Gaussian Meta-Space Augmentation for Stacking Ensembles found that high-capacity tree combiners overfit small sets of held-out predictions, while linear combiners were already well regularized. Ensembling Tabular Foundation Models warns that ensembling can distort calibration. We therefore default to a low-capacity meta-learner and report calibration as a primary outcome. SuperLearner for MODS prediction stacks a deep-wide network with eight classical learners and selects members by Q-learning. It does not tune member hyperparameters jointly with the stack, and it does not score selections on an unseen site.
Research Program
The pool of possible members is a conditional search space that the tree-structured Parzen estimator explores. The first family is XGBoost on engineered window statistics and clinical scores, following SailOcean. The search decides whether the measurement-frequency features are included at all. The second family is an LSTM used as a feature extractor feeding boosted trees, following He et al. The third is an attention-pooled bidirectional LSTM, following HEA-LSTM. The fourth is logistic regression on the standard clinical scores. Each family carries its own hyperparameters.
The meta-learner is a logistic regression. Its inputs are the members' log-odds predictions plus two context variables, hours since admission and current measurement density, so it can learn that some members are more reliable early in a stay or when data are sparse. Held-out predictions for meta-learner training are produced with folds grouped by patient inside each training hospital. This prevents hours from the same patient from appearing on both sides of a split.
Let $m$ be the ensemble size and $\mathcal{S}$ the set of hospitals. When the optimizer proposes candidate $h$ for slot $j$, it builds the stack with slot $j$ replaced. For each hospital $s$ it then refits the members and meta-learner on all other hospitals and picks the alarm threshold $\tau_{-s}$ on those other hospitals as well. The objective is
$$J(h) = \frac{1}{|\mathcal{S}|}\sum_{s \in \mathcal{S}} U_s\big(g_{-s}(f_1,\dots,f_{j-1},h,f_{j+1},\dots,f_m);\ \tau_{-s}\big),$$
where $g_{-s}$ is the meta-learner fitted without site $s$ and $U_s$ is the PhysioNet normalized utility on site $s$. The utility rewards alarms within the actionable window and penalizes late detections and false alarms.
Choosing the threshold on the training sites matters. If it were tuned on the held-out site, as is easy to do by accident, the utility would quietly inflate. We also test a smoothed utility variant, replacing hard alarms with sigmoid-weighted ones, because Levesque et al. found that smooth objectives give the optimizer's internal model a better-behaved surface.
The budget is counted in member-model fits, since meta-learner refits are cheap. Every method in the comparison receives the same number of member fits.
Baselines
The naive reference points are the standard clinical scores used as alarm rules (SOFA, qSOFA, NEWS) and a never-alarm predictor. The never-alarm predictor anchors the normalized utility at its defined floor. These show whether anything learned matters at all.
The strongest single-model competitor is the SailOcean recipe: TPE-tuned XGBoost on the 168 engineered features, five fold-models averaged, and the threshold tuned on utility. Its reported normalized utility is 0.364 on the full hidden test set, with a local AUROC of 0.847 (SailOcean). The challenge winners' range was 0.33–0.36 (Reyna et al.). We will reimplement it under our splits, budget, and training-site threshold rule. Our method must beat our reimplementation, not the published number, because the published number was measured on a test set we may not be able to access. Beating it means a higher held-out-site utility, with a 95% interval from patient-level bootstrap that excludes zero difference.
The strongest ensemble competitor is "tune separately, combine later." Each family is tuned by TPE on its own score, and the top members are then combined three ways: uniform averaging, greedy weighted selection in the AutoPrognosis style, and the same logistic meta-learner we use. This competitor is run under both a random patient-grouped split and the leave-one-hospital-out split. It isolates whether joint optimization adds anything beyond a good split and a good combiner.
Two ablations isolate our key ingredients. The first is joint optimization with the objective computed on random patient-grouped folds instead of held-out hospitals. The second is joint optimization with uniform averaging, which is Levesque's original setting. A further ablation replaces the logistic meta-learner with XGBoost, with and without the meta-space augmentation of Nelson et al., to test whether combiner capacity helps or hurts under site shift.
Why not just…?
Why not just use one BO-tuned XGBoost, as the top PhysioNet entry did? This is a serious threat. SailOcean reached 0.364 with exactly that, and van de Water et al. chose a single LightGBM over Transformers and GRUs on ward telemetry. The case for an ensemble rests on Levesque et al.'s rank gap between ensembles and the best single model, and on SuperLearner beating every single learner. Neither result was measured on an unseen hospital. If our reimplemented single XGBoost matches the stack on held-out-site utility at equal budget, we will report that stacking does not pay off under site shift. That would itself settle a question the literature currently argues from in-distribution numbers.
Why not just tune members separately and stack them? That is what AutoPrognosis and most stacking papers do, and with a good combiner it may capture most of the gain. Our argument is that tuning a member on its own score favors members that are individually strong, and on a shifted site those members tend to share the same mistakes. Joint optimization instead scores members by what they add to the stack. The experiment that settles this is the matched comparison under the leave-one-hospital-out split, together with a measurement of the correlation of member errors on the held-out site.
Why not just switch every method to leave-one-hospital-out validation and skip joint optimization? This could plausibly explain most of any improvement, so the design is a two-by-two: split type crossed with optimization type. If the split's effect dominates and joint optimization adds nothing beyond it, the honest conclusion is that the evaluation protocol, not the ensemble method, is what matters. We would recommend that protocol for any early-warning model. The proposal claims the interaction, and the two-by-two is what tests it.
Proposed Experiments and Decisive Checks
The primary benchmark is the public PhysioNet 2019 training data from hospitals A and B. Each hospital in turn is held out as the unseen site. If the organizers' hidden hospital C remains available for scoring, we will submit there once at the end. MIMIC-IV, mapped to the challenge's 40 variables with Sepsis-3 labels, serves as a third, external site. Here differences in variable mapping are an acknowledged confound.
We report held-out-site normalized utility, AUROC, area under the precision-recall curve, false alarms per patient-day, and expected calibration error.
The decisive checks are three. First, the optimism gap, meaning the optimizer's best internal objective minus the held-out-site utility, should be clearly smaller for our method than for the separately tuned stacks; this is the original failure being targeted. Second, across all optimizer trials, the rank correlation between the objective and the held-out-site utility shows whether each objective actually steers toward deployment performance. Third, on the held-out site, members selected by joint optimization should have lower pairwise error correlation than members selected for individual strength. If the members are not more diverse there, the proposed mechanism is wrong even if the scores improve.
A secondary diagnostic tracks how often the search keeps the measurement-frequency features under each objective. The prediction is that the held-out-site objective drops them more often.
Continuous wearable telemetry is the intended deployment setting but not the test bed here. The ward wearable cohort (CASSANDRA/CASHEWS+, median AUROC 0.90) is not assumed to be accessible. The healthy-volunteer smartwatch data (Baigutanova et al.) has no outcomes, so it supports only a side study of how charging gaps change measurement-density features. It does not support evaluating deterioration prediction.
References
- Early Sepsis Prediction Using Ensemble Learning with Features Extracted from LSTM Recurrent Neural Network
- Early Prediction of Sepsis From Clinical Datavia Heterogeneous Event Aggregation
- Bayesian Hyperparameter Optimization for Ensemble Learning
- AutoPrognosis: Automated Clinical Prognostic Modeling via Bayesian Optimization with Structured Kernel Learning
- Early Prediction of Sepsis Using Multi-Feature Fusion Based XGBoost Learning and Bayesian Optimization
- Early Prediction of Sepsis from Clinical Data: the PhysioNet/Computing in Cardiology Challenge 2019
- Gaussian Meta-Space Augmentation for Stacking Ensembles in Multimodal IPMN Risk Stratification
- Ensembling Tabular Foundation Models: A Diversity Ceiling and a Calibration Trap
- Early prediction of MODS interventions in the intensive care unit using machine learning
- Continuous Multimodal AI with Wearable Vital Signs Predicts Postoperative Complications in the General Ward
- A continuous real-world dataset comprising wearable-based heart rate variability alongside sleep diaries