Intensive care units score patients in two ways. Generic severity scores such as SOFA and APACHE IV are the same checklists of organ failure and deranged vital signs everywhere, so they mean roughly the same thing at every hospital, but they miss what matters in a particular disease. Disease-specific scores such as POSMI for sepsis mortality pick up signals the generic scores miss, like hypothermia and lactate. The cost is fragility: they depend on labs that some hospitals order less often, and on disease definitions that change over time. This note proposes training one small model on each of the two kinds of evidence and combining them with a tiny second-stage model. That combiner is trained to give probabilities that are equally trustworthy across many hospitals, so at an unfamiliar hospital it learns how far to lean on the fragile, sharp signal versus the stable, blunt one.
The goal is not a higher AUC, the standard measure of how well a model ranks sick patients above well ones. Single gradient-boosted trees already rank ICU patients well. The goal is a risk estimate whose stated probabilities stay honest when the model is moved to a hospital or year it never saw. In practice, a "40% mortality risk" should still mean 40% there. That property matters for threshold-based care decisions, and existing clinical stacking work never measures it.
Core Claim
Stacked generalization means training a second-level "meta-learner" on the out-of-fold predictions of several first-level models. The claim here has three parts.
First, the first-level models should be separated by measurement scope, meaning generic physiology versus disease-specific variables, rather than by algorithm. Second, their training folds should be grouped by hospital, so that the meta-learner sees how each model degrades at an unseen site. Third, the meta-learner itself should be a low-capacity model penalized for calibration differences across hospitals.
Together these should make the combined sepsis model better calibrated at held-out hospitals and years than a single boosted-tree model trained on all the same features and recalibrated afterward, with no loss in AUC.
The design deliberately confines invariance pressure to a combiner with a handful of parameters. Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine found that domain-generalization methods applied to full ICU models lost to plain pooled training. Looking for Out-of-Distribution Environments in Critical Care found that smaller models transfer better across eICU hospitals. Both results argue against imposing invariance on a high-capacity learner, and they constrain this design toward a tiny combiner.
Relation to Prior Work
Three clinical stacking papers motivate the gap but do not fill it.
Explainable Stacking-Based Model for Predicting Hospital Readmission for Diabetic Patients stacks four tree ensembles under logistic regression and reports AUC 0.6736. Early prediction of MODS interventions in the intensive care unit using machine learning stacks nine learners with a Q-learning selector. Gaussian Meta-Space Augmentation for Stacking Ensembles in Multimodal IPMN Risk Stratification smooths an XGBoost meta-learner with synthetic Gaussian meta-features and reaches AUC 0.839. All three diversify by algorithm or imaging stream. None separates generic from disease-specific evidence, none reports calibration under site shift, and all use random rather than site-grouped out-of-fold splits.
Two further papers supply the problem and the method.
Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trap shows that logistic-regression stacking gets competitive AUC but the worst log-loss and calibration among ensemble variants, while greedy selection preserves calibration. That result is the threat this proposal must beat, and it is the reason greedy selection appears as a baseline. On Calibration and Out-of-Domain Generalization supplies the method. It introduces a penalty on calibration error computed within each training environment (CLOvE) and a multi-environment isotonic post-processing step. It proves, in linear-Gaussian settings, that calibration across environments rules out reliance on spurious features, and it reports roughly 4.5% out-of-domain accuracy gains from post-processing on Camelyon17 and FMoW. It was never applied to a stacking combiner or to tabular health records.
On the clinical side, Development and validation of a score to predict mortality in ICU patients with sepsis builds POSMI on eICU, with AUC 0.831 in development versus 0.728 for SOFA and 0.773 for APACHE IV, and 0.781 in an external US South cohort. That paper defines both evidence streams for this proposal. Machine learning for predicting acute myocardial infarction in patients with sepsis shows generic scores collapsing to AUC near 0.5 on a sepsis sub-question, which serves here as a sanity test.
Research Program
The first-level models are built as follows. The generic model is a small logistic model on SOFA and APACHE IV components. The disease-specific model is a gradient-boosted tree on the twelve POSMI variables plus missingness indicators for lactate, albumin and bilirubin. As an optional third learner, the integer POSMI score can be used directly. Each first-level model produces out-of-fold log-odds, $z_g$ for generic and $z_d$ for disease-specific. The folds are grouped by hospital, so every meta-feature is a prediction made on a hospital that model never trained on.
The meta-learner is a gated logistic combiner:
$$\text{logit}\, p = a + (w_g + u_g^\top m)\, z_g + (w_d + u_d^\top m)\, z_d,$$
The terms in this equation are:
- $m$ is a short vector of missingness and data-availability indicators;
- $a$, $w_g$, $w_d$ are scalar parameters;
- $u_g$, $u_d$ are small weight vectors that let trust in each evidence stream shift when disease-specific labs are absent.
The combiner is fit by minimizing
$$\mathcal{L} = \sum_{e} \text{LogLoss}_e + \lambda \sum_{e} \text{MMCE}_e,$$
where $e$ indexes training hospitals, or regions when hospitals are small. MMCE is the kernel-based calibration error from the CLOvE paper, and $\lambda$ is chosen by leave-one-region-out validation. With fewer than twenty parameters, the combiner cannot overfit the small per-hospital samples in the way the IPMN paper documents for tree combiners.
The analysis focuses on where the combined model moves weight toward the generic score, and whether that happens at hospitals with sparse lactate measurement. Those learned weights double as a clinically readable explanation.
Baselines
The naive reference is the POSMI integer score applied as published, alongside SOFA and APACHE IV. This anchors the clinical status quo: POSMI's development AUC is 0.831 and its external US South AUC is 0.781, per the POSMI paper.
The strongest competitor is a single CatBoost model on the union of generic and disease-specific features, in the style of E-CatBoost. That paper reports AUROC 0.83 to 0.91 across twelve eICU disease groups using first-24-hour data; no sepsis-specific calibration result was found on Lacuna for it or for POSMI. It is the strongest competitor because it sees every feature the proposal sees and can model interactions the partition forbids. For fairness it receives the same training hospitals and the same hyperparameter search budget as the two first-level learners combined. It is also evaluated with multi-hospital isotonic recalibration (the robust calibration post-processing of the CLOvE paper) fit on the same training hospitals.
The ensemble competitors are logistic-regression stacking over an algorithm-diverse pool of random forest, XGBoost, CatBoost and logistic regression, following the diabetic readmission design, and greedy forward selection following the tabular foundation model benchmark.
Three ablations isolate the key ingredients:
- Partition ablation: the same combiner over two first-level learners built on a random split of the features rather than a scope split. This tests whether the generic/disease-specific partition itself matters.
- Fold ablation: random rather than hospital-grouped out-of-fold splits.
- Penalty ablation: plain log-loss with the calibration penalty set to zero.
The proposal beats the strongest baseline if two conditions hold on held-out regions. Its worst-region calibration error and calibration slope must be better than recalibrated CatBoost's, with bootstrap intervals that do not overlap. Its pooled AUC must be no more than 0.005 lower.
Why not just…?
Why not just train one gradient-boosted tree on all features and recalibrate it? This is the real threat. The only reason to expect it to lose is that isotonic recalibration fit on training hospitals corrects average miscalibration but not site-specific shifts in feature availability. The partitioned combiner, by contrast, reweights evidence streams explicitly per patient. If the recalibrated tree matches the proposal on leave-region-out calibration, scope partitioning is unnecessary, and the paper will say so.
Why not just average the models or use greedy selection, which the tabular foundation model benchmark found avoids the calibration trap? Averaging cannot shift weight when lactate is missing. The benchmark's pool was near-redundant (pairwise error agreement Q = 0.961), whereas generic and disease-specific scores disagree strongly on sub-questions such as sepsis-associated myocardial infarction. The greedy baseline tests this directly.
Why not just refit a logistic regression with POSMI and APACHE IV as covariates, as Demystifying machine learning for mortality prediction would suggest? That model is the proposal with the gating and calibration penalty removed, and it is run as the penalty ablation. If it performs as well, the conclusion is that parsimonious rescoring suffices, which clinicians would welcome.
Proposed Experiments
The primary experiment uses eICU sepsis admissions. The model is trained on Midwest hospitals, mirroring the POSMI development split, and tested on held-out West and South regions. The outcome is ICU mortality. Metrics are AUC, expected calibration error, calibration slope and intercept, Brier score, and decision-curve net benefit at clinically used risk thresholds, all reported per hospital with worst-region values emphasized.
The second experiment tests temporal shift on MIMIC-IV, training on 2008 to 2016 and testing on later years in the protocol of the temporal shift study, which reported sepsis AUROC drops of up to 0.090. MIMIC-IV lacks APACHE IV, so the generic stream there is SOFA only.
The third experiment is a sanity check on the sepsis-associated myocardial infarction task from Fang et al. The best single model in that paper, a gradient-boosted classifier, reports AUC 0.838. If the gated weights do not drive the generic weight toward zero on this task, where generic scores sit near AUC 0.5, the combiner is not learning when to trust each lens.
The decisive check is the partition ablation on held-out eICU regions. If a random feature split calibrates as well as the generic/disease-specific split, the central claim fails.
References
- Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine
- Looking for Out-of-Distribution Environments in Critical Care: A case study with the eICU Database
- Explainable Stacking-Based Model for Predicting Hospital Readmission for Diabetic Patients
- Early prediction of MODS interventions in the intensive care unit using machine learning
- Gaussian Meta-Space Augmentation for Stacking Ensembles in Multimodal IPMN Risk Stratification
- Ensembling Tabular Foundation Models: A Diversity Ceiling and a Calibration Trap
- On Calibration and Out-of-Domain Generalization
- Development and validation of a score to predict mortality in ICU patients with sepsis: a multicenter retrospective study
- Machine learning for predicting acute myocardial infarction in patients with sepsis
- E-CatBoost: An efficient machine learning framework for predicting ICU mortality using the eICU Collaborative Research Database
- Demystifying machine learning for mortality prediction