When a clinical risk model built at one hospital is tested at another, the usual finding is that a linear risk score falls further behind gradient-boosted trees, an ensemble of small decision trees fitted one after another. The standard reading is that linear models are too rigid for real patients. That reading mixes two different failures. The first is wrong shape: an additive linear equation cannot fit curved or interacting effects even in the hospital where it was built. The second is shifted hospitals: the fitted weights stop being right because the new site measures, triages, or labels patients differently. The two failures call for different responses. Wrong shape argues for switching models. Shifted hospitals argue for re-estimating a few parameters with a modest number of local outcomes, and that response may help boosting just as much.
This note proposes a simple accounting procedure, run on four public intensive care unit (ICU) databases, that splits the external gap between logistic regression and boosting into those two parts. The procedure further divides the shifted-hospital part into what cheap recalibration recovers and what it does not. The practical output is a decision rule. Given a model, a new site, and a budget of local labels, it says whether switching model class, recalibrating, or refitting will actually close the gap. It also tests whether that answer can be predicted before deployment from source sites alone.
Core Claim
Our hypothesis is that, on harmonized multi-center ICU data, most of the external advantage of boosting over logistic regression is already present inside the development hospital, so it comes from wrong shape. We expect the part caused by the new hospital to be similar in size for both model families and mostly recoverable by re-estimating an intercept and a slope from a few hundred local outcomes. If this holds, the observation that "regression transports worse" is largely misattributed. If it fails, and logistic regression's transport loss turns out larger and concentrated in a few coefficients, we learn which predictors carry hospital-specific meaning. Either result is new because no existing comparison separates the two terms.
The decomposition uses log loss, a proper scoring rule that rewards both ranking and calibrated probabilities. Write $L_m(S \to T)$ for the log loss of model family $m$ fitted on source site $S$ and evaluated on held-out patients at target site $T$. Write $L_m(T \to T)$ for the same family fitted and evaluated within $T$. The quantity of interest is the external gap between logistic regression (LR) and gradient boosting (GB):
$$L_{LR}(S\to T) - L_{GB}(S\to T) = \underbrace{[L_{LR}(T\to T) - L_{GB}(T\to T)]}_{\text{wrong shape at the target}} + \underbrace{[\Delta_{LR} - \Delta_{GB}]}_{\text{differential transport loss}}$$
Here $\Delta_m = L_m(S\to T) - L_m(T\to T)$ is how much family $m$ loses by being built elsewhere. Each $\Delta_m$ is further split into the portion removed by logistic recalibration on $n$ target labels, which refits only an intercept and a slope on the model's output score, and the remainder, which we call coefficient drift. The identity is exact, so every term is estimable with bootstrap intervals.
Research Program
The data are the four databases harmonized by Yet Another ICU Benchmark: MIMIC-IV, a single US center; eICU, more than 200 US hospitals; HiRID, from Switzerland; and AUMCdb, from the Netherlands. That benchmark reports logistic regression mortality AUROC of 0.84 to 0.87, close to deep networks. It also found that cohort definitions and preprocessing mattered more than architecture. This finding motivates our premise that harmonization reduces the shifted-hospital term, and it constrains our design: we must fix one preprocessing pipeline across all model families, or the decomposition will absorb pipeline differences. The tasks are YAIB's in-hospital mortality and a binary length-of-stay task. We use static and summarized time-series features, so that a linear model sees the same inputs as boosting.
For every ordered pair of sites, giving 12 transport directions, we fit plain logistic regression, LASSO-penalized logistic regression, logistic regression with restricted cubic splines and pairwise interactions, and gradient boosting (LightGBM, the implementation YAIB already uses). We then compute the decomposition. The within-target fits $L_m(T\to T)$ use the target's own development split, so wrong shape is measured where the model is deployed. Recalibration is repeated at target label budgets of 50, 200, 1000, and 5000 patients. This traces how quickly each family's transport loss disappears.
The second component asks whether the shifted-hospital term can be predicted without target labels. We use multiple-source cross-validation, the leave-one-source-out scheme of Multiple-source cross-validation. That paper showed random k-fold cross-validation is optimistically biased when data come from several sources, and it supplied variance estimators. It was demonstrated on text reviews. We apply it to clinical tabular data: within three sites we estimate $\Delta_m$, then check it against the fourth, held-out site.
The third component is a per-coefficient drift analysis for logistic regression. We test which standardized coefficients differ across sites beyond sampling error. We then ask whether drift concentrates in protocol-sensitive laboratory variables, whose ordering patterns differ across hospitals, rather than in physiology such as age or vital signs. This links to the operational shift mechanism described for the Denmark–UK COVID-19 cohort.
Baselines
The naive reference point is the locked source model applied unchanged at the target. This is the derivation-then-lock protocol that external validation practice treats as standard. It defines the full transport loss that any intervention must reduce. Every model family is run in locked form first.
The strongest existing competitor is gradient boosting transported without updating. For this question, the most relevant reported figure is from Mortality Prediction in Severe Traumatic Brain Injury. Training on the CENTER-TBI China registry and validating on the European registry, XGBoost reached an external C-statistic of 0.88 (95% CI 0.87–0.88). The established regression scores CRASH and IMPACT reached 0.82 and 0.80, and that study reported that logistic regression degraded as predictors grew to 54. Prediction of Outcome in Acute Lower Gastrointestinal Bleeding Using Gradient Boosting reports 83% external accuracy for boosting against 71% for multiple logistic regression on severe bleeding, with 170 development patients. On the ICU databases themselves, the only reported number we found is YAIB's 0.84–0.87 mortality AUROC range across models. No reported result was found on Lacuna for directed site-to-site LR-versus-boosting log loss or calibration slope on these databases, so we will establish those numbers ourselves. For fairness, every family gets the same harmonized features, the same imputation, the same random-search budget of 100 configurations tuned only on source data, and the same target labels whenever updating is allowed.
The key ablations isolate the two terms. The spline-and-interaction logistic model tests the wrong-shape term: if it closes most of the in-target gap to boosting, the problem was shape, not the regression framework. Recalibration-only updating, compared with full refitting of all coefficients at matched label budgets, isolates coefficient drift. A pooled multi-source fit, trained on three sites and tested on the fourth, addresses YAIB's finding that eICU-trained models transport better.
Success has two forms. The first is a recalibrated or spline logistic model whose external log loss lies within the bootstrap interval of equally recalibrated boosting while being cheaper to update. The second is a source-only prediction of the transport term that ranks the 12 directions correctly, with Spearman correlation above 0.7.
Why not just…?
Why not just use XGBoost everywhere, since it wins externally? That choice is reasonable if the gap is wrong shape, and our first term measures exactly that. However, the CENTER-TBI and bleeding studies compared locked models. Neither recalibrated the regression, which the TBI paper leaves open, and neither measured within-target performance. A boosting advantage that disappears once both models are recalibrated is not a reason to give up interpretable coefficients. If the wrong-shape term dominates even after splines, we will say so, and the recommendation becomes boosting.
Why not just recalibrate the intercept and slope at each site, as clinical practice already recommends? This is the most serious threat, because recalibration could be the whole story. The proposal therefore treats recalibration as a measured component, not as a contribution. The experiment that would settle it is the comparison of recalibration-only with full refitting. If recalibration recovers nearly all of both $\Delta_{LR}$ and $\Delta_{GB}$ at 200 labels, the finding is that transport failure on harmonized ICU data is a calibration problem. All models are local argues for recurring local validation, citing the Epic Sepsis Model's drop from 0.73–0.83 to 0.60–0.64 AUC. That finding would support its position with quantified label costs.
Why not just pool more sites? YAIB shows that eICU-trained models generalize better. However, Cross Population Generalizability in Clinical Machine Learning reports that scale does not remove majority-population anchoring. We include pooled leave-one-out fits and apply the same decomposition, to see whether pooling shrinks the drift term or only the shape term.
Why not just train an invariant regression (IRM)? Empirical or Invariant Risk Minimization? shows that ERM and IRM coincide under pure covariate shift and that IRM needs many environments. Four sites are far too few for its guarantees. We therefore treat IRM as out of scope. The coefficient-drift analysis tells us whether there is any non-covariate shift for an invariant method to exploit.
Proposed Experiments
The primary experiment computes the full decomposition for mortality across all 12 directed site pairs. It reports log loss, AUROC, calibration intercept and slope, and Brier score, with 1000 patient-level bootstrap intervals. The first decisive check is whether the wrong-shape term accounts for more than half of the LR–GB external gap in most directions. If it does, the "regression transports badly" narrative is largely misattributed. The second check is the label-budget curve. We locate the target sample size at which recalibrated logistic regression becomes statistically indistinguishable from recalibrated boosting, or show that no such size exists below 5000. The third check is predictive: whether leave-one-source-out estimates of $\Delta_m$ from three sites correctly order the actual transport losses at the held-out fourth site.
A secondary replication applies the same decomposition to the CENTER-TBI China-to-Europe transfer, if registry access is granted, and compares it with the published 0.88 versus 0.82/0.80 figures. Because those two registries share a collection protocol, we expect a small drift term there. A large one would show that protocol harmonization alone does not stop regression weights from drifting. The full set of results, for whichever hypothesis survives, gives practitioners a measured basis for choosing between switching models and spending local labels.
References
- Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML
- Multiple-source cross-validation
- Developing and validating COVID-19 adverse outcome risk prediction models from a bi-national European cohort of 5594 patients
- Mortality Prediction in Severe Traumatic Brain Injury Using Traditional and Machine Learning Algorithms
- Prediction of Outcome in Acute Lower Gastrointestinal Bleeding Using Gradient Boosting
- All models are local: time to replace external validation with recurrent local validation
- A foundation model for generalizable disease detection from retinal images
- Empirical or Invariant Risk Minimization? A Sample Complexity Perspective