A clinical prediction model that works well at the hospital where it was built often performs worse at another hospital. The usual explanation is that the patients, clinicians, and equipment differ. A second explanation gets less attention: the two sites may have prepared their data differently. They may define "sepsis onset" at a different timestamp, exclude different patients, fill missing lab values differently, or bin vital signs over different windows. Each of these choices is known to move performance a lot within a single hospital. On MIMIC-III, three valid readings of the Sepsis-III onset time alone swing LightGBM's AUROC (area under the ROC curve) from 0.829 to 0.869. That is a larger effect than switching between tree models, recurrent networks, and survival models (Cohen et al., 2024). Nobody has measured how much of the cross-hospital gap these choices account for.
We propose to measure it directly. We will rerun one prediction task across five public ICU databases. In each run, the data-preparation pipeline at the training site and at the evaluation site is either deliberately matched or deliberately mismatched, one choice at a time. We then fit a variance-components model of the kind meta-analysts use for heterogeneity between studies. That model splits the transfer gap into a part caused by which hospital the patients came from and a part caused by mismatched preparation choices. The result is a number with clear practical stakes. If mismatch explains a large share, then reporting and sharing pipelines is a first-order fix for external validation. If it explains little, then harmonization tools have already done their job and the remaining gap really is about patients.
Core Claim
The core claim concerns the quantity that random-effects meta-analysis calls between-study heterogeneity, $\tau^2$: the spread in true performance across sites beyond what sampling noise would produce. Meta-analysis treats $\tau^2$ as an unexplained residual. We claim that in multi-site clinical machine learning, a measurable and controllable fraction of $\tau^2$ comes from pipeline mismatch rather than from population differences, and that this fraction can be estimated causally because we choose the pipelines ourselves.
For each cell of the design we record test AUROC $y$. A cell is defined by a training site $s$, a test site $t$, a training pipeline $p$, a test pipeline $q$, and a random seed. We model it as
$$ y = \mu + a_s + b_t + c_{st} + \beta^\top x_p + \gamma^\top x_q + \delta^\top m_{pq} + \varepsilon . $$
The terms are defined as follows. $a_s$ and $b_t$ are random effects for training site and test site. $c_{st}$ is a random site-pair interaction, which captures population shift. $x_p$ and $x_q$ encode the levels of each preprocessing factor. $m_{pq}$ is a vector of indicators, one per factor, marking whether training and test disagree on that factor. $\varepsilon$ is residual noise. We estimate the variance of $\varepsilon$ per cell by bootstrapping the test set. This handles the uneven noise that comes from very different site sizes and prevalences, in the same way meta-analytic weights $1/(\sigma_i^2+\tau^2)$ do.
The quantity of interest is the share of the mean transfer gap that $\delta^\top m_{pq}$ accounts for, compared with $c_{st}$. A transfer gap is the drop from in-site to cross-site AUROC.
Relation to Prior Work
Several papers motivate the gap without closing it.
Yet Another ICU Benchmark (YAIB) harmonizes MIMIC-III, MIMIC-IV, eICU, HiRID, and AUMCdb. It reports cross-dataset transfer heatmaps and shows that cohort and preprocessing choices matter more than model class. However, it runs one fixed pipeline per task, so it cannot separate mismatch from population shift. We reuse its datasets, tasks, and harmonization layer, and add the pipeline as a manipulated variable on both sides of the transfer.
Cohen et al. (2024) and Johnson et al. (2017) supply the evidence that single-site pipeline choices matter. Johnson et al. found reproduced MIMIC cohorts deviated by more than 25% in half of 38 experiments, and that gradient boosting matched or beat published novel models in 82% of cases. Both studies are confined to one hospital system. Neither asks whether this sensitivity grows, shrinks, or interacts with site when models move between hospitals.
ACES and MEDS-DEV make cohort definitions declarative and identical across sites. They supply our method for writing pipelines as reproducible configuration files. Both assume a single agreed definition. Our design instead treats several reasonable definitions as experimental factors.
Bell et al. (2022) formalize "multiverse analysis", meaning evaluation over the full set of defensible analysis choices, using Gaussian-process surrogates on CIFAR-10 at roughly 1,138 GPU hours. They assume noise is the same everywhere and use a single dataset. Our multiverse spans sites, has uneven noise by construction, and uses cheap gradient-boosted models, so a fractional factorial grid can be run exhaustively.
Wang et al. (2021) argue that forcing heterogeneous MRI data into a shared representation discards useful signal. That is a constraint our design must respect. Matching pipelines need not always help, and a mismatch effect $\delta$ with a positive sign is a possible result we must report.
Research Program
The design rests on four pipeline factors, each drawn from documented sources of variation.
The first factor is the sepsis onset definition. It takes the three levels from Cohen et al.: organ-dysfunction time, suspected-infection time, or the earlier of the two. The second factor is the cohort exclusion rule, which we vary in minimum length of stay and minimum age, following the ambiguities Johnson et al. catalogued. The third factor is imputation, which takes three levels: forward fill, forward fill plus missingness indicators, and per-site median. We include it because missingness patterns differ sharply between high-resolution HiRID and lower-frequency eICU and MIMIC. The fourth factor is aggregation window, with hourly and four-hourly levels. This follows the discretization choices in MIMIC-Extract.
We implement every combination as ACES-style configurations on top of YAIB's harmonized data. This guarantees that "same pipeline" means byte-identical logic at every site.
Sepsis prediction is the primary task because its onset ambiguity is best documented. In-hospital mortality is a second task. Its label is less ambiguous, so we expect the mismatch share to be smaller there. If it is not, that would itself be informative.
The full product of training and test pipelines is too large to run. We therefore run three kinds of cells. All cells with matched pipelines, across all site pairs, identify $a_s$, $b_t$, $c_{st}$ and the main effects of each pipeline. Single-factor mismatches identify each component of $\delta$ cleanly. A random sample of multi-factor mismatches tests whether mismatch effects add up.
The model is LightGBM with a fixed Bayesian tuning budget per cell, as in YAIB, over five seeds.
Label-definition mismatch needs care, because AUROC against a different label is a different question. We therefore report two evaluation modes. In the "deployer's labels" mode, the test site scores the model against its own definition, which is the realistic deployment case. In the "reference labels" mode, every site is scored against a single shared definition. Comparing the two modes separates harm caused by shifted model inputs from harm caused by a moved target.
Baselines
The naive reference point is what external validation reports today. That means a single fixed pipeline, applied identically everywhere, with the transfer gap attributed entirely to the site. In our model this corresponds to dropping the pipeline terms. The decomposition is only useful if adding the mismatch terms explains a meaningful fraction of the cross-site variance this baseline leaves unexplained.
The strongest existing method is the YAIB protocol itself: harmonized extraction, a fixed cohort, and tuned LightGBM, evaluated across datasets. YAIB reports baseline mortality AUROC of 0.84 to 0.87 across datasets. For in-site sepsis on MIMIC-III, the strongest reported figure is LightGBM AUROC 0.869 under the suspected-infection onset (Cohen et al., 2024). The file contains no reported cross-dataset transfer AUROC values for YAIB, so no reported result was found on Lacuna for those cells. We will regenerate them using YAIB's own code before adding any variation, and our matched cells must reproduce them within seed noise.
The statistical baseline is a standard random-effects meta-analysis of per-site AUROC with no pipeline covariates. Our model beats it if the pipeline-mismatch terms reduce the estimated $\tau^2$ by a margin that clears a likelihood-ratio test. It must also predict AUROC in held-out mismatch cells more accurately.
The key ablation removes one factor at a time from $m_{pq}$. This shows which single choice carries the mismatch effect.
Why not just…?
Why not just standardize everything with ACES or MEDS and declare the problem solved? Standardization fixes the pipeline but does not pick the right one. Cohen et al. show that three defensible fixed choices give AUROCs spanning 0.829 to 0.869 within one hospital. Our matched-pipeline cells are exactly the standardized setting, so the design measures the residual site gap under standardization directly. If mismatch effects turn out negligible, this alternative wins, and we will say so.
Why not just run random-effects meta-analysis across sites? This is the closest threat, because our estimator is a meta-regression. The difference lies in the data. Observational meta-regression on published studies confounds pipeline with site, since each hospital comes with its own conventions. Here every pipeline runs at every site, so the moderator effects are interventions rather than correlations. A plain random-effects fit is our statistical baseline, and the comparison of $\tau^2$ with and without mismatch terms settles whether the extra design is worth it.
Why not just train on multi-center eICU and rely on diversity? YAIB reports that eICU-trained models transfer better but still lose performance on European sites. Pooling also does not change the fact that one pipeline was applied. We include eICU as a training site, so the analysis will show whether its advantage persists or shrinks once pipelines are mismatched.
Why not use a Gaussian-process surrogate multiverse as in Bell et al.? With gradient-boosted trees, the fractional grid is affordable to run exhaustively, and exhaustive cells avoid surrogate error. We will still fit a surrogate to the run cells as a secondary check.
Decisive Checks
The headline result is the share of the mean transfer gap attributable to mismatch, with a bootstrap interval over sites and seeds. We pre-register a threshold. If mismatch explains under 10% of the sepsis transfer gap and under 5% for mortality, the core claim fails and harmonization is sufficient.
A second check uses prediction. The fitted model, trained on all cells except those involving one held-out test site, must predict that site's mismatch AUROCs better than the site-only random-effects model.
A third check examines sign. If any $\delta$ component is positive, meaning mismatch helps, that supports the concern of Wang et al. that rigid harmonization can remove useful site-specific signal. We will report it as a result, not as noise.
Finally, the two evaluation modes, deployer's labels and reference labels, should differ substantially for onset-definition mismatch and only slightly for imputation mismatch. That pattern would confirm that the decomposition separates moved targets from shifted inputs.
References
- Subtle variation in sepsis-III definitions markedly influences predictive performance within and across methods
- Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML
- Reproducibility in critical care: a mortality prediction case study
- ACES: Automatic Cohort Extraction System for Event-Stream Datasets
- MEDS Decentralized, Extensible Validation (MEDS-DEV) Benchmark: Establishing Reproducibility and Comparability in ML for Health
- Modeling the Machine Learning Multiverse
- Embracing the Disharmony in Heterogeneous Medical Data
- MIMIC-Extract: a data extraction, preprocessing, and representation pipeline for MIMIC-III