Borrowing Strength Across Hospitals to Validate Rare-Outcome Models Locally

A hospital that adopts a clinical risk model built elsewhere has to answer two questions before trusting it: are the predicted risks right for our patients, and does the model still separate sick from well here? Answering either question requires local cases of the outcome. For rare outcomes, such as hemorrhage after stroke or death in a small ICU subgroup, a typical hospital may see only a handful of these cases a year. The usual remedies split into two camps, and both run into this data shortage. One camp validates a model once on an external cohort and then freezes it. The other argues that "all models are local" and re-validates at every site. This proposal sidesteps the shortage by treating the many hospitals that have already deployed the model as a source of prior knowledge about how sites differ. A new hospital then has to learn only where it sits within that known spread of variation, and it can do this from a few labeled events plus the model's scores on all of its unlabeled patients.

Concretely, we fit a hierarchical model, meaning a model in which each hospital has its own parameters but those parameters are themselves drawn from a network-wide distribution. It describes how the frozen model's risk scores are distributed among positives and negatives at each hospital, and how common the outcome is there. At a new site, this network prior is combined with two kinds of local evidence: the few labeled events, and the full unlabeled score distribution. The result is a posterior for the local calibration curve, the local sensitivity and specificity, and the local positive predictive value, each with honest uncertainty. The claim is that this reaches, with roughly 20 local events, the calibration and performance-estimation accuracy that standard local recalibration needs hundreds of events to reach. It should also flag sites where the model's discrimination has collapsed.

Why this gap, and what the literature already settles

The evidence that a gap exists comes from multi-center ICU and stroke work. Johnson et al. trained logistic regression on pooled eICU hospitals and tested it on held-out ones. Area under the ROC curve (AUROC) ranged from 0.688 to 0.933 across 46 hospitals, and standardized mortality ratios ranged from 0.560 to 1.43. Recalibrating on local data fixed the mortality ratios but not AUROC. Local models trained from scratch started near AUROC 0.61 and needed about 2,400 local patients to approach the transferred model. Wang et al. validated a hemorrhagic-transformation model across 10 OHDSI databases, with a mean external AUC of 0.71. The model fell to AUC 0.50 on a German outpatient database, and fixed thresholds were applied despite differing incidence. Seneviratne et al. use the Epic Sepsis Model's fall from a marketed 0.73–0.83 to 0.60–0.64 to argue for recurrent local validation. They do not say how a site with few events would carry it out. Taken together, these papers establish three facts. Sites differ mostly in calibration, and sometimes badly in discrimination. Pooled models are the right starting point. Local checking is necessary but starved of data. That last fact is the tension we target.

Two lines of work constrain the design. LaSCal recalibrates without target labels by estimating the new class balance from unlabeled data. It assumes that only prevalence changes, with features given the class held fixed, and it needs on the order of 4,000 unlabeled target samples. Multi-center data violate that assumption: WILDS shows Camelyon17 accuracy dropping from about 93% to about 70% from scanner and staining shift alone. Our model therefore lets the score distributions of positives and negatives move per site, and it learns from the network how far they typically move. Varoquaux & Colliot and Kweon et al. supply the evaluation lens. Sensitivity and specificity do not depend on prevalence, whereas positive predictive value and accuracy do. AUROC can hide subgroup miscalibration, as in Kweon et al.'s APACHE IV finding of AUROC 0.907 versus 0.868 alongside worse calibration for the higher-AUROC subgroup. Kweon et al.'s Bounded DCA Log Score, a log-loss restricted to clinically realistic prevalence ranges and error costs, is the decision-level metric we report alongside calibration error.

Core Claim

The model treats the frozen classifier's score $s$ (on the logit scale) at hospital $j$ as a two-component mixture:

$$ s \mid y=k, j \sim \mathcal{N}(\mu_{jk}, \sigma_{jk}^2), \qquad y \mid j \sim \text{Bernoulli}(\pi_j), $$

where $k\in\{0,1\}$ is the outcome, $\pi_j$ is the site's prevalence, and $(\mu_{jk}, \sigma_{jk})$ describe how scores of non-events and events are spread at that site. The site vector $\theta_j = (\text{logit}\,\pi_j, \mu_{j0}, \mu_{j1}, \log\sigma_{j0}, \log\sigma_{j1})$ is drawn from a network prior $\mathcal{N}(m, \Sigma)$ that is estimated from the development hospitals. Given $\theta_j$, the local calibrated risk is $\pi_j\phi_{j1}(s) / (\pi_j\phi_{j1}(s) + (1-\pi_j)\phi_{j0}(s))$, where $\phi_{jk}$ is the density of component $k$. Local AUROC, sensitivity and specificity at a threshold, and positive predictive value all follow in closed form from the same parameters.

Labeled events pin down the event component directly. The unlabeled scores constrain prevalence and the mixture shape. The prior covariance $\Sigma$ encodes regularities across hospitals, for example whether high-prevalence sites also tend to have shifted event scores. This lets a few events go further. Pure label-shift correction is the special case in which $\mu_{jk}$ and $\sigma_{jk}$ are pinned to the source values. Plain local recalibration corresponds to dropping both the prior and the unlabeled term. The equal-variance normal score model is a convenience: we test it against a monotone spline mixture and keep whichever has better held-out fit.

Research Program

The work has three parts. The first fits the network prior on eICU-CRD. We start from Johnson et al.'s setup of 82 first-day features and L2 logistic regression, and also use a gradient-boosted model in the spirit of E-CatBoost, so that we can check that the method does not depend on the classifier. We estimate $m$ and $\Sigma$ by empirical Bayes over the development hospitals, using each hospital's labeled scores.

The second part builds the local estimator. At a held-out hospital it runs full Bayesian inference, using Markov chain Monte Carlo or a Laplace approximation, on $k$ labeled events, a matched sample of labeled non-events, and all remaining unlabeled admissions. It returns posterior intervals for calibration intercept and slope, the standardized mortality ratio, AUROC, and sensitivity, specificity, and positive predictive value at the deployed threshold.

The third part adds a failure detector. It reports the posterior probability that local AUROC falls below a clinically set floor, and it flags sites whose unlabeled score distribution sits far outside the network prior. That second check targets cases like Wang et al.'s outpatient German site, where the problem is less a prevalence shift than a different setting.

To study rarity directly, the outcome must be rare at held-out sites. We therefore run eICU mortality both at its natural rate and within diagnosis subgroups where it is rarer. We also run controlled experiments that downsample events at the held-out site to 1–2% prevalence. These change only the class balance, so we pair them with Camelyon17-WILDS. There, the covariate shift is real and severe, and we inject per-hospital tumor-patch prevalence by subsampling. Because Camelyon17 has only five hospitals, its prior is weak by design. It serves as a stress test of whether the method degrades gracefully, and of whether the out-of-prior flag fires, when the network is small and shifts are large.

Baselines

The naive reference point is to deploy the pooled model unchanged, with no local recalibration. Its local calibration error and estimated mortality ratio set the ceiling of harm. Johnson et al. report mortality ratios ranging from 0.560 to 1.43 under this regime. Comparing against it tells us how much any local procedure buys.

The strongest existing competitor is Johnson et al.'s transferred-then-recalibrated model: pooled training followed by logistic recalibration of intercept and slope on whatever local labeled data exist. The file reports that this fixes the mortality ratio but leaves AUROC unchanged. It reports no number for calibration error as a function of local event count, so for that curve there is no reported result found on Lacuna. We will therefore produce the curve ourselves. Both methods get identical local labeled subsamples, identical seeds, and 200 resamples per event count $k\in\{5,10,20,50,100,300\}$. Recalibration uses the standard maximum-likelihood fit, with and without a ridge penalty chosen by leave-one-out within the local sample. That gives it a tuning budget no smaller than ours. To beat it, our estimator must match its calibration error at 100 events using 20 events or fewer, and its 90% intervals must cover the full-site truth at 85–95% across held-out hospitals.

The unsupervised competitor is LaSCal, run on the same unlabeled pool with zero labels. On Camelyon17 the file has no reported per-dataset number; it reports only CIFAR-100-LT calibration error falling from 70.47 to 5.87. Our zero-label variant must beat it on eICU and Camelyon17 when unlabeled pools fall below a few thousand.

For local performance estimation, as opposed to recalibration, the baseline is the empirical local AUROC with a DeLong interval computed from the same $k$ events, as described by Varoquaux & Colliot.

Three ablations isolate the key ingredients. The first removes the network prior, using a flat prior with the same likelihood. The second removes the unlabeled term, keeping the prior and labels. The third pins the class-conditional score parameters to source values, which reduces the model to label shift only. If removing the prior costs nothing, the core idea fails.

Why not just…?

Why not just recalibrate locally with Platt scaling? This is the real threat. With a few hundred events it will be as good as anything else, and we expect our advantage to vanish in that regime. Below about 50 events, a two-parameter logistic fit has wide error bars, and its slope can come out with the wrong sign. Johnson et al.'s learning curves for local models show how badly small local samples behave. The event-count curve in our experiments settles the question. If plain recalibration matches us at 20 events, the proposal is not worth adopting, and we will say so.

Why not just estimate local prevalence from unlabeled data, as LaSCal does? This requires that event and non-event patients look the same across sites. Camelyon17's 23-point accuracy drop from scanner and staining differences shows that they do not. It also needs thousands of unlabeled samples. Our label-shift-only ablation is exactly this assumption placed inside our model, so the comparison tells us whether letting class-conditional scores move per site matters in practice.

Why not just train or fine-tune a local model? Johnson et al. show that local models start near AUROC 0.61 and need about 2,400 patients, and the number of events required is far larger again for outcomes below 1% prevalence. We include a local refit only as a reference at the largest $k$.

Why not just report the network-average performance with a random-effects meta-analysis and skip local data? This is the closest classical idea, and our model with zero labeled and zero unlabeled local data is exactly that prediction. It ignores whether this hospital is unusual, which is precisely what failed at Wang et al.'s outpatient site. Showing that the posterior moves meaningfully away from the network average at sites like that is the experiment that separates the two approaches.

Decisive Checks

We use leave-one-hospital-out evaluation on eICU-CRD across all hospitals with enough admissions to define a full-site ground truth. Each held-out hospital's complete labeled data serves as the truth for calibration error, mortality ratio, AUROC, and Bounded DCA Log Score over a 0.1–5% prevalence range. The primary outcome is the number of events each method needs to reach a fixed calibration error. Secondary outcomes are interval coverage and the area under the curve of the low-AUROC-site detector, with sites below 0.75 AUROC treated as failures; Johnson et al.'s 0.688 shows that such sites exist.

The decisive negative result would be that removing the network prior leaves performance unchanged. That would mean hospitals are too heterogeneous for borrowing strength to help, and the honest conclusion would be that few-event local validation is not feasible. The Camelyon17 prevalence-injected runs check whether the out-of-prior flag fires rather than producing confident wrong answers when only five hospitals are available.