A clinical risk model that looks excellent at the hospital where it was built often floods clinicians with false alarms when it moves somewhere else. The cause is simple arithmetic. The share of flagged patients who are truly sick, called positive predictive value (PPV), depends on how common the disease is in the patients being screened. When a model moves to a site where the disease is rarer, or rarer within one demographic group, the same decision cutoff produces mostly false alarms. Those alarms also fall unevenly across groups.
We propose a deployment-time procedure that lets a hospital make a concrete promise, such as "at least 1 in 4 alerts is a true case, for every patient group." It keeps that promise during the first weeks of a silent trial, a period when the model runs on live data and its outputs are logged but not shown to clinicians, and when only a handful of outcomes have been observed. The procedure combines two cheap and imperfect signals. One is the disease rate implied by the model's own outputs on unlabeled local patients. The other is the small trickle of confirmed outcomes. A hierarchical estimator shares information across patient groups, and the gap between the two signals serves as a built-in alarm for when the "only the disease rate changed" assumption has broken.
Core Claim
The existing pieces do not cover this case. Unsupervised label-shift correction such as LaSCal adjusts calibration without target labels, but it assumes that only the disease rate changes, $P(Y)$, while the appearance of sick and healthy patients, $P(X\mid Y)$, stays fixed. It also needs roughly 4,000 target samples. Cross-hospital transfer violates that assumption through scanner, protocol and documentation differences (Ling Zhang et al., Md Kowsher et al.). Small subgroups make the sample requirement worse.
The obvious alternative is to recalibrate on target labels, but at deployment time minority subgroups have very few positive outcomes. Pfohl et al. showed that group-specific calibration plus deliberate threshold choice preserves net benefit better than fairness penalties. They did this with plentiful retrospective labels, and they gave no way to choose those thresholds when prevalence shifts and labels are scarce.
Our claim is that a shrinkage estimator achieves something neither signal achieves alone. It anchors each group's calibration on the unlabeled label-shift estimate and corrects that anchor with few labels. Concretely, it should meet a per-group PPV target with an order of magnitude fewer labeled outcomes than label-only recalibration. It should also avoid the silent failures of label-free correction under covariate shift.
Mechanism
Let $s(x)$ be the source model's log-odds score. For target group $g$ we model target calibration as $\mathrm{logit}\,P_t(Y=1\mid x, g) = a_g + b\,s(x)$. Under pure label shift with a calibrated source model, $b=1$ and $a_g$ equals the log-odds change in group $g$'s prevalence. That change can be estimated from unlabeled target scores, giving an anchor $\hat a_g^{\mathrm{LS}}$. We place a prior on the intercepts:
$$a_g \sim \mathcal{N}\big(\hat a_g^{\mathrm{LS}} + \mu,\ \tau^2\big),$$
Here $\mu$ is a shared departure from label shift and $\tau$ is between-group spread. Both are learned from whatever delayed labels have arrived. Small groups therefore borrow from the unlabeled anchor and from other groups.
A large posterior $\mu$ or $\tau$ is a direct, testable signal that the shift is not pure label shift. The procedure then moves weight toward labels and reports this on the monitoring dashboard. For each group we choose the threshold $t_g$ as the lowest cutoff whose posterior lower bound on PPV meets the target $\rho$. This is the most sensitive rule consistent with the promise:
$$t_g = \min\{t : \Pr\big(\mathrm{PPV}_g(t) \ge \rho \mid \text{data}\big) \ge 1-\alpha\}.$$
Two design choices follow from the evidence in the file. Fairness enters through equal guarantees on a clinically meaningful quantity, not through equal error rates. Pfohl et al. showed that forcing equal error rates breaks calibration, and that when prevalence differs, some error-rate difference is unavoidable. The promised quantity is PPV, or equivalently the alerts-per-true-case burden, because clinicians want actionable risk stratification (Johnson et al.) and acute-care staff reject tools that interrupt without payoff (ICU physician survey). As a secondary operating mode, we also support net benefit at a fixed clinical risk threshold, which avoids the cost-ratio elicitation that Aligning Evaluation requires.
Research Program
We build the procedure as a post-hoc layer over any frozen model. It is evaluated in a simulated silent trial on the harmonized ICU databases of Yet Another ICU Benchmark: MIMIC-III, MIMIC-IV, eICU, HiRID and AUMCdb. The tasks are mortality and sepsis. Sepsis has roughly 1% hourly prevalence, so it is the stress case. AUMCdb has a higher baseline mortality, which supplies natural prevalence variation across sites.
For each ordered source–target pair, we train logistic regression, LightGBM and a recurrent model at the source. We then stream target stays in chronological order. Outcomes are revealed only after a realistic delay, at discharge for mortality, so at each point in time the method has a growing unlabeled pool and a smaller labeled one.
Subgroups are race, sex and age where the database records them. For eICU and MIMIC we use race and sex, and we audit with the fairness pipeline of the MIMIC-IV processing pipeline. HiRID and AUMCdb lack race, so there we use age and sex.
To separate the mechanism from site idiosyncrasies, we also build semi-synthetic targets inside a single site. In these, we subsample positives per group to create pure label shift, where the anchor should be nearly sufficient. We then add controlled covariate corruption by perturbing lab units and introducing missingness patterns, where the anchor should be detected as wrong.
Baselines
The naive reference keeps the source-site threshold that achieved the target PPV at development. This is what static deployment engines like DEPLOYR do today. DEPLOYR documented retrospective-to-prospective decay (WBC model AUROC 0.76 to 0.69) but did not correct thresholds. Beating it means fewer PPV-promise violations and no loss of sensitivity.
The label-free competitor applies the Bayes log-odds prior correction. Prevalence is estimated from unlabeled target scores, globally and per group, and LaSCal temperature scaling is applied on top. LaSCal is the strongest published label-free calibrator in the file, reducing calibration error on shifted CIFAR-100-LT from 70.47 (uncalibrated) and 28.54 (standard calibration) to 5.87 (LaSCal). We found no reported result on Lacuna for LaSCal on any ICU database or for per-group PPV targets. All methods use the same unlabeled pool.
The label-only competitor fits group-specific Platt recalibration, followed by the threshold calibration of Sahoo et al., using exactly the labels our method has seen at each time step. It gets the same tuning budget. As an upper reference, we run Pfohl-style group-specific recalibration with all target labels. For context on discrimination, YAIB reports mortality AUROC of 0.84 to 0.87 across these databases. We found no reported result on Lacuna for per-group PPV attainment on YAIB.
Beating the competitors means one of two things. Either we meet the PPV target in every group with at least five times fewer labels than label-only recalibration, or we keep violations below $\alpha$ on cross-site pairs where the label-free correction exceeds them. Ablations remove the unlabeled anchor, the cross-group pooling, and the shift diagnostic one at a time. If removing the anchor costs nothing, the core ingredient is not earning its place.
Why not just…?
Why not just recalibrate with temperature or Platt scaling on a local holdout? With enough labels this works, and it is our upper reference. The problem is timing and group size. In the first weeks of deployment a minority group may have a dozen positive outcomes. Aligning Evaluation shows that globally calibrated APACHE IV on eICU was miscalibrated for African-American patients despite a higher AUROC (0.907 versus 0.868), so pooling all groups hides exactly the failure we target. The label-budget curves will show whether label-only recalibration catches up quickly. If it matches us at under 100 labels per group, the contribution shrinks to a diagnostic, and we will say so.
Why not just apply Bayes' rule with an estimated prevalence? This is the real threat on pure label shift, and our semi-synthetic arm is designed so that it can win there. Its weakness is silent failure: when scanners, lab practices or documentation change, the prevalence implied by the model's scores is itself biased, as LaSCal's own stated limitation acknowledges. The decisive comparison is on the cross-site pairs. The test is whether label-free correction violates the PPV promise and whether our discrepancy term flags those pairs before violations accumulate.
Why not just add fairness penalties during training? Pfohl et al. showed on 5.8 million patients that enforcing equalized odds broke calibration and collapsed net benefit at the 7.5% statin threshold. Their result also implies that unequal error rates under unequal prevalence are expected rather than a defect. We include an equalized-odds-regularized source model only as a sanity check.
Why not just train on multi-center data? YAIB finds that eICU-trained models transfer better, so we report every method on top of eICU-trained source models. Better transfer of discrimination does not fix the threshold, though. PPV still moves with local prevalence, as the 17% external PPV of ECG-WMA-Net illustrates. RETFound's demographic anchoring suggests that scale does not remove group skew either.
Decisive Checks
The primary outcome is the fraction of (target site, group, week) cells in which realized PPV falls below the promised $\rho$. We report it at $\rho$ values of 0.2 and 0.33 for mortality and 0.1 for sepsis, alongside sensitivity at the chosen thresholds, alerts per true case, net benefit, and the Bounded DCA log score.
The proposal succeeds if three things hold together. Violation rates must stay near $\alpha$ in every group across all twenty site pairs. Sensitivity must stay within a few points of the full-label oracle. And the label budget needed to reach that point must be several-fold smaller than for label-only recalibration.
It also carries a falsifiable diagnostic claim. The learned departure $\mu$ should be near zero on the semi-synthetic label-shift targets and clearly nonzero on the pairs where label-free correction fails. If $\mu$ does not separate these cases, the anchor cannot be trusted, and the method reduces to label-only recalibration with extra steps.
References
- LaSCal: Label-Shift Calibration without target labels
- Robust Pancreatic Ductal Adenocarcinoma Segmentation with Multi-institutional Multi-phase Partially-Annotated CT Scans
- Predicting the Appropriate Mode of Childbirth using Machine Learning Algorithm
- Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare
- Perspective Toward Machine Learning Implementation in Pediatric Medicine: Mixed Methods Study
- Intensive Care Unit Physicians’ Perspectives on Artificial Intelligence–Based Clinical Decision Support Tools: Preimplementation Survey Study
- Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs
- Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML
- An Extensive Data Processing Pipeline for MIMIC-IV
- DEPLOYR: a technical framework for deploying custom real-time machine learning models into the electronic medical record
- Reliable Decisions with Threshold Calibration
- Identification of cardiac wall motion abnormalities in diverse populations by deep learning of the electrocardiogram
- A foundation model for generalizable disease detection from retinal images