Site-Calibrated Generalized Posteriors for Safe Referral Under Hospital Shift

Medical image classifiers fail most dangerously when they are wrong and sure of it. A model trained on scans from a few hospitals meets a new hospital, its accuracy drops, and its confidence does not. Even Bayesian neural networks, which are built to report uncertainty, give low-uncertainty wrong answers on shifted data. As a result, an automatic "refer the uncertain cases to a specialist" rule sends the easy cases to humans and lets the confident mistakes through (Avoiding Catastrophic Referral Failures In Medical Images Under Domain Shift). We argue that part of the problem is how wide the posterior is allowed to be. A standard Bayesian posterior is as narrow as it would be if every training image were an independent draw from one population. In reality images arrive in clusters, one cluster per hospital, and the quantity that matters at deployment is how much the fitted model would move if a different hospital had been in the training set.

The proposal is to set the spread of a generalized Bayesian posterior (a "Gibbs posterior," which updates beliefs through a loss function rather than a trusted likelihood) using hospital-level resampling instead of patient-level resampling. Concretely, the posterior over a network's last layer gets a covariance that is inflated only along the directions in which the training hospitals disagree. Test images whose features point along those directions then receive wide predictive uncertainty and are referred. Everything is estimated from source hospitals alone, with no target-hospital validation data. Because the inflation depends on the direction of each input's features rather than rescaling all logits uniformly, it changes which cases get referred. A temperature or Platt rescaling cannot do this.

Core Claim

Generalized Bayesian inference has a known and unsolved calibration problem. A loss-based posterior has no natural scale, so its learning rate $\eta$ must be chosen somehow (A General Framework for Updating Belief Distributions). General Bayesian updating and the loss-likelihood bootstrap chooses it by matching the posterior's information to that of a bootstrap that resamples individual observations. That choice is correct when observations are exchangeable. It is wrong when the deployment risk is a new cluster, which is exactly the situation in medical imaging. Our claim is that matching the posterior to a cluster-level (hospital- or slide-level) resampling distribution, in matrix rather than scalar form, yields uncertainty that anticipates unseen-hospital errors. It should reduce confident out-of-domain errors at a fixed referral rate without any target data.

The mechanism fits in one equation. Let $\theta$ be the last-layer weights, $\hat\theta$ their fitted value, $H$ the Hessian of the summed training loss at $\hat\theta$, $g_i$ the loss gradient of image $i$, and $G_s=\sum_{i\in s} g_i$ the summed gradient of hospital $s$. The posterior is Gaussian around $\hat\theta$ with covariance

$$\Sigma_\kappa = H^{-1}\Big(\sum_i g_i g_i^\top + \kappa \sum_s G_s G_s^\top\Big)H^{-1},$$

where $\kappa\ge 0$ weights the between-hospital term. Setting $\kappa=0$ recovers a patient-level sandwich covariance, which is the matrix analogue of Lyddon et al.'s calibration. Replacing the bracket by $H$ recovers the ordinary Laplace posterior. The predictive variance of an input with last-layer features $\phi(x)$ is $\phi(x)^\top\Sigma_\kappa\phi(x)$. The between-hospital term has rank at most one less than the number of hospitals, so it widens the posterior only along a few directions. That low rank is the point. It encodes "the way hospitals have been observed to differ" rather than an isotropic dose of doubt. We pick $\kappa$ by leave-one-hospital-out predictive log-likelihood on source hospitals only.

Relation to Prior Work

Three papers motivate the proposal by documenting the gap.

- Avoiding Catastrophic Referral Failures shows that Bayesian models produce low-entropy false positives under shift. It repairs referral with class-split logit shifts and Platt scaling tuned on an out-of-domain validation set. We change the posterior itself and never touch target data.

Other papers supply the method or constrain it.

  • The Gibbs posterior of Bissiri et al. and the calibration-by-bootstrap idea of Lyddon et al. supply the method. We change the resampling unit from patient to site, and the calibration target from a scalar to a matrix.
  • Adversarial Interpretation of Bayesian Inference casts the posterior as the solution of a game in which an adversary perturbs the loss. Read that way, our covariance hedges against an adversary restricted to the span of observed hospital shifts. This gives an interpretation, not a proof of robustness.
  • The robust-divergence work of Jewson et al. 2018 and Jewson et al. 2023 bounds the influence of tail observations in the training data. It also warns that robust losses lose efficiency when the model is correct. This constrains the design. Down-weighting outlying training images is a different remedy from widening uncertainty for shifted test images, so we treat the β-divergence loss as a separate factor rather than as the core mechanism.
  • WILDS finds that IRM and CORAL rarely beat plain training on real shifts. This cautions against assuming that environment labels help at all.

Research Program

The first stage builds the estimator. We train a standard network by ordinary cross-entropy on the source hospitals and freeze the feature extractor. We then compute $H$, the per-image gradients and the per-hospital summed gradients for the last layer, form $\Sigma_\kappa$, and select $\kappa$ by leave-one-hospital-out. The whole computation is a few passes over the training set, much cheaper than an ensemble.

The second stage uses finer grouping. Camelyon17 images nest inside slides and patients inside hospitals, so we replace or augment hospital clusters with slide clusters. This gives many more clusters, and therefore a higher-rank between-cluster term, and tests whether the benefit comes from cluster structure generally or from hospital identity specifically.

The third stage crosses the covariance choice with the loss choice. We compare cross-entropy against the β-divergence loss of Jewson et al. to see whether the two remedies add up or overlap.

The fourth stage is a stretch goal: extending the construction to the per-voxel output layer of a 3D segmentation network on AMOS, treating each scanner as a cluster.

Baselines

Do nothing. The naive reference is plain cross-entropy training (ERM) with softmax confidence used for referral. On Camelyon17-WILDS, WILDS reports that accuracy drops from about 93% in-distribution to about 70% out-of-distribution for this model. That drop defines the size of the problem. The same backbone and features are shared by every last-layer method below, so differences come only from the posterior.

Standard Bayesian references. These are the ordinary last-layer Laplace posterior (bracket replaced by $H$) and a deep ensemble of five independently trained networks. They are the "standard BNN" that Srivastava et al. found to fail. Beating them shows the failure comes from posterior width rather than being inherent to Bayesian uncertainty. The ensemble gets five times the training compute. We report that disparity rather than hide it.

Ablation isolating the key ingredient. This is the patient-level sandwich ($\kappa=0$), plus a scalar version with $\eta$ chosen by Lyddon et al.'s information matching. If $\kappa>0$ does not beat $\kappa=0$, the hospital-level resampling idea is refuted, whatever the overall gains.

Strongest existing competitor for accuracy. This is CLOvE from On Calibration and Out-of-Domain Generalization. It reports more than 6% absolute out-of-domain accuracy gain over ERM on Camelyon17, and about 4.5% from its post-hoc robust calibration. CLOvE is retrained with the same backbone, the same hospital labels and a matched hyperparameter-search budget.

Strongest existing competitor for referral. This is Split Platt Scaling from Avoiding Catastrophic Referral Failures, run in two versions. The original version is tuned on target-domain validation data, which we treat as an oracle upper reference. The fair version is tuned on a held-out source hospital. No reported result was found on Lacuna for its referral numbers on EyePACS→APTOS or Kermany→ChestX-ray14, so we rerun it ourselves.

What counts as success. We win if we reduce confident out-of-domain errors at matched referral rates, compared with the source-tuned Split Platt Scaling and CLOvE. We also want to close a substantial fraction of the gap to the oracle Split Platt Scaling, while matching CLOvE's accuracy when our covariance is placed on a CLOvE-trained model.

Why not just…?

Why not just temperature-scale on a held-out source hospital? For binary referral, a global temperature is a monotone transform of the logit. It leaves the ranking of cases by confidence unchanged, so it cannot alter which cases are referred, only how the probabilities read. Srivastava et al.'s own move to class-split transforms reflects this. Our input-dependent variance changes the ranking. The risk–coverage comparison will show directly whether that reranking helps, and ECE after temperature scaling is reported separately so that calibration and referral are not conflated.

Why not just use a deep ensemble, or bag over hospitals? Ensembles are the strongest simple uncertainty method and a real threat. However, independently seeded ensembles have no reason to disagree specifically along hospital directions. Bagging over three training hospitals yields very few distinct resamples. We run both. If a hospital-bagged ensemble matches the sandwich covariance, the contribution shrinks to "a cheap closed-form version of site bagging," which is still useful but smaller, and we would say so.

Why not just train with CLOvE or group-robust training? These change what the features encode. Our method changes how uncertain the predictions are given those features, so the two are complementary rather than rivals. The decisive run places $\Sigma_\kappa$ on a CLOvE-trained model. If CLOvE alone already removes the confident errors, our method adds nothing, and the experiment will say so.

Why not just use a robust divergence loss? The results of Jewson et al. 2023 concern contaminated training data. The failure here is a shifted test population, and such a loss can also cost efficiency on in-distribution data. The factorial loss × covariance experiment tests whether the β-divergence loss contributes anything beyond widening the posterior.

Proposed Experiments and Decisive Checks

Primary benchmark. Camelyon17-WILDS trains on three hospitals and tests on the unseen hospitals, with the WILDS validation hospital kept out of all tuning. We report four quantities:

  • out-of-domain accuracy;
  • ECE;
  • area under the risk–coverage curve;
  • the rate of catastrophic referrals, meaning wrong predictions that the model keeps rather than refers at 10%, 20% and 30% referral rates.

Replication benchmarks. These are the two shifts used by Srivastava et al., EyePACS→APTOS for diabetic retinopathy and Kermany→ChestX-ray14 for pneumonia. They have weaker cluster structure. We form clusters from acquisition metadata such as image resolution and device where available and report honestly if these pseudo-sites fail. That failure would itself show the method needs genuine multi-site sources.

Stretch benchmark. AMOS MRI segmentation holds out one scanner at a time. It tests whether scanner-level inflation flags the voxels behind the more than 20% mDice collapse on unseen scanners that AMOS reports.

Decisive check 1: the $\kappa=0$ ablation. Hospital-level inflation must beat patient-level inflation on unseen-hospital referral.

Decisive check 2: alignment. We measure the projection of each test hospital's mean gradient onto the span of the source hospitals' gradients. If unseen-hospital shifts are largely orthogonal to what three training hospitals reveal, the low-rank inflation cannot help. That projection should predict per-hospital gains, which gives a falsifiable account of when the method works.

Decisive check 3: slide-level clustering. It should interpolate between the patient-level and hospital-level results. If hospital-level clustering is not better than slide-level clustering, the benefit comes from cluster structure generally rather than hospital identity specifically.