Site-Held-Out Calibration for Estimating Segmentation Quality on New Scanners

A hospital that installs a segmentation model needs to know how well it works on its own scanner before anyone has drawn reference contours there. Several methods now estimate a model's Dice score (the overlap between the predicted and true mask) without labels. They compute a label-free signal, such as how much ensemble members disagree or how well a foundation model can reuse the prediction, and map that signal to a predicted score. The weak point is that mapping. It is always fitted on labeled data that looks like the training data, and then it is applied to a new scanner whose image physics differ. This note proposes fitting and testing that mapping across sites instead of within them. Each labeled source site is held out in turn, so the calibration learns how far the signal drifts when the scanner changes. Leave-one-site-out residuals are then wrapped in conformal prediction, a method that turns past errors into intervals with a stated hit rate. The goal is per-organ intervals for Dice and for HD95 (the 95th-percentile boundary distance) that still contain the true value on a scanner the system has never seen.

The claim is deliberately narrow. On real cross-scanner shift, a meaningful share of the error in existing estimators should come from a calibration map fitted on i.i.d. data, not from the quality signal itself. If so, swapping the calibration protocol should improve interval coverage across several different signals. The mapping should also be fitted per organ, because organs fail very differently under shift. If this holds, it changes how medical performance estimators should be built and validated. If it fails, we learn that the signal itself must be replaced.

Core Claim and Positioning

Existing work splits into two camps that do not yet meet.

The first camp is reverse evaluation. Classic Reverse Classification Accuracy trains an auxiliary segmenter on the test image and its predicted mask, then scores that segmenter on labeled reference cases. It reports Pearson r > 0.95 on 35 whole-body MRI subjects, but needs about a minute per case and struggles with Hausdorff distance. The UniverSeg-based SPE replaces training with in-context segmentation and reports r = 0.956 ± 0.046 and MAE = 0.025 ± 0.019 on Dice. That quality spread comes from training checkpoints, not scanner shift, and the authors state the method fails on HD95. ConfIC-RCA adds retrieval and split-conformal intervals and brings latency down to 0.37–0.70 s per image. The authors themselves note that the guarantees assume calibration and test data are exchangeable, which domain shift breaks.

The second camp is confidence and disagreement signals. The failure-detection benchmark finds that deep ensembles scored by pairwise Dice rank failing scans best on M&Ms, BraTS and prostate data, while Mahalanobis feature distance does poorly. That benchmark ranks scans by risk; it does not regress absolute values or produce intervals. The conformal accuracy estimator for medical devices handles classification only. It reaches 1.1% ± 0.7% error on Fitzpatrick17K but 8.8% ± 2.8% on the five-hospital Camelyon17. That gap is consistent with our premise that cross-hospital shift is where calibration breaks.

Our proposal reuses both camps' signals unchanged; they supply the method. The difference lies in three places:

  • Calibration protocol. The unit of exchangeability is the acquisition site, not the image.
  • Target. Per-organ Dice and HD95 in 3D multi-organ volumes, where AMOS documents MRI drops above 20% on unseen scanners and steep losses on small organs.
  • Output. Intervals with tested coverage on new sites, not correlations on pooled data.

The ATC paper proves unlabeled estimation is impossible without assumptions. It constrains the design: we make our assumption explicit, namely that the new site is drawn like the source sites, and test where it breaks.

Research Program

Fix a frozen segmenter $f$, for example a 3D U-Net trained on some AMOS scanners. For each image $x$ and organ $k$, compute a label-free signal vector $s_k(x)$. It contains three components:

  • pairwise Dice across a five-member ensemble;
  • mean voxel entropy inside and at the boundary of the predicted mask;
  • the in-context reverse-evaluation score from UniverSeg.

We also include simple mask descriptors such as predicted volume and the number of connected components. Connected components matter because stray islands of false positives drive HD95.

A quantile regressor $g_k$ maps $s_k(x)$ to the lower and upper quantiles of the true metric $m_k(x)$. For HD95 we model log-distance, so the large outliers that break linear maps become manageable. Organs share one model with an organ embedding, so that small organs borrow strength from large ones.

The key step is how $g_k$ is fitted and calibrated. Suppose the source data come from sites $1,\dots,J$. For each held-out site $j$, we fit $g_k$ on the other sites and record conformal residuals on site $j$. Pooling these leave-one-site-out residuals gives an interval width of the form

$$\hat q_k = \text{Quantile}_{1-\alpha}\{\, r_{k}(x_i) : x_i \text{ scored by a model that never saw its site} \,\},$$

where $r_k(x_i)$ is the conformalized quantile-regression residual on image $x_i$ and $\alpha$ is the allowed miss rate. The resulting interval for a new image is $[g_k^{lo}(x) - \hat q_k,\ g_k^{hi}(x) + \hat q_k]$.

If sites are exchangeable with one another, which is a two-level sampling assumption, these residuals reflect cross-site drift by construction. Per-image intervals then aim at $1-\alpha$ coverage averaged over new sites. Coverage within a single unlucky site is not guaranteed; we will measure it rather than claim it.

For cohort-level reporting, such as "this scanner's pancreas Dice is about 0.62", we average per-image estimates. We attach an interval obtained by conformalizing site-mean residuals. With only 4–8 sites this interval will be coarse, and we report that honestly.

Baselines

The do-nothing reference is source-validation performance reported as the target estimate. It answers whether any estimator beats assuming nothing changed, and on AMOS MRI it should be badly wrong given the reported drop above 20%.

The first confidence baseline is temperature-scaled average voxel confidence. The second is ATC adapted per organ: a confidence threshold is chosen on source validation data so that the fraction of voxels above it matches Dice, then applied to target voxels. Both are cheap and are the default a practitioner would try.

The strongest existing estimators are UniverSeg SPE with its linear map and ConfIC-RCA with i.i.d. split-conformal intervals. Both are run exactly as published, but on our 3D data processed slice-wise, with the same reference pool and the same frozen segmenter. For SPE the reported figure is MAE 0.025 ± 0.019 on Dice across six datasets (SPE). That figure comes from a checkpoint quality spectrum rather than scanner shift, so it is an optimistic ceiling. No reported result was found on Lacuna for ConfIC-RCA coverage under cross-site shift. No reported AURC number was found on Lacuna for ensemble pairwise Dice on M&Ms.

The key ablation keeps every signal and regressor identical and changes only the calibration protocol: i.i.d. split versus leave-one-site-out. A second ablation removes per-organ conditioning. All methods share the same segmenter, reference pool, ensemble members and hyperparameter search budget, and every tuning choice is made on source sites only.

We beat the strongest baseline if three things hold on held-out scanners:

  • coverage at nominal 90% stays within a few points of 90%, while ConfIC-RCA's i.i.d. intervals fall clearly short;
  • the median interval width for Dice stays usable, at around 0.15 or less;
  • HD95 intervals cover at nominal rates, where linear SPE-style maps for HD95 do not.

Why not just…?

Why not just use ensemble pairwise Dice, which already wins the failure-detection benchmark? We do use it, as a signal. The benchmark shows it ranks scans well. Ranking does not say what Dice value to expect, and its own quality regressors were volatile across datasets. Prior analysis of disagreement further shows that under strong shift, members with shared inductive biases agree on wrong answers, so the disagreement-to-error relation itself moves. The calibration ablation tests directly whether fixing the mapping is enough or whether the signal also needs replacing.

Why not just reweight calibration images by a domain classifier, as covariate-shift methods do? PAPE does this for tabular data. It is a real competitor, and we include a density-ratio-weighted conformal variant. Our expectation is that cross-scanner MRI shifts have little support overlap, so the weights become extreme and the intervals either explode or under-cover. If the weighted variant matches leave-one-site-out coverage at similar width, we will report that the simpler single-site approach suffices.

Why not just label five scans at the new site? This is the most serious threat, and we test it head-on. Five labeled scans give an unbiased but noisy per-organ mean, with no per-image information. We compare its cohort-level error against our unlabeled estimate as the number of labeled scans grows. We also report the crossover point at which a small labeled audit wins; the proposal's value lies below that point and in per-image triage.

Proposed Experiments and Decisive Checks

The main testbed is AMOS: 500 CT and 100 MRI scans from eight scanners at two centers. Each scanner is held out as a target in turn, with all 15 organs reported separately so that liver and adrenal glands are never averaged together. The second testbed is M&Ms cardiac MRI from four vendors, with each vendor held out in turn. A third is the MS-CMRSeg bSSFP-to-LGE sequence shift, where post-hoc calibration still leaves 15.37% expected calibration error. That shift lets us test whether our intervals hold when voxel confidence is known to be miscalibrated.

We report the following metrics:

  • absolute error of point estimates;
  • interval coverage and width per organ and per held-out site;
  • worst-site coverage;
  • cohort-mean error;
  • AURC for triage, for comparability with the benchmark.

Three outcomes would settle the question. First, if i.i.d. and site-held-out calibration give similar coverage on held-out scanners, the core claim is false and calibration is not the bottleneck. Second, if leave-one-site-out restores coverage only for large organs, the method needs organ-specific signals and the claim narrows accordingly. Third, if HD95 coverage holds while the HD95 point estimates remain poor, we have shown something new: interval estimation sidesteps the non-linearity that defeated RCA and SPE.