Medical imaging pipelines routinely map every scan onto a standard anatomical template, or atlas, so that the same coordinates refer to the same piece of anatomy in every patient. The working assumption is that once anatomy is aligned, a model's output at a given location means the same thing whichever hospital produced the scan. That assumption is never tested. Psychometrics has a mature test for exactly this kind of claim. Differential item functioning (DIF) asks whether people with the same underlying trait level answer a questionnaire item differently depending on which group they belong to. We propose to treat each atlas region as an "item" and each hospital as a "group." We then ask whether a segmentation model, given tissue whose true boundary lies at the same place, draws that boundary differently depending on the site that acquired the scan.
The payoff is a diagnostic that answers a specific question: which anatomical regions are miscalibrated by site, and in what way. That question has more value than knowing only that external Dice drops. We test whether the diagnostic earns its keep in a single way. If it is computed only on the training hospitals, does it predict where a model will fail at a hospital it has never seen better than simpler per-site error maps do? If it does not, the psychometric framing adds nothing, and we will say so.
Core Claim
Psychometrics separates two failure types. The first is uniform DIF, a constant shift in how readily an item is endorsed, which is the item-level counterpart of failing scalar invariance (equal item intercepts across groups). The second is non-uniform DIF, a change in how sharply the item responds to the trait, which is the item-level counterpart of failing metric invariance (equal factor loadings across groups). The BESAA validation shows that this distinction matters in practice: loadings held across groups while intercepts did not, so mean comparisons were unsafe even though correlations were fine.
Segmentation offers a natural matching variable that surveys lack: the signed distance $d$ (in mm) from a voxel to the ground-truth lesion or organ boundary. For atlas region $r$, voxel-level model output $\hat y$, site indicator $s$, and case-level covariates $c$ (lesion volume, local registration Jacobian), we fit
$$ \operatorname{logit} P(\hat y = 1 \mid d, s, c) = \beta_0^{r} + \beta_1^{r} d + \gamma_s^{r} + \delta_s^{r} d + \eta^{r\top} c . $$
The term $\gamma_s^r$ is uniform DIF. Divided by $\beta_1^r$, it becomes a site-specific boundary displacement in millimetres, that is, systematic over- or under-segmentation. The term $\delta_s^r$ is non-uniform DIF, a site-specific change in boundary sharpness, which shows up as blur or overconfidence. The covariates $c$ absorb case mix. Without them, a site that sees larger tumors would look biased. This matters because Holzschuh et al. report strong center imbalance (96 of 161 patients from one site). It matters equally because tertiary centers see systematically different disease.
Our claim is that the maps of $\gamma^r$ and $\delta^r$, estimated on training sites alone, predict region-level degradation at a held-out site. We also claim they separate registration-driven failure from acquisition-driven failure, which per-site Dice cannot do.
Relation to Prior Work
Several papers motivate the gap. BraTS 2021 registers 2,040 multi-site scans to the SRI24 atlas and finds that radiogenomic prediction remains fragile, which suggests that site signal survives spatial alignment. The benchmark never asks where that signal survives. Li et al. align features to a pelvic atlas across six institutions and show that removing alignment costs at least 12% in some cross-institution tests. They use alignment as a performance device and do not check whether aligned features behave the same across sites. Pellegrini et al. document inflated single-center accuracy in neuroimaging. Feng et al. are a typical instance: a single-hospital AUC of 0.905 with no external check. Both describe the problem in terms of overall accuracy only.
FormInv supplies the methodological precedent. It ports DIF ideas into model evaluation and shows that accuracy can hide instability: 86% accuracy coexisted with a 41.9% answer-flip rate on correctly answered items. FormInv perturbs text. We instead condition on a physical ground truth and use the hospital as the grouping variable, so its machinery does not transfer directly. SDNet separates anatomy from imaging appearance by design, but it does not use atlas coordinates and never tests whether its "anatomy" factor is actually invariant. Our audit could be applied to its anatomy maps as one of the models under test. Score-based DIF tests from IRT toolboxes supply the testing logic we reuse for per-region significance with multiple-comparison control.
One constraint shapes the design. BraTS notes that atlas warping distorts tissue around space-occupying lesions. Region-level DIF near tumors could therefore reflect registration error rather than scanner effects. This is why the local registration Jacobian enters $c$, and why we report DIF with and without it.
Research Program
We first build the audit. This means per-region logistic DIF fitting with site as a random effect, which stabilizes estimates for small sites such as Dresden ($n=14$). It also means bootstrap intervals over patients and false-discovery-rate control across regions. We then run the audit on nnU-Net models trained under leave-one-center-out (LOCO) splits. We compute DIF maps from held-in validation cases only and compare them with the observed region-level change in Dice and boundary distance on the held-out center.
Second, we decompose DIF into three sources. We use registration choices (rigid versus affine versus deformable registration to the template), input resampling (native versus $1\,\text{mm}^3$ isotropic, which matters for thick-slice T2 as in Wang et al.), and site labels. This separates DIF created by normalization from DIF the scanner brings with it.
Third, we run one targeted intervention. In regions flagged for uniform DIF, we apply a site-conditioned boundary offset correction. In regions flagged for non-uniform DIF, we apply region-local temperature recalibration. We compare both against the same correction applied uniformly everywhere.
Baselines
The naive reference is no audit at all: report pooled internal Dice and assume it transfers. For calibration of scale, nnU-Net for BraTS reports a BraTS 2020 test whole-tumor Dice of 88.95. Holzschuh et al. report a LOCO median DSC of 0.74 against 0.76 for mixed training on PSMA PET. The gap between the pooled number and the held-out-site number in each LOCO fold is the quantity any diagnostic must explain.
The strongest competing diagnostic is a per-site, per-region error map. This is simply Dice or mean boundary error computed separately for each training site and atlas region, with the spread across sites used as the bias score. It is the obvious thing a practitioner would compute and needs no psychometrics. We run it on identical held-in validation cases, the same atlas, and the same regions, with matched bootstrap resampling. No reported result was found on Lacuna for predicting held-out-site regional degradation from internal diagnostics, so there is no published number to beat. The criterion is a statistically significant gain in rank correlation with held-out-site regional degradation, tested by paired bootstrap across LOCO folds.
The third baseline is a site-classifier probe: a linear classifier predicting site from region-pooled encoder features, scored by its accuracy per region. It is trained on the same features with the same cross-validation budget.
The ablation that isolates our key ingredient is DIF with the matching variable $d$ and the covariates $c$ removed. This reduces the audit to a site effect on raw predicted probability. If this ablation performs as well as the full model, conditioning on the latent ground truth is doing no work.
Why not just…?
Why not just compute per-site Dice for each atlas region? This is the real threat, and the experiment is built around it. Per-site error confounds case mix with acquisition. A site with small, low-contrast tumors will show low Dice in every region, so the error map does not separate a scanner effect from a harder caseload. It also cannot distinguish a consistent boundary shift from blur, and those call for different fixes (an offset versus recalibration). If the per-region error map predicts held-out-site degradation as well as the DIF map across LOCO folds, we will conclude that the conditioning adds interpretability but no predictive value.
Why not just probe features for site information? A site probe measures whether site is decodable, not whether it changes decisions. Holzschuh et al. show that models trained on several centers generalize reasonably (LOCO 0.74) even though site is almost certainly decodable from PET intensities. Site information can be present but harmless. DIF measures site effects on the output conditional on true anatomy, which is the decision-relevant quantity. The probe baseline tests this directly: if probe accuracy tracks held-out-site degradation as well as DIF does, this argument fails.
Why not just train on more centers and skip the audit? Multi-center training helps. Holzschuh et al. found significant gains for small centers at matched sample size. But it does not tell you which regions remain miscalibrated, and dominant-site skew persists. The audit is complementary. Our intervention experiment tests whether DIF-targeted correction beats uniform correction at an equal budget on top of multi-center training. If it does not, the audit's practical value is limited to reporting.
Proposed Experiments
The primary testbed is the six-institution prostate T2 MRI cohort of Li et al., with eight pelvic structures and an existing pelvic atlas. Six held-out folds give enough sites to estimate rank correlations. The second testbed is BraTS 2021, using source archive (TCGA, IvyGAP, CPTAC, and private contributors) as a coarse site proxy. It lets us test the confound between lesions and registration in the setting where atlas warping is known to struggle. If access permits, the four-center PSMA PET cohort is a third, deliberately imbalanced case.
The decisive check is the following. Across LOCO folds, we compute the Spearman correlation between region-level DIF magnitude on held-in sites and region-level change in boundary distance on the held-out site. We compare it against the per-region error-map baseline and the probe baseline using a paired bootstrap. We also require that uniform-DIF regions show a signed boundary shift at the new site in the predicted direction, which is a sharper prediction than "Dice gets worse."
There are two secondary checks. Adding the registration Jacobian should remove DIF concentrated around tumors in BraTS but not DIF near air–tissue interfaces, which separates registration artifacts from acquisition artifacts. DIF-targeted correction should improve held-out-site boundary error more than uniform correction at the same tuning budget. A negative result on the primary check would itself be informative: it would mean that for segmentation, case-mix adjustment does not help predict cross-site failure, a conclusion the field currently assumes without evidence.
References
- Psychometric properties of the Chinese version of the Body Esteem Scale for Adolescents and Adults (BESAA) among adolescents
- The impact of multicentric datasets for the automated tumor delineation in primary prostate cancer using convolutional neural networks on 18F-PSMA-1007 PET
- The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification
- FEW-SHOT Image Segmentation for Cross-Institution Male Pelvic Organs Using Registration-Assisted Prototypical Learning
- Machine learning of neuroimaging to diagnose cognitive impairment and dementia: a systematic review and comparative analysis
- Comparison and analysis of multiple machine learning models for discriminating benign and malignant testicular lesions based on magnetic resonance imaging radiomics
- FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks
- Factorised Representation Learning in Cardiac Image Analysis
- An R toolbox for score-based measurement invariance tests in IRT models
- Automatic Segmentation of Vestibular Schwannoma from T2-Weighted MRI by Deep Spatial Attention with Hardness-Weighted Loss
- nnU-Net for Brain Tumor Segmentation