A radiomic model turns a medical scan into a few hundred hand-defined numbers describing the region of interest: intensity histograms, lesion shape, and texture statistics that capture how neighboring voxels co-vary. It then predicts a clinical label from those numbers. Many such models now pass "external validation," meaning they keep a respectable area under the ROC curve (AUC) at a second hospital. AUC only measures whether the model ranks sick patients above healthy ones. Clinicians act on the predicted probability crossing a threshold, and that probability depends on calibration, which a change of scanner or reconstruction protocol quietly breaks. A model can rank well at a new site and still recommend the wrong number of biopsies.
We propose a fix at development time, before any labels exist at the new site. During calibration, the model is shown its own training scans after physically plausible acquisition changes: resampled slice thickness, reconstruction-kernel blur, added noise, and nonlinear intensity remapping. Features are re-extracted from each perturbed scan, and the probability calibrator is fit on these synthetic "new scanners." A label-free distance between the new site's radiomic feature distribution and the training distribution then flags the sites where even this calibration is unlikely to be enough. We measure success in clinical-utility units, namely net benefit at prespecified decision thresholds, and compare against the obvious alternatives: feature harmonization, image normalization, and recalibrating on a small set of labeled local cases.
Core Claim
Our claim is that most of the clinical-utility loss of a locked radiomic model at an unseen site comes from miscalibration caused by acquisition shift, not from lost discrimination. We further claim that much of this loss can be recovered without target labels by fitting the calibrator on scanner-simulating perturbations of the training images, provided the perturbations are applied before feature extraction. The test is whether acquisition-perturbed calibration recovers a substantial share of the net-benefit gap between standard in-domain calibration and an oracle that recalibrates on labeled target-site cases. We also test whether a radiomic distribution distance tells us, in advance, which sites still fall short.
Several papers motivate this claim. Multicenter radiomics studies such as Yao et al.'s rectal-cancer Ki-67 model (external AUC 0.86), Wan et al.'s GIST risk model (external AUC 0.867 on n=25), and Song et al.'s gastric-cancer fusion model (external AUC 0.805 on a public archive cohort) all report AUC that survives transfer. Each then applies decision curves at cutoffs fixed at the development site, and none tests whether those probabilities stay calibrated. The decision-curve literature summarized in Quantifying Clinical Utility with Decision Curve Analysis shows that high-AUC models can have zero or negative net benefit when miscalibrated, which is exactly the failure static external validation cannot see. All models are local argues for recurrent local validation and recalibration, citing a sepsis model whose AUC fell from 0.73–0.83 to 0.60–0.64 at independent sites. That argument takes local labels as given; we ask how much can be done before they arrive.
Other papers supply the method. Tomani and Gruber show that temperature scaling and isotonic regression fit on in-domain validation data become overconfident under drift, and that fitting them on affine- and noise-perturbed validation images reduces calibration error. Their perturbations are generic and their setting is natural images. We replace generic corruptions with acquisition physics and move the perturbation upstream of a hand-crafted feature extractor. Because texture features respond nonlinearly to blur and resampling, the effect of a perturbation on the features cannot be predicted from its effect on pixels. Fréchet Radiomic Distance (FRD) supplies the shift score: a 2-Wasserstein distance over 464 standardized radiomic features that detects out-of-distribution datasets with AUC 0.94 and is stable with 10–100 images. Its authors use it as a static dataset comparison with a modest correlation to downstream degradation (r = −0.43). We test it as a gate on calibration quality, a more specific use.
One paper constrains the design. Baili et al.'s multi-centre normalisation study found that ComBat, an empirical Bayes method that removes per-site offsets from extracted features, erased center identity yet slightly hurt disease classification at unseen centers. Region-based piecewise linear histogram matching (PLHM), which aligns image intensities inside the region of interest, generalized best. We therefore treat image-level normalization as a component to combine with, not a competitor to replace, and we do not assume that removing site information is the goal.
Research Program
The pipeline has three parts. The first is a fixed radiomic classifier: features extracted with PyRadiomics, LASSO selection, and logistic regression or a random forest. Its parameters are locked after development, following the two-stage protocol that external validation requires. The second is a calibrator, Platt scaling or isotonic regression, fit on held-out development cases. Each held-out case is replicated K times under sampled acquisition perturbations, and features are re-extracted from every replicate. The perturbation family is modality-specific. For MRI it covers bias-field and intensity nonlinearity, in-plane resolution and slice-thickness resampling, and Rician noise at a range of levels. For CT it covers Gaussian kernel blur standing in for reconstruction-kernel changes, slice-thickness resampling, and dose-like noise. Perturbation strengths are capped by the range of acquisition parameters seen across the development cohort's own scanners, so that we do not simulate implausible hardware. The third part is a label-free gate. FRD is computed between the target site's unlabeled scans and the development distribution, and also against the cloud of perturbed replicates. The intended reading is that a site inside the perturbation cloud should be well served by the perturbed calibrator, while a site far outside it should be routed to local labeled recalibration before use.
Net benefit at threshold probability $p_t$ is $NB(p_t) = TP/N - (FP/N)\cdot p_t/(1-p_t)$, where $TP$ and $FP$ count true and false positives among $N$ patients classified positive above $p_t$. Our primary endpoint is net benefit averaged over a clinically prespecified threshold window, fixed before any target data are touched. We report it as the fraction of the gap closed between in-domain calibration (fraction 0) and oracle target recalibration on all target labels (fraction 1). Secondary endpoints are calibration slope and intercept, expected calibration error, and AUC. AUC should barely move, since calibration is monotone. That makes it a useful check that the gains come from calibration rather than from changes in discrimination.
Baselines
The naive reference point is the locked model with standard in-domain Platt scaling, applied unchanged at the target site. This is what the multicenter radiomics papers above effectively deploy. Every other method is scored by how much of its net-benefit loss it recovers.
The strongest label-free competitor is PLHM image normalization from Baili et al.. On the M&Ms cardiac MRI benchmark it reached balanced accuracies of 0.78 ± 0.08 to 0.79 ± 0.09 at unseen centers, versus 0.45 ± 0.28 to 0.60 ± 0.22 without normalization. We take this as the discrimination bar our pipeline must match. No reported result was found on Lacuna for calibration error or net benefit on M&Ms, so the utility comparison has no published number and will be produced here. We also run ComBat, fit on development sites only and applied to the target with its estimated site parameters. ComBat is the method most reviewers will name, and the same paper reports that it hurt unseen-center accuracy. PLHM plus in-domain calibration, and PLHM plus perturbed calibration, are both run, so that normalization and calibration effects can be separated.
The calibration competitor is the generic perturbation recipe of Tomani and Gruber: affine and noise corruptions with the same K, the same calibrator family, and the same tuning budget. This is the ablation that isolates our key ingredient. If generic corruptions close as much of the gap as acquisition-physics corruptions applied before feature extraction, the physics adds nothing. A third ablation applies perturbations directly to the feature vectors, as Gaussian noise on standardized features, instead of re-extracting features from perturbed images. This tests whether passing perturbations through the extractor matters.
The upper reference is local labeled recalibration, refitting the calibration intercept and slope on k labeled target cases with k in {10, 25, 50}. This is what recurrent local validation would do. We report the k at which local recalibration matches our label-free method, so that our method's value can be read as a number of labeled cases saved. All methods share the same locked classifier, features, development data, threshold window, and leave-one-site-out splits. No method is tuned on target data except the explicit k-shot references. To beat the naive reference, perturbed calibration must close a gap fraction whose bootstrap interval excludes zero on the majority of held-out sites. To beat the strongest label-free baseline, it must exceed PLHM alone on gap closed without lowering balanced accuracy below PLHM's reported range.
Why not just…?
Why not just recalibrate on a few labeled cases at each new site? This is the real threat, and All models are local argues for exactly that. Our proposal does not deny it; it measures the trade-off. If 10 labeled cases already close the gap, our method is worth little and we will say so. Even then, the label-free method covers the window before ground truth exists, which in oncology can mean waiting months for pathology or survival outcomes. The k-shot curves settle the question quantitatively.
Why not just harmonize features with ComBat? ComBat needs site batches estimated from target-site features, it aims to remove site signal rather than to fix probabilities, and on M&Ms it removed center identity while slightly degrading unseen-center classification (Baili et al.). We run it anyway, as a direct comparator on net benefit.
Why not just normalize the images with PLHM and stop? PLHM corrects intensity scale but not resolution, slice thickness, or noise texture, which are what move texture features. It is nonetheless a serious competitor. If PLHM plus in-domain calibration already matches PLHM plus perturbed calibration on net benefit, the perturbation step is unnecessary, and the factorial design will show that directly.
Why not just skip radiomics and train a deep network? On the small cohorts considered here, hand-crafted features were more stable. A random forest on radiomic features reached 76.5% accuracy against 51.5% for a fully connected network on BraTS 2019 survival (Brain Tumor Survival Prediction using Radiomics Features), and radiomic distributions stabilize at far smaller sample sizes than deep embeddings (FRD). The calibration question also applies to deep models, so this is not a dodge. We restrict to radiomics because that is where external-validation practice is most entrenched.
Proposed Experiments
The primary benchmark is M&Ms cardiac MRI (healthy vs. hypertrophic cardiomyopathy, five centers), evaluated leave-one-center-out. It is the one multi-center radiomics dataset in the file with a published unseen-center baseline. The threshold window represents referral for further cardiac workup and is fixed in advance. The second benchmark is the TCGA-GBM and TCGA-LGG MRI collections (Bakas et al.), with grade classification and leave-one-contributing-institution-out splits, using the provided automated segmentations to remove contouring variability. The third setting is a CT check modeled on the gastric and GIST designs, if a public multi-site CT cohort can be assembled from the cancer imaging archive. No reported result was found on Lacuna for that exact setup.
The first decisive check is whether, on held-out sites, the AUC drop is small while the net-benefit drop under in-domain calibration is large. If both drops are small, the premise fails. The second is whether acquisition-perturbed calibration closes more of the gap than generic perturbation and feature-noise ablations, and how many labeled cases local recalibration needs to match it. The third is whether FRD computed against the perturbation cloud predicts residual net-benefit loss across held-out sites better than FRD against the raw development set. With few sites per dataset this correlation will be noisy. We therefore pool site-by-method combinations across datasets and also create additional synthetic held-out sites by perturbing whole held-out centers at strengths outside the calibration range. We will treat the gate as validated only if the rank ordering it predicts holds on the real held-out centers, not just the synthetic ones.
References
- A novel radiomics based on multi-parametric magnetic resonance imaging for predicting Ki-67 expression in rectal cancer: a multicenter study
- A CT-based radiomics model for preoperative risk stratification of gastrointestinal stromal tumors
- Development and validation of a fusion model based on multi‑phase contrast CT radiomics combined with clinical features for predicting Ki‑67 expression in gastric cancer
- All models are local: time to replace external validation with recurrent local validation
- Post-hoc Uncertainty Calibration for Domain Drift Scenarios
- Fréchet radiomic distance (FRD): A versatile metric for comparing medical imaging datasets
- Minimising multi-centre radiomics variability through image normalisation: a pilot study
- Brain Tumor Survival Prediction Using Radiomics Features
- Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features