Predicting Which Scanner Shifts Break a Locked Segmentation Model

A medical segmentation model that works at the hospital where it was built can lose a large share of its accuracy at a hospital with different scanners. Worse, nobody can say in advance which new scanner will cause the collapse. Regulators also mostly forbid fixing the model after approval. We propose a cheap test that runs before deployment and uses only unlabeled images from the new site. It measures two things separately and multiplies them. The first is how much the model's accuracy depends on each spatial-frequency band: coarse shape, mid-scale edges, and fine texture and noise. The second is how much the new site's images contain, in each band, content that a source-trained autoencoder cannot reproduce. An autoencoder here is a network that compresses an image and rebuilds it. The claim is that failure happens when the new scanner changes exactly the bands the model relies on. That would explain why shift from scanner A to B can be harmless while B to A is catastrophic.

The same measurement links problems that are usually studied apart. Autoencoder blurriness, meaning the loss of fine detail in reconstructions, becomes a tool with a known limit. A blurrier autoencoder cannot see high-frequency novelty, so its capacity sets how far the probe can see. Model capacity becomes a hypothesis about sensitivity: larger segmenters may lean more on fine-frequency bands. Synthetic-to-real transfer becomes a controlled test bed. Images passed through a generator's autoencoder lose high-frequency texture in a way we can dial up or down.

Core Claim

For a fixed segmenter $f$ and a new target site, we predict the Dice drop as

$$\hat{\Delta}(f, \text{target}) = \alpha \sum_{b=1}^{B} s_b(f)\, n_b(\text{target}),$$

where $b$ indexes $B$ bands of a 3D discrete cosine transform (DCT) and $\alpha$ is a single scale fitted on held-out pairs. The sensitivity $s_b(f)$ is the Dice lost per unit of random, non-adversarial energy injected into band $b$ of labeled source validation volumes. The novelty $n_b$ is the excess band-$b$ energy of the autoencoder residual $x - \mathrm{AE}(x)$ on target images, relative to held-out source images.

The multiplicative form is the simplest model consistent with two facts. A shift in a band the model ignores should cost nothing. A band the model relies on should cost nothing if it is unchanged. The testable content is threefold. This predictor should rank directed site pairs by actual Dice drop better than distribution distances that ignore the model. It should get the direction of asymmetric pairs right. And the autoencoder residual should beat raw power-spectrum differences, because it discounts anatomical differences such as body size and field of view that the source data already covers.

Why Existing Work Motivates and Constrains This

Three results supply the motivation. Studying Robustness of Semantic Segmentation under Domain Shift in cardiac MRI benchmarks nnU-Net across vendors in the multi-vendor cardiac challenge (M&Ms). It reports strongly asymmetric transfer and says openly that it cannot explain it; that asymmetry is our primary target. AMOS shows two things we build on. A plain 3D-UNet beats Swin-UNetr on CT (89.04% vs 86.32% mean Dice), and accuracy falls by over 20% on unseen MRI scanners. The first result motivates our capacity-versus-sensitivity hypothesis, and the second gives us a second shift regime. Regulating AI Adaptation supplies the deployment constraint. Only 1.4% of 416 FDA-cleared devices were retrained within two years, and site retraining that recovered up to 0.23 AUC cost an average of 0.176 AUC at the original site. This is why we aim to predict failure without adapting the model.

The band-sensitivity measurement reuses a technique from Frequency Domain Adversarial Training for Robust Volumetric Medical Segmentation. That paper perturbs 3D DCT coefficients and shows UNETR falls from 74.3% to 32.2% Dice under frequency attacks, versus 62.7% under voxel-space attacks. We borrow the band decomposition but use random, energy-matched perturbations. We want an average-case sensitivity profile that relates to natural scanner differences, not a worst case.

The idea of an autoencoder as a site-invariant lens comes from Transfer Learning from Synthetic to Real Images Using Variational Autoencoders. That work maps real robot-camera images into a blurry "pseudo-synthetic" space so that nuisance detail disappears. We reject its design choice for medicine, for a reason several papers give. U-HVED and ProtoPatchNet both report that variational-autoencoder blur destroys the fine boundaries clinical tasks need. So we do not filter images through the autoencoder. We read its residual as a measurement and treat its blur as the probe's resolution limit.

MAISI generates 3D CT through a compressing autoencoder stage followed by a latent diffusion model. Its synthetic tumors improve segmentation Dice by 4–6.5%. This is why we expect the synthetic-to-real gap in current medical generators to sit partly in the bands their compression stage smooths away. Finally, ErrorNet is the closest work that uses an autoencoder to improve cross-dataset generalization without target data. It does so by injecting shape errors into 2D binary retinal masks, which makes it a repair method. It does not predict failure and does not model image-level frequency content.

Research Program

The work has three parts. The first builds the sensitivity profiles. For each segmenter we inject Gaussian energy restricted to one DCT band at a time into source validation volumes, sweeping energy levels, and record the Dice slope per band. The segmenters are 3D-UNet at two widths, nnU-Net with batch normalization, Swin-UNetr, and UNETR, each trained with the same augmentation recipe. Here we test the capacity hypothesis directly: do the transformer and wider models carry more of their sensitivity in the upper bands?

The second part builds the novelty probe. We train 3D convolutional autoencoders on unlabeled source volumes at a ladder of bottleneck sizes, from very blurry to near-lossless. We compute band-resolved residual energy on target and held-out source images. The ladder shows how probe resolution limits prediction. We expect a heavily smoothed autoencoder to have large high-band residuals on all images, so target novelty in those bands drowns in baseline error. That would be a quantitative statement of where blurriness stops being tolerable.

The third part creates a controlled synthetic-to-real regime. We pass labeled source CT volumes through autoencoders of graded capacity and use the reconstructions, with their original labels, as "synthetic" training data. Segmenters trained on these are tested on real volumes. Because we set how much high-frequency content is removed, this yields a dose–response curve on which the predictor can be checked under a known cause. If MAISI outputs are obtainable, they are a second, uncontrolled synthetic source.

Baselines

The naive reference predicts the same Dice drop for every target: the mean drop over training pairs. Any useful predictor must beat it in rank correlation, and it has zero ability to order asymmetric pairs.

The strongest model-agnostic competitor is a raw spectral distance. This is the per-band difference in mean DCT power between target and source images, combined with our same sensitivity profile $s_b(f)$. It is the most dangerous baseline because it shares everything except the autoencoder. It therefore directly tests whether conditioning on the source anatomy matters.

The strongest model-internal competitor is the segmenter's own behavior on unlabeled target volumes. We use mean predictive entropy and softmax confidence, and separately a maximum mean discrepancy between source and target encoder features. These are standard, need no extra networks, and see exactly what the model sees.

Two ablations isolate the key ingredient. Sensitivity-only uses the model profile with uniform novelty. Novelty-only uses the autoencoder residual summed across bands with uniform sensitivity. If either matches the full product, the interaction claim is wrong.

For fairness, every predictor uses the same unlabeled target volumes, capped at 20 per site to mimic a realistic pre-deployment sample. Every predictor gets the same single fitted scale on the same held-out pairs, and the same segmenter checkpoints. Tuning budgets are matched: band count and autoencoder size for ours, kernel bandwidth and feature layer for the discrepancy baseline.

No paper in the file reports a failure predictor of this kind. There is no reported result found on Lacuna for predicting cross-scanner Dice drop from unlabeled images. The reference numbers we must explain are the M&Ms vendor B→A Dice of 0.6622 with augmentation alone and 0.8457 with added batch normalization (Studying Robustness…), and the over-20% AMOS unseen-MRI drop. Beating the baselines means Spearman correlation between predicted and actual drop that exceeds the best baseline's under a paired bootstrap over pairs. It also means correct direction on a clear majority of asymmetric pairs where the true asymmetry exceeds 0.05 Dice.

Why not just…?

Why not just compare the power spectra of the two sites directly? This is the most serious threat, and we built it in as the main ablation rather than argue it away. A raw spectrum mixes scanner signature with anatomy and field of view. For example, abdominal CT from larger patients shifts power without any scanner change. The autoencoder residual subtracts what the source distribution already explains. If AMOS scanners, which cover mixed anatomy, show no gain from the residual over the raw spectrum, we will report that the autoencoder is unnecessary and keep the simpler probe.

Why not just read the segmenter's confidence on the new site? Softmax confidence is known to stay high on shifted inputs. Frequency Domain Adversarial Training… shows how badly models fail on band-limited changes that do not look unusual in voxel space. That suggests confidence will miss exactly the high-frequency vendor shifts we care about. The comparison on M&Ms, where vendor differences are largely reconstruction-filter and contrast differences, settles it.

Why not just train a more robust model with heavy augmentation and batch normalization and skip prediction? We should, and we include nnU-Net with batch normalization as one of the models. But the same study that reached 0.8457 Dice for B→A still reports unexplained asymmetry. Also, AMOS models fall by over 20% on unseen MRI scanners. Robust training lowers the risk, but it does not tell a hospital whether its scanner is the one that breaks the model. We also test whether augmentation flattens the sensitivity profile. If it does, the predictor should correctly forecast smaller drops for augmented models, which is a further check rather than a competitor.

Why not just fine-tune on local data? Retraining at the new site cost up to 0.268 AUC at the original site in Regulating AI Adaptation. Most cleared devices are never retrained. Prediction serves the locked setting that actually exists.

Proposed Experiments

On M&Ms we train each segmenter on vendor A or B and evaluate on every other vendor and center. This gives directed pairs that include the known asymmetric A/B pair. On AMOS we split by the 8 scanner systems, training on one CT group and testing on held-out CT and MRI scanners, to cover both mild and severe shift. On the synthetic ladder we train on autoencoder-reconstructed CT at five bottleneck sizes and test on real CT.

The decisive checks are these. The full product beats raw-spectrum, entropy and feature-discrepancy baselines in rank correlation across pooled pairs. It gets asymmetric directions right on M&Ms. On the synthetic ladder, predicted and actual drop rise together as reconstruction capacity falls, and they rise faster for models whose sensitivity sits in the upper bands. We also test whether Swin-UNetr and UNETR carry measurably more upper-band sensitivity than 3D-UNet. If that holds, it gives a mechanism for the AMOS capacity result.

Two outcomes would falsify the idea. If the novelty-only ablation matches the full predictor, the model-specific part adds nothing. If Spearman correlation on the synthetic ladder, where we control the cause, is not clearly positive, the approach fails even in its most favorable setting.

As a short follow-up, we can exploit a model's predicted weak band. Fine-tuning the model on its source data only, with the frequency-consistency loss of VAFT limited to that band, should shrink the actual drop on the predicted-worst target. That would show the diagnosis points at a real cause.