When a segmentation network trained at one hospital is run on a scan from another, a common test-time fix is to generate several adapted versions of the model and trust the one whose output is least uncertain. Uncertainty here is predictive entropy, a measure of how spread out the model's class probabilities are. The weak point is that domain shift is the regime where medical models are most often confidently wrong. We propose a different rule. Each candidate adaptation is scored by how stable its segmentation is under small, realistic intensity perturbations of the input, measured separately for each anatomical structure. Candidates are then combined structure by structure using those stability scores. The bet is that a candidate which is wrong because of a scanner artifact will often stay confident, but its mask will move when the image's contrast or bias field is nudged.
This matters because single-image adaptation is what hospitals can actually deploy. Privacy rules keep source data away from the deployment site, and scans arrive one patient at a time. The best existing method in this setting ranks candidates by entropy. It inherits exactly the failure that the referral literature documents: under shift, a low-entropy prediction can be confidently incorrect. If stability predicts which candidate is correct better than confidence does, that is a cheap change to a working pipeline. It should matter most for the small structures, such as adrenal glands and duodenum, where current models fail under scanner change.
Core Claim
Among candidate adaptations of a frozen source model on one shifted test image, per-structure agreement under intensity perturbation ranks candidates by true Dice more accurately than balanced entropy does. Weighting candidates by this agreement, separately for each structure, improves single-image test-time segmentation over entropy weighting. The gains should concentrate on cases where the model is confidently wrong and on small organs.
The mechanism is simple. Let $k$ index candidate models, $c$ index anatomical structures, and $m = 1..M$ index random intensity perturbations (gamma, contrast, bias field). Let $\hat{y}^{c}_{k,m}$ be candidate $k$'s hard mask for structure $c$ on perturbed input $m$. Stability is the mean pairwise Dice among these masks:
$$S_{k,c} = \frac{2}{M(M-1)} \sum_{m<m'} \mathrm{Dice}(\hat{y}^{c}_{k,m}, \hat{y}^{c}_{k,m'}).$$
The final probability map for structure $c$ is $\sum_k w_{k,c}\, \bar{p}^{c}_k$. Here $\bar{p}^{c}_k$ is candidate $k$'s perturbation-averaged probability for $c$, and $w_{k,c} \propto S_{k,c}^{\gamma}$, normalized over $k$. The exponent $\gamma$ controls how sharply the weighting focuses, in the same way that $(\text{entropy})^\gamma$ does in the sample-weighting literature (Wang et al.; Entropy-Based Adaptive Weighting for Self-Training). Because $S$ is computed on hard masks, rescaling the logits cannot change it, whereas it does change entropy. That matters because Entropy Weighted Adversarial Training shows that entropy-based weights are sensitive to how logits are scaled.
Relation to Prior Work
The closest work supplies the method we modify. InTEnt builds candidate models for a single test image by interpolating batch-normalization statistics between source and test values. It then averages their outputs, giving more weight to candidates with lower foreground–background-balanced entropy. We keep its candidate generation and replace the weighting signal. We also move from one global weight per candidate to one weight per structure, because averaging over a single "foreground" lets the liver's entropy swamp the adrenal glands in a 15-organ setting like AMOS.
OptTTA learns photometric test-time augmentations by minimizing conditional entropy, with a marginal-entropy term to prevent collapse. It then averages its top views equally. We use perturbations only as a probe and do no per-patient gradient optimization.
Two papers motivate the change of signal. Avoiding Catastrophic Referral Failures In Medical Images Under Domain Shift shows that under geographic and demographic shift, standard and even Bayesian classifiers make low-entropy false positives. That undercuts any method that trusts low entropy on shifted data, although the paper studies whole-image classification rather than segmentation. Improved post-hoc probability calibration for out-of-domain MRI segmentation uses per-pixel instability under jitter as an input to a temperature map. It reports lower calibration error than local temperature scaling (ECE 12.78% vs 14.48% on simulated artifacts). This is evidence that instability carries information about out-of-domain error, but that paper only rescales confidence and never chooses among adaptations.
One benchmark finding constrains our design. The M&Ms challenge found that intensity-type augmentations, rather than spatial ones, close the gap to unseen vendors. We therefore restrict the probe to the intensity perturbations that mimic scanner differences.
Research Program
We will build the method on the released InTEnt-style candidate generator and apply it to a frozen source network. The number of candidates $K$ and perturbations $M$ will be set so that total forward passes fit a fixed latency budget.
The first stage is a diagnostic study that needs no new method. For every test image and every candidate, we record true per-structure Dice, balanced entropy, calibrated entropy and stability. We then measure within-image Spearman correlation between each score and Dice. We report this separately for the subset of confidently wrong candidates, defined as low entropy but Dice below the median.
The second stage evaluates the combination rule itself, with ablations that isolate its two ingredients: the stability signal and per-structure weighting.
The third stage extends the rule to 3D AMOS volumes with 15 classes. There we ask whether per-structure weights recover small-organ Dice under the cross-scanner MRI drop of "over 20%" reported in AMOS.
Baselines
The naive reference is the unadapted source model. OptTTA reports that adaptation beats it by 22.5% Dice on M&Ms, so we expect it to be far behind. We also include a uniform average of the same InTEnt candidates. This tests whether any weighting is needed at all.
The strongest existing competitor is InTEnt itself, which reports an average Dice of 71.6% across its 24 spinal cord, chest X-ray and retinal splits, against 68.7% for SITA. We will run InTEnt with the same source checkpoints, the same candidate set, and a matched number of forward passes. Beating it means a statistically significant gain in average Dice over those 24 splits under that equal-compute condition, plus a larger gain on the confidently-wrong subset.
We also run OptTTA with its released code on M&Ms and spinal cord grey matter. It gets the same tuning budget for its policy search. We report accuracy and latency, since OptTTA's per-patient optimization costs far more.
TENT and SAR are included as standard entropy-minimization adapters. InTEnt reports that they degrade in the single-image setting.
Two ablations isolate our ingredients. The first uses per-structure entropy weights, which tests whether the per-structure split alone explains any gain. The second is plain test-time augmentation averaging: it averages over the same $M$ perturbations per candidate but gives candidates equal weight. This separates the benefit of the weighting from the benefit of perturbation ensembling.
Why not just…?
Why not just calibrate the model first and keep entropy weighting? Temperature scaling fitted on source data changes the size of the entropy but rarely its ranking across candidates. The referral paper also found that fixing out-of-domain overconfidence required class-specific logit shifts tuned on labeled out-of-domain validation data, which a single-image deployment lacks. We will still run InTEnt on calibrated logits. If it matches stability weighting in the diagnostic correlation study, the stability signal is unnecessary and we will say so.
Why not just train with strong intensity augmentation and skip test-time weighting? This is a real threat. M&Ms reports that intensity augmentation cut the unseen-vendor gap from about 6% to 1.5% with nnU-Net. We will use an nnU-Net source model trained with that augmentation as the frozen network. The question becomes whether our rule adds anything on top of it, especially on vendor D and on AMOS small organs, where the remaining failures live. If gains vanish on this stronger source, the method is only useful for weakly augmented models.
Why not just average the perturbed predictions, which already buys the stability? Averaging makes predictions smoother, but it cannot down-weight a candidate that is consistently wrong. That said, much of any gain could come from the averaging itself. The test-time augmentation ablation at equal compute settles this directly.
Why not use random weights, which provably favor invariant features? Random Sample Weighting addresses a different problem: it reweights training samples to build pseudo-environments. It has been validated only on tabular data and, as a training-time method, cannot pick among test-time candidates for one scan. The uniform-average baseline is its test-time analogue, and it is already in the comparison.
Proposed Experiments
The benchmarks are InTEnt's 24 single-image splits (spinal cord MRI across four sites; Shenzhen, Montgomery and JSRT chest X-rays; CHASE, HRF and RITE fundus), M&Ms trained on vendors A and B and tested on unseen vendor D, and AMOS cross-scanner MRI. Metrics are Dice per structure and averaged, with AMOS small-organ Dice reported separately, plus ECE of the fused output and seconds per image.
The decisive check comes first and is cheap. If within-image rank correlation with true Dice is not higher for stability than for balanced or calibrated entropy, especially on confidently wrong candidates, the core claim fails and we stop.
The second check is the full weighting rule against InTEnt and the test-time augmentation ablation at equal compute. The third is whether per-structure weighting improves adrenal and duodenum Dice on AMOS more than on large organs. The third also asks whether the optimal $\gamma$ transfers across datasets or has to be tuned for each one, which would weaken deployability.
References
- Entropy-Based Adaptive Weighting for Self-Training
- Entropy Weighted Adversarial Training
- Medical Image Segmentation with InTEnt: Integrated Entropy Weighting for Single Image Test-Time Adaptation
- OptTTA: Learnable Test-Time Augmentation for Source-Free Medical Image Segmentation Under Domain Shift
- Avoiding Catastrophic Referral Failures In Medical Images Under Domain Shift
- Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation
- Multi-Centre, Multi-Vendor and Multi-Disease Cardiac Segmentation: The M&Ms Challenge
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Covariate-Shift Generalization via Random Sample Weighting