Style-Invariant Surprise: Choosing What to Mask When Pretraining on Multi-Hospital Scans

Masked image modeling pretrains a network by hiding patches of an image and asking it to fill them in. Recent medical variants stop hiding patches at random and hide the "surprising" ones, meaning the patches a teacher network reconstructs worst, on the theory that these are where the anatomy lives. That theory holds on curated single-hospital data. It breaks on the unlabeled archives that actually exist, which mix scanners, protocols and stains. In those archives a patch can be hard to reconstruct for two reasons: it holds an unusual lesion boundary, or it carries an acquisition quirk such as a bias field, a vendor-specific reconstruction filter or a stain batch. An error-guided masking rule cannot tell these apart. So it spends its capacity learning to paint site-specific noise.

We propose a simple test to separate the two. Re-render the image in the "style" of other hospitals by swapping the low-frequency part of its Fourier amplitude spectrum, which carries global intensity and contrast but little shape. Then check whether the patch is still hard to reconstruct. Patches that stay hard under every style carry anatomical surprise and are the ones worth hiding. Patches whose difficulty appears or disappears with style carry acquisition surprise, and the model should not be trained to reproduce them. We further predict that this style-invariant surprise lands on the same places where expert annotators disagree. We will test that prediction directly against multi-rater annotations, and it is why the method should matter most for the uncertain, low-label segmentation problems clinicians care about.

Core Claim

Three prior works are directly relevant. AnatoMask uses the patch-wise reconstruction error of a moving-average teacher to pick patches to mask, moving from random to error-guided masking over training. It reports 84.3% Dice with STU-Net-B on TotalSegmentator, against 82.2% for UniMiSS. MPS-AMS clusters patch covariance statistics to find lesion-like patches and reports 0.5914 Dice on BUSI with 10% labels, against 0.5639 for plain MAE. Both were built and tested within single-domain data. Improved post-hoc probability calibration for out-of-domain MRI segmentation shows that out-of-domain MRI artifacts produce large, confident errors. Put together, these imply that error-guided masking will chase artifacts once the unlabeled pool spans sites.

Our claim is that masking saliency should be defined as reconstruction error that persists across plausible acquisition styles. Concretely, for patch $i$ of image $x$ we draw $K$ style-perturbed copies $x^{(1)},\dots,x^{(K)}$ by mixing the low-frequency Fourier amplitude of $x$ with that of images from other sites in the pool. We then score

$$a_i = \min_{k} e_i\big(x^{(k)}\big), \qquad s_i = \max_{k} e_i\big(x^{(k)}\big) - \min_k e_i\big(x^{(k)}\big),$$

where $e_i(\cdot)$ is the teacher's reconstruction error on patch $i$. The first score, $a_i$, is the invariant surprise and sets the masking probability. The second, $s_i$, is the style sensitivity and flags acquisition-dominated patches. We take the style perturbation from MiDSS, which mixes low-frequency amplitude spectra to build intermediate domains between a labeled source hospital and unlabeled target hospitals. MiDSS applies this mixing to uniformly chosen regions during fine-tuning. We instead use it as a measuring instrument during pretraining.

The same rule gives a concrete answer to the unresolved hide-versus-preserve debate. One side (AttMask, CIM, FreMIM) hides salient content to force the model to reason from context. The other side (MST, GrabDAE) preserves the salient foreground and suppresses background to avoid entangling domain noise with content. Our answer is to hide what is anatomically surprising, and to keep acquisition-surprising patches visible or drop them from the reconstruction loss. Both sides are then partly right, and the dividing variable is measurable.

Research Program

The first component is the masking rule itself, built on a 3D and 2D MAE-style encoder. Training starts with uniform random masking, as AnatoMask does. Once the teacher is stable, it shifts to sampling masked patches in proportion to $a_i$. Patches in the top quantile of $s_i$ are treated in one of two ways, and these form an explicit ablation arm. In the "leave visible" variant they are never masked. In the "exclude from loss" variant they may be masked, but their reconstruction is not penalized.

The extra cost is $K$ teacher forward passes per image. We will report wall-clock overhead and test whether $K=2$ or $K=3$, or computing scores once every few epochs, is enough. The file lists online saliency overhead as an open problem for this family of methods.

The second component is a diagnostic study of what the two scores actually capture, which we will run before any downstream claim. On datasets with multiple raters, we compute pixel-level inter-rater disagreement: the entropy of the rater label distribution. We then measure its rank correlation with $a_i$, with raw AnatoMask error, with $s_i$, and with the covariance-cluster assignment of MPS-AMS. On multi-vendor data, we measure how much of each score's variance is explained by site identity. The mechanism is supported only if $a_i$ tracks disagreement better than raw error does, and depends less on site identity.

The third component is downstream fine-tuning under realistic scarcity. Pretraining uses the full unlabeled multi-site pool. Labels come from one site at 5–20%. Segmentation heads are either deterministic or ambiguity-aware. For the ambiguity-aware setting we reuse, unchanged, the rater-latent model Pionono, which gives each rater a Gaussian in a latent space that conditions a U-Net, and the semi-supervised multi-rater method AmbiSSL, which generates diverse pseudo-labels by randomly pruning decoder weights. The contribution here is the encoder, not the head.

Evaluation always compares against multiple raters at the target site whenever such raters exist. This matters because a single-rater target mask would confound scanner shift with a change in contouring convention, which is exactly the confound the file identifies between rater-modeling and domain-adaptation work.

Baselines

The naive reference is uniform random masking at a 75% ratio (MAE) on the same multi-site unlabeled pool, followed by identical fine-tuning. This is the point every masking policy must beat. MPS-AMS reports that on BUSI with 10% labels it is beaten by a guided policy, 0.5639 against 0.5914 Dice. That result is in-domain only, so whether random masking stays behind under site mixing is an open question here.

The strongest existing masking method is AnatoMask. We will re-implement it on the same backbone, the same unlabeled pool, the same number of pretraining epochs and the same curriculum schedule. Its hyperparameters get the same tuning budget as ours, selected on a held-out source-site validation split. The best reported number for it in the file is 84.3% Dice on TotalSegmentator with STU-Net-B. That is in-domain CT. Its reported transfer to AMOS22 MRI has no number attached, and we found no reported result on Lacuna for AnatoMask under cross-scanner target evaluation. Beating it therefore means higher target-site Dice and lower Generalized Energy Distance on held-out sites, across seeds, with source-site performance not degraded.

The second covariance-based guided baseline is MPS-AMS, run the same way. It tests whether a cheap statistical saliency cue is already robust enough to site changes.

For the fine-tuning stage on shifted data, the strongest method is MiDSS. It reports 87.66% average Dice on multi-center fundus, against a fully labeled upper bound of 88.46%, and a 13.57% Dice gain on multi-site prostate MRI over prior methods. We will run MiDSS fine-tuning on top of each pretrained encoder, so the question becomes whether better pretraining still adds anything once strong shift-aware fine-tuning is used. The gap between 87.66% and 88.46% on fundus is small, so any Dice gain there will be modest. The main fundus claim therefore rests on distributional metrics against multiple raters, not on Dice.

For ambiguity metrics, the reference points are AmbiSSL's 0.1620 GED on LIDC-IDRI with 10% labels and Pionono's unweighted kappa of 0.758 on Gleason 2019. Both are in-domain.

The key ablations isolate the ingredient. First, AnatoMask trained on amplitude-augmented inputs, which uses the style perturbation as augmentation but not as a selection signal. Second, masking by $a_i$ computed with random-phase noise instead of cross-site amplitude, which tests whether the specific cross-site perturbation matters. Third, the two treatments of high-$s_i$ patches.

Why not just…?

Why not just harmonize the inputs and then run AnatoMask? This is the most serious threat. If amplitude mixing or intensity standardization removes acquisition surprise from the input, raw error may become a clean anatomical signal and our selection rule would add nothing. That is why "AnatoMask plus amplitude augmentation" is a first-class baseline, not a footnote. Our bet rests on a difference in what each approach does. Augmentation averages the loss over styles, while $a_i$ selects patches by their worst-case invariance. Structured artifacts such as bias fields or motion ghosts are spatially local and are not fully captured by a global low-frequency amplitude swap. If the augmented AnatoMask matches us on target-site GED and on the disagreement correlation, we will report that the simpler recipe suffices.

Why not just use random masking with more multi-site data? Random masking cannot be misled by artifacts, which is a real advantage in shifted pools. The in-domain evidence from MPS-AMS and AnatoMask favors guided masking, but it is in-domain only. A clean outcome where random masking beats AnatoMask on held-out sites while our rule beats both would confirm the core mechanism. An outcome where random masking beats everything would falsify the premise that saliency is worth computing under shift.

Why not skip clever pretraining and rely on MiDSS-style fine-tuning? MiDSS already bridges sites during fine-tuning, and its fundus result sits close to the fully labeled upper bound. Our experiments stack every encoder under MiDSS for exactly this reason. If the pretraining gains vanish there, the method's value is restricted to settings without shift-aware fine-tuning, and we will say so. MiDSS evaluates against single deterministic masks, so its effect on multi-rater calibration at target sites is untested.

Why not merge raters into a majority-vote mask and ignore disagreement? DiFF and the Latent Doctor Model both show that collapsing raters discards clinically useful structure. More importantly for us, a single target-site mask makes a drop in Dice uninterpretable: it could come from the scanner or from a different contouring convention. We keep majority-vote Dice as a secondary metric, but draw conclusions from the multi-rater metrics.

Proposed Experiments

The decisive diagnostic comes first. On LIDC-IDRI (12 radiologists), REFUGE-2 cup and disc (multi-rater) and Gleason 2019 (6 pathologists), we compute the rank correlation between each saliency score and pixel-level rater-disagreement entropy. On M&Ms (4 cardiac MRI vendors) and AMOS MRI (multiple scanners, with a reported cross-scanner mDice drop of over 20% in AMOS), we compute the fraction of score variance explained by scanner identity. If invariant surprise $a_i$ does not beat raw error on both counts, we stop and report a negative result.

The main transfer experiments follow. For fundus, we pretrain on the multi-center fundus pool used by MiDSS, fine-tune with labels from one center, and test on the others. REFUGE-2's multiple raters provide GED and soft Dice. For prostate, we use leave-one-site-out on multi-center prostate MRI and M&Ms, reporting Dice and calibration error on each held-out site. For pathology, we use Gleason 2019 as source and Arvaniti TMA as external test, fine-tune with Pionono, and report kappa per rater and the distributional KL to the rater panel as in LDM. LIDC-IDRI serves as the in-domain control: gains there should be smaller than on shifted benchmarks if the mechanism is specifically about shift.

All comparisons use three or more seeds, matched pretraining compute (with our $K$ extra forward passes counted against us), and label fractions of 5%, 10% and 20%.

The result that would confirm the proposal has three parts. Our rule beats AnatoMask and augmented AnatoMask on held-out-site GED and Dice. It matches them on source-site data. The gap widens as site heterogeneity in the unlabeled pool grows, which we will vary directly by adding sites one at a time.