Separating Scanner Shift from Boundary Ambiguity in Cross-Site Pseudo-Labels

Hospitals have plenty of unlabeled scans from many scanners but only a handful of expert-traced masks, usually from one site. The standard way to use the unlabeled scans is pseudo-labeling: a model labels them and then trains on its own guesses. This fails in two opposite ways. It trusts wrong guesses on images from unfamiliar scanners, and it throws away correct but uncertain guesses at tissue boundaries, which experts themselves draw differently. Current methods use a single uncertainty score to decide which guesses to keep, so they cannot tell these two situations apart. They either suppress the boundaries or let scanner-induced errors through.

We propose to separate the two with a generative segmenter, a model that outputs many plausible masks for one image instead of a single mask. We run it on each unlabeled image under several simulated scanner styles. For each pixel, the spread of masks within a single style measures genuine boundary ambiguity, and that pixel is kept as a soft, probabilistic target. How much the average mask moves when only the style changes measures sensitivity to the scanner, and those pixels are suppressed. The claim to test is specific: this split yields better pseudo-labels than any single-score filter when labels are scarce and the unlabeled data come from other sites. It should also yield better-calibrated predictions on a scanner vendor never seen in training.

Core Claim

Let $a$ index a style perturbation of the input (a Fourier-amplitude swap that changes contrast while keeping anatomy, as used by Yao et al.). Let $z$ index the noise seed of a conditional Bernoulli diffusion segmenter (BerDiff), and let $y_i(a,z)\in\{0,1\}$ be the sampled label at pixel $i$. The law of total variance splits the pixel's total predictive variance exactly:

$$ \mathrm{Var}_{a,z}[y_i] \;=\; \underbrace{\mathbb{E}_a\big[\mathrm{Var}_z[y_i \mid a]\big]}_{U^{\text{amb}}_i:\ \text{ambiguity within a style}} \;+\; \underbrace{\mathrm{Var}_a\big[\mathbb{E}_z[y_i \mid a]\big]}_{U^{\text{shift}}_i:\ \text{movement across styles}}. $$

At a fuzzy myocardial border, a well-behaved sampler gives roughly half foreground under every style. That makes $U^{\text{amb}}$ high and $U^{\text{shift}}$ low, so the pixel should be trained on the soft target $\bar p_i=\mathbb{E}_{a,z}[y_i]$. At a pixel where a vendor's contrast flips the prediction, $U^{\text{shift}}$ is high and the pixel should get little weight.

The pseudo-label loss for a student network with prediction $q_i$ is

$$ \mathcal{L}_{\text{unl}} = \sum_i \exp(-\lambda\, U^{\text{shift}}_i)\;\mathrm{CE}(\bar p_i, q_i), $$

where $\lambda$ is a single temperature. There is deliberately no penalty on $U^{\text{amb}}$.

The contrast with the closest method is exact. CACPS (Yao et al.) down-weights pixels by the variance of a deterministic network's output across one Fourier swap. At an ambiguous border, a hard-edged deterministic prediction flips under small contrast changes, so CACPS counts ambiguity as shift and suppresses exactly the boundary pixels. DyCON showed that suppressing boundary pixels is harmful: on ISLES'22 with 5% labels, 61.48% Dice against 53.53% for BCP.

Positioning in Prior Work

The motivating evidence comes from two papers that seem to disagree. Navigating Uncertainty in Medical Image Segmentation finds that plain softmax entropy matches heavy probabilistic models when annotator variation is just drawing noise. It also proposes, only conceptually, a two-step "epistemic gatekeeper" that screens out unfamiliar inputs before modeling ambiguity. Improved post-hoc probability calibration for out-of-domain MRI segmentation shows deterministic segmenters become badly overconfident off-domain, with calibration error of 16.78% under local temperature scaling for bSSFP→LGE. That method is a post-hoc fix with a cardiac shape prior and never touches training. Our decomposition is a training-time version of the gatekeeper. It needs no shape prior, so in principle it also applies to irregular structures.

Diffusion has entered semi-supervised segmentation in two ways, and neither produces sampled uncertainty on unlabeled out-of-domain images. DiffRect uses latent diffusion to clean pseudo-label features, is evaluated only in-domain, and bypasses sampling at inference. TEDM uses a frozen diffusion model as a feature extractor and averages over timesteps for accuracy. Its Montgomery result of 90.5 Dice with 1% labels shows diffusion features transfer across sites, but it never treats the spread as uncertainty.

The method supply comes from two papers. Diffusion Models for Implicit Image Segmentation Ensembles shows diffusion sample variance traces lesion margins more tightly than MC-dropout, and that Dice plateaus after about 20 samples. BerDiff supplies the discrete formulation, which suits binary masks, converges faster than Gaussian SegDiff (21,000 vs 86,500 iterations), and samples in about 0.4 seconds.

The cross-domain semi-supervised methods set the bar we must beat. MiDSS mixes amplitudes and copy-pastes to build intermediate domains, and DAC co-trains two networks with asymmetric auxiliary tasks. Both are fully deterministic, and DAC's authors report failures on low-contrast, overlapping anatomy. UMCT is the one earlier cross-domain method that weights pseudo-labels by sampled uncertainty, but it uses MC-dropout and a single total-uncertainty score.

Research Program

The teacher is a 2D BerDiff segmenter trained on the scarce labeled slices, with the same Fourier style augmentation applied to labeled inputs. For each unlabeled slice we draw $A=4$ style perturbations and $S=8$ seeds, using sub-sequence sampling. From these we compute $\bar p$, $U^{\text{amb}}$ and $U^{\text{shift}}$.

The student is a standard U-Net trained on labeled data plus the weighted soft targets. After each round, the teacher is fine-tuned on labeled data plus confident student pseudo-labels. Pseudo-labels are refreshed on a rotating subset of the unlabeled pool rather than all of it at once.

At deployment only the student runs, so inference cost matches the baselines. An optional variant keeps the teacher at test time to produce reviewable maps of both uncertainty terms.

Cost is the main practical risk. At roughly 0.4 s per sample, 32 samples cost about 13 s per slice. We therefore budget refreshes of a fixed fraction of the pool per round and report GPU-hours for every method.

Baselines

The naive reference point is a U-Net trained only on the labeled slices, with the same intensity augmentation. The M&Ms challenge found this kind of intensity augmentation was the single most important ingredient for cross-vendor transfer. Any gain must be measured against this strong version, not a bare network.

The strongest existing competitor on the main benchmark is CACPS (Yao et al.). It targets exactly this setting, which is scarce labels, unlabeled multi-site data and an unseen test site. It reports 84.52% Dice on M&Ms with 2% labels and 85.72% with 5%. On Fundus we compare to MiDSS, which reports an average of 87.66% Dice against a fully supervised 88.46%. We also compare to DAC, which reports 76.35% optic-cup Dice at 10% labels. For DyCON-style continuous entropy weighting, no reported result was found on Lacuna for any cross-domain protocol, so we reimplement it inside our pipeline.

Fairness rules apply to every method. All use the same labeled splits, the same unlabeled pool and the same held-out vendor. All deployed students share one U-Net backbone and the same Fourier augmentation, and get an equal hyperparameter-search budget in GPU-hours. The teacher's sampling cost is charged to our method. Beating CACPS means a Dice gain on the held-out vendor at 2% labels that exceeds seed-to-seed variation over at least three label splits. It also means lower expected calibration error (ECE, the average gap between predicted confidence and actual accuracy) on that vendor.

Three ablations isolate the key ingredient. The first uses the same diffusion teacher but weights pixels by total variance, which tests whether the decomposition matters. The second keeps the decomposition but draws $z$ from MC-dropout, and the third keeps it but draws $z$ from a five-member deep ensemble. These two test whether generative sampling matters. The fourth replaces $\mathrm{Var}_z$ with the softmax Bernoulli variance $p(1-p)$ of a deterministic network, which tests whether sampling is needed at all.

Why not just…?

Why not just use CACPS-style Fourier variance with a deterministic network? It is the direct competitor, and our argument is that it misreads boundary ambiguity as shift. That is a claim about specific pixels, so we test it directly. If CACPS's retained-pixel masks do not under-cover boundary bands relative to ours, and our Dice does not improve, the thesis fails.

Why not just do the same decomposition with a deterministic softmax? This is the real threat. Navigating Uncertainty found softmax entropy competitive when ambiguity is small, and a calibrated softmax can express a 50/50 border. Against that, Ouyang et al. show softmax saturates under shift. The fourth ablation settles the question. If it matches the diffusion teacher, we will report that the decomposition, not generative sampling, carries the gain, and drop the diffusion teacher.

Why not just use MC-dropout or a deep ensemble, as UMCT does? These are much cheaper. Wolleb et al. report coarser, over-smoothed dropout maps, but that evidence is qualitative and in-domain. Ensembles detect unfamiliar inputs well according to Zepf et al.'s detection-GED analysis, so they may estimate $U^{\text{shift}}$ as well as diffusion does. The second and third ablations answer this at matched compute.

Proposed Experiments

The main benchmark is M&Ms under the semi-supervised domain generalization protocol at 2% and 5% labels, holding out one vendor (Canon, vendor D) as never seen. A second benchmark is Fundus in the mixed-domain protocol, and SCGM serves as a third cross-scanner check. For each method we report Dice, 95th-percentile Hausdorff distance (HD95) and ECE on the held-out site.

The first decisive check is pixel-level. On the held-out vendor, we bin pixels by $U^{\text{shift}}$ and $U^{\text{amb}}$ separately and measure error rate in each bin. The prediction is that error rises with $U^{\text{shift}}$ at fixed $U^{\text{amb}}$, while error at high $U^{\text{amb}}$ concentrates on boundary bands.

The second check validates $U^{\text{amb}}$ against real expert disagreement. M&Ms has only one annotator per image, so we use LIDC-IDRI's four raters with synthetic Fourier and bias-field shifts applied to the test set. There, $U^{\text{amb}}$ should track rater disagreement and stay stable under the shifts, while $U^{\text{shift}}$ should grow with shift strength. If $U^{\text{amb}}$ inflates under shift, the decomposition leaks, and the method is not separating the two sources.

The third check is the ablation ladder at matched GPU-hours. It determines which of the three ingredients earns its cost: the diffusion sampler, the variance split, and soft targets rather than thresholded ones.