Remask and Repaint: Masked Discrete Diffusion as an Anatomical Repair Step for Scanner-Shifted Segmentation

When a segmentation network trained at one hospital runs on scans from another, it rarely fails everywhere at once. It fails in particular places: around the thin duodenum, at a fragmented vessel, at an organ boundary blurred by an unfamiliar MRI sequence. It is also most confident exactly where it should not be. This proposal treats that repair problem the way non-autoregressive text generators treat a rough draft. Text models of this kind write all words in parallel and then fix them in a few passes. Here, we keep the frozen segmenter's confident voxels and "mask out" the voxels it is unsure about, meaning we replace their labels with a blank token. A learned model of multi-organ anatomy then fills those blanks back in over a handful of parallel steps. Running several independent fill-ins gives, for free, a probability for each voxel that reflects anatomical plausibility rather than the shifted image's misleading contrast.

The tool for the fill-in is masked discrete diffusion: a generator that learns to turn a partially blanked sequence of categorical tokens back into a complete one by unmasking a fraction of positions per step. MaskGIT used the same procedure for image codebook tokens. The bet is that anatomy changes much less across scanners than intensity does. A model of what plausible label maps look like is therefore a more stable thing to lean on under shift than a better image encoder. The specific claim is that combining three choices yields cross-scanner gains in both overlap and calibration that a single-pass refiner, heavier augmentation, or temperature scaling cannot match. The three choices are: remask where the base model is uncertain rather than at random; train the label prior on source data alone; and read voxel probabilities off repeated sampling.

What Exists and What Is Different

The closest work is SegRefiner. It recasts mask refinement as a discrete diffusion between two pixel states, "fine" and "coarse." Its forward process erodes and dilates ground-truth masks, and a conditional U-Net reverses that process in about six steps. It was evaluated only on natural-image benchmarks (BIG, LVIS, DIS5K) with binary masks and no distribution shift. We depart from it in three ways:

  • Label space. Our tokens are the full multi-class label vocabulary (15 abdominal organs on AMOS) plus an absorbing blank token, rather than a binary fine/coarse flag.
  • Where to repair. At test time the positions to repair are chosen by the frozen base model's own uncertainty, not by a fixed corruption schedule.
  • Output. We sample several chains and treat their agreement as a calibrated probability, rather than outputting a single refined mask.

ErrorNet supplies the motivating evidence. Under shift, failures are topological: broken vessels and holes. A learned shape prior that injects realistic errors, followed by a one-shot additive correction, lifted cross-domain retinal Dice to 71.0. ErrorNet's correction is deterministic and single-pass, and its variational shape prior is fit to one structure. We test whether iterative, multi-class, stochastic repair does better.

Improved post-hoc probability calibration for out-of-domain MRI segmentation constrains the design. It shows that a denoising-autoencoder shape prior improves out-of-domain calibration over local temperature scaling. It also notes that such priors stay tied to rigid, consistent shapes like the heart. A masked diffusion prior is a natural candidate for the multi-organ, variable-shape case, because iterative unmasking is built to represent several plausible completions rather than a single average. The cardiac result on its own does not show that this works, which is part of what we test.

Two results from the text side shape the design. Promises and Pitfalls of Generative Masked Language Modeling finds that parallel iterative decoders do well when the output distribution has low entropy and poorly when it has high entropy. It also finds that training with larger mask ratios is statistically more efficient. Label maps conditioned on mostly-known surrounding anatomy are a low-entropy target, which is the favorable regime. We therefore train with aggressive mask ratios. Bringing Stability to Diffusion reports that masked diffusion training has high gradient variance, and we adopt its variance-reduction training techniques. The confidence-ordered unmasking schedule comes from MaskGIT.

Core Mechanism

Let $y^0$ be the frozen segmenter's label map and $c_i$ its confidence at voxel $i$, measured as the maximum softmax probability averaged over test-time augmentations. We replace with the blank token every voxel whose confidence falls below a quantile threshold $\tau$, chosen on source validation data. The refiner $p_\theta(y_i \mid y_{\text{visible}}, x)$ predicts labels for blanked voxels from the visible labels and the image $x$. Starting from the blanked map, it commits the most confident fraction of predictions at each of $S$ steps, as in MaskGIT. The calibrated probability of class $k$ at voxel $i$ is the frequency

$$\hat p_i(k) = \frac{1}{K}\sum_{j=1}^{K} \mathbb{1}[\,y_i^{(j)} = k\,],$$

where $y^{(j)}$ is the output of the $j$-th of $K$ independent sampling chains. Confident voxels keep the base model's probability.

The refiner is trained only on source-domain ground truth. Its corruptions mix two kinds of blanks. Uniform random masks follow the high-ratio recipe. "Failure-shaped" masks are placed where the base model is uncertain on source images corrupted with synthetic MRI bias fields, motion, ghosting and k-space spikes, generated using Torchio as in the calibration paper.

The image input carries the same shift as the base model's input, so it is fed through a dropout channel. With some probability during training the image is withheld, and at test time the image's influence is blended with a weight $w$:

$$\log p_w = \log p_\theta(\cdot \mid y_{\text{vis}}) + w\,\big[\log p_\theta(\cdot \mid y_{\text{vis}}, x) - \log p_\theta(\cdot \mid y_{\text{vis}})\big].$$

Here $p_w$ is the blended prediction, $p_\theta(\cdot \mid y_{\text{vis}})$ is the anatomy-only prediction, and $p_\theta(\cdot \mid y_{\text{vis}}, x)$ is the image-conditioned prediction. Setting $w=0$ gives pure anatomy repair, and $w=1$ gives the fully image-conditioned refiner. The weight $w$ is selected on synthetic-artifact source validation, never on target data.

For 3D volumes the refiner operates on overlapping patches of the label map at reduced resolution around blanked regions. This keeps memory manageable.

Research Program

We build the refiner on top of frozen base segmenters. On cardiac MRI the base is a 2D U-Net, and on AMOS it is a 3D U-Net, matching the AMOS finding that the plain 3D U-Net is the strongest architecture there. We then answer three questions in order.

The first question is whether uncertainty-guided remasking actually targets the errors. We measure what fraction of base-model errors on shifted targets fall inside the blanked set. If this coverage is poor, the method cannot work and we stop.

The second is whether iterative stochastic repair beats one-shot repair on matched compute and data.

The third is whether sampling-frequency probabilities give better out-of-domain calibration than learned temperature maps. This matters most for small, variable organs (adrenal glands, duodenum), where AMOS shows every architecture degrades.

Baselines

The naive reference is the frozen base segmenter trained with the same intensity and artifact augmentation the refiner sees. Any gain must come from repair, not from extra augmentation. Simple morphological post-processing (keep the largest connected component per organ, fill holes) is reported alongside it as a zero-learning repair.

The strongest existing method for overlap under shift is ErrorNet. On the retinal CHASE to DRIVE/STARE/ARIA/HRF transfer it reports an average Dice of 71.0, about 2.5 points above the best domain-adaptation baseline it compared against. We re-run ErrorNet on that benchmark and on cardiac MRI with the same base U-Net and the same source data. Its tuning budget, measured in GPU-hours and validation trials, is matched to ours. Beating it means a higher average target Dice on the retinal transfer than our re-run of ErrorNet, and a reproduction of the 71.0 within noise.

The strongest calibration method is the shape-prior calibration network. It reports Expected Calibration Error (ECE), the average gap between a model's confidence and its accuracy, of 15.37% for bSSFP to LGE cardiac MRI, against 16.78% for local temperature scaling. On simulated artifacts it reports 12.78% against 14.48%. We must go below 15.37% on bSSFP to LGE, on the same frozen base model, without hurting Dice.

SegRefiner ported directly to medical masks is the closest competitor at the intersection. We run its binary fine/coarse formulation per organ with its six-step schedule. For SegRefiner on any medical cross-scanner benchmark, there is no reported result on Lacuna, so it is evaluated by our re-implementation.

For AMOS cross-scanner MRI, the benchmark reports only that unseen-scanner drops exceed 20% mDice, with no per-scanner baseline figure. The base 3D U-Net we re-train is the reference.

Three ablations isolate the key ingredients. The first replaces uncertainty-guided remasking with random remasking at the same mask ratio. The second fixes $w=1$, removing the anatomy-only path. The third sets $S=1$ and $K=1$, collapsing the method to a single-pass deterministic refiner of the same size. The third ablation is the direct test of whether "non-autoregressive iterative" matters at all.

Why not just…?

Why not just train the 3D U-Net with heavier intensity augmentation? This is the most serious threat, because AMOS shows that the plain convolutional baseline beats more elaborate architectures. Augmentation acts on the image side and leaves the model free to output anatomically impossible maps when contrast is unfamiliar. The over-20% drop on unseen MRI scanners shows augmentation-trained models still collapse. We settle this by giving the base model and the refiner identical augmentation. If the augmented base closes the gap and the refiner adds nothing on AMOS MRI or LGE, the proposal's claim is false for those shifts, and we will report it that way.

Why not just use a one-shot refiner such as ErrorNet? ErrorNet works, and it is our main overlap baseline. Its correction is a single deterministic pass with a single-structure prior. It produces no probability, so it cannot address the overconfidence documented under shift. The $S=1, K=1$ ablation is the fair version of this question, holding the network fixed. If one step matches six on Dice and ECE, the iterative part is unjustified and we would recommend the simpler model.

Why not just apply temperature scaling, or a denoising-autoencoder shape prior, for calibration? Local temperature scaling only rescales confidences voxel by voxel. As shown in the calibration paper, it misses spatially coherent anatomical violations, losing 16.78% to 15.37% ECE on bSSFP to LGE. The autoencoder prior is that paper's own fix, and the paper restricts it to rigid shapes. We compare against both on cardiac data, where they are strongest, and run the autoencoder on AMOS to test whether a single-mode prior breaks on 15 organs.

Why not just keep connected components and fill holes? Morphological cleanup removes stray fragments but cannot reconnect a broken vessel or decide which organ a mislabeled boundary region belongs to. Reporting it next to the naive baseline shows how much of any gain is trivial.

Decisive Checks

The experiments use three transfers. The first is cardiac MRI, training on ACDC and MS-CMRSeg bSSFP and testing on LGE, plus Torchio-simulated artifacts. The second is retinal vessels, training on CHASE and testing on DRIVE, STARE, ARIA and HRF. The third is AMOS, training on CT plus seen-scanner MRI and testing on held-out MRI scanners. FeTA 2024 multi-center fetal MRI serves as an extra transfer with no reported baseline. The metrics are Dice and mDice, a per-organ breakdown for small structures, ECE, and a topology error count (connected components versus ground truth). We also report wall-clock refinement time per volume as a function of $S$ and $K$.

The proposal stands or falls on four outcomes. Error coverage of the blanked set must exceed the coverage achieved by random remasking at the same ratio. The full method must beat the matched single-pass ablation on both Dice and ECE. It must also beat the equally augmented base on AMOS unseen-scanner MRI, with the gain concentrated on small organs. Finally, on bSSFP to LGE it must report ECE below 15.37%. If calibration improves but overlap does not, the contribution narrows to uncertainty estimation, and we would say so.