Translate Only What the Segmenter Sees: Masked Partial Synthesis for Cross-Modality Segmentation

Medical segmentation networks trained on one scanner type, such as contrast-enhanced T1 MRI, usually fail on another, such as T2 MRI. The standard fix translates every labeled source scan into the style of the target scanner with an image generator, then trains the segmenter on the translated images using the original labels. This works well; winning teams on the CrossMoDA challenge use it. It is also expensive. Diffusion translators, which build an image by denoising random noise over many steps, pay for every pixel of every slice, and exemplar-style methods multiply that cost by the number of target styles. Meanwhile, masked training, where the model sees only part of its input and must infer the rest, is known to make networks rely on global context rather than local shortcuts. Put these two facts together and a question follows. If the segmenter is trained to work from 25–40% of the image, why translate the other 60–75%?

We propose masked partial synthesis. The expensive translator runs only on a sparse, anatomy-aware set of visible patches in each source volume. Everything else is replaced by mask tokens, and the segmenter learns to predict the full source label map from target-style visible patches plus masked context. The occlusion-robust objective is the mechanism, cross-modality segmentation is the application, and synthesis compute is the budget being spent. The claim is that, at a fixed translation budget, many partially translated volumes with more style diversity beat a few fully translated ones. The same training should also make the segmenter robust to real field-of-view truncation on the target modality.

Core Claim

Translation compute is wasted on regions whose appearance the segmenter never needs. A masked segmenter uses target-style appearance only for visible tokens, so the cost of synthesis scales with the visible fraction $\rho$. With per-volume full-translation cost $c$ and a fixed budget $B$, the number of distinct translated source–style pairs becomes $M = B/(\rho c)$ instead of $B/c$. This holds roughly for convolutional translators and diffusion samplers, whose cost is close to linear in pixel count when run on crops with a fixed context margin.

We make two testable predictions. First, at equal compute, partial synthesis with anatomy-aware masks matches or exceeds full translation on target Dice. Second, the gain comes from spending the budget on more anatomies and styles, not from masking as regularization alone. The decisive ablation applies the identical masks to fully translated volumes. If partial and full-then-masked translation score the same, translating the masked region was pure waste.

Relation to Prior Work

Three works motivate the gap and one constrains it.

AdaptDiff trains a conditional diffusion translator on noisy target pseudo-labels to turn fundus images into OCT-A-style vessel images. It reaches 81.94% Dice on OCTA-500(6M), against 73.16% without adaptation and 84.78% for a supervised oracle. However, it samples full images through many steps and uses no masking. IntraStyler captures scanner variability by translating each source volume under many exemplar styles, which multiplies synthesis cost; we reuse its style-diversity goal but pay for it per visible patch. SynthMix is the closest existing work. It blends a source image with its own translation at patch level on an 8×8 grid and judges the blend with a patch discriminator, reaching 82.5% Dice and 3.8 ASSD on MR→CT cardiac adaptation versus 74.1%/7.0 for SIFA. SynthMix still translates the whole image and uses the mix to sharpen the translator. We never translate the unused patches, and we use masking to train the segmenter.

Mx2M masks one sensor stream and predicts it from a paired second stream. It reports that masked reconstruction conflicts with adversarial distribution matching. That constraint shapes our design: the only adversarial component is the separately trained translator, and the segmenter sees a plain supervised masked objective, so the two objectives never share gradients.

Two works supply the masking method. HU-based Foreground Masking shows that restricting masked modeling to tissue-dense 16³ cubes beats uniform masking (84.64% BTCV, 92.43% Flare22 Dice) and transfers to MRI. We borrow foreground selection but replace its fixed threshold with a per-volume intensity threshold, because MRI lacks Hounsfield units. Hole-robust Wireframe Detection constrains masks so that thin lines stay partly visible. We apply the same idea to the cochlea and capillaries, which random 75% masking (Chen et al.) can erase entirely. Because the source labels are known, we can guarantee that every thin labeled structure keeps a visible anchor.

The overhead framing comes from NLP distillation, where the teacher's generation bill dominates cost: TrueTeacher used a 540B-parameter teacher to annotate 1.4M examples. Beyond Quantity finds that raw synthetic volume saturates without diversity, which is why we spend the saved budget on distinct anatomy–style pairs rather than more copies.

Research Program

We build three pieces.

The first is a patch-level translator. For CycleGAN/CUT, which the CrossMoDA winners use, this is a translator applied to 3D crops with a context margin. For diffusion, it is AdaptDiff's conditional model sampled on crops. We measure seam artifacts at the boundaries between adjacent visible blocks.

The second is a mask policy. It chooses a visible set of block-shaped regions covering fraction $\rho$. Selection is weighted toward foreground by a per-volume Otsu threshold and always includes at least one block overlapping each thin labeled structure. Style exemplars can differ across volumes, and optionally across blocks.

The third is a segmenter, nnU-Net or SwinUNETR. It is trained on target-style visible blocks with learned mask tokens elsewhere and predicts the full label map. A short final phase on a small fraction of fully translated volumes closes the train–test gap, since real target scans arrive unmasked. This is followed by the standard CrossMoDA self-training on real target pseudo-labels, which costs no synthesis.

Compute is logged as translator-inference GPU-hours. Translator training cost is reported separately and honestly, because partial synthesis does not reduce it. We sweep $\rho \in \{0.2, 0.3, 0.5, 1.0\}$ at several fixed budgets to trace accuracy–compute curves.

Baselines

The naive reference is the source-trained segmenter applied directly to target scans. For OCTA-500 this is AdaptDiff's reported 73.16% Dice.

The strongest full-translation baselines are domain-specific. For CrossMoDA, we use the Bridge-and-Refine pipeline described in the crossMoDA 2021–2023 review: CycleGAN/CUT translation, nnU-Net, then self-training. Winning 2021 teams report median Dice of 87.0% for VS and 85.7% for cochlea (CrossMoDA 2021), and the review identifies a plateau near 88%. For MM-WHS MR→CT, we use SynthMix at 82.5% Dice and 3.8 ASSD. For OCTA-500, we use AdaptDiff at 81.94%.

We re-run each baseline ourselves on the same data splits, backbone, augmentation, and tuning budget, so its compute can be logged. Our method beats a baseline if it matches the baseline's Dice within noise at no more than half its translation GPU-hours, or exceeds it at equal GPU-hours. No reported result was found on Lacuna for translation GPU-hours of any of these methods, so all compute numbers will be our own measurements.

Four further baselines and ablations isolate the key ingredient:

  • A cheap full translator: few-step diffusion, or CycleGAN in place of diffusion, at the same budget.
  • ROI cropping followed by full translation of the crop, following Li et al., whose crop plus augmentation reached 0.740 VS and 0.731 cochlea Dice.
  • Full translation followed by the identical masks. This is the decisive ablation.
  • Random masks versus anatomy-aware masks.

Why not just…?

Why not just use a faster translator? This is a real threat. If a CycleGAN or a four-step diffusion sampler translates full volumes cheaply enough, partial synthesis saves little. We settle it with compute-matched curves: partial synthesis is also applied on top of the cheap translator, so the question becomes whether the savings compound. If full translation with the cheap translator dominates, we report that masking contributes robustness but not efficiency.

Why not just crop to the region of interest? Cropping already removes most background on CrossMoDA, as Li et al.'s jump from 0.408 to 0.740 Dice shows. We therefore apply partial synthesis inside the crop and treat cropping as a baseline, not a competitor. On MM-WHS, the heart fills much of the field, and on OCTA-500, vessels span the whole image. There, cropping cannot save compute, and those benchmarks test whether masking can. If partial synthesis helps only where cropping is impossible, we will state the claim that narrowly.

Why not just mask fully translated images, or pretrain with masked modeling on target scans? Masking fully translated images is the decisive ablation. Equal accuracy there shows that the masked-region translation was waste; higher accuracy shows that translated context matters and falsifies our claim. Masked pretraining on unlabeled target scans in the style of HU-based masking is orthogonal to our method, so we run it under both the full and the partial translation arms.

Why not skip synthesis and align latents? Latent VAE prior matching removes pixel translation entirely and reaches 72.18% Dice from one CT scan. That sits roughly ten points below SynthMix on the same benchmark and lacks anatomical constraints, so we include it only as a zero-synthesis reference point.

Proposed Experiments and Decisive Checks

We evaluate on CrossMoDA 2021 and 2023 (Dice and ASSD for VS and cochlea), MM-WHS MR→CT (Dice and ASSD), and OCTA-500(6M) (Dice). For each method, we plot target Dice against translation GPU-hours.

Three checks decide the outcome. The first is the full-then-masked ablation against partial synthesis at identical masks. The second is compute-matched diversity: at a fixed budget, 1/ρ times more source–style pairs should beat full translation, and the gap should widen when IntraStyler-style multi-exemplar sampling is enabled. The third is thin-structure recall: cochlea Dice and vessel connectivity under anatomy-aware versus random masks at $\rho=0.25$.

Finally, we truncate the field of view of target test scans synthetically, with blocks and slab crops at 10–40%. The masked-trained segmenter should degrade less than full-translation baselines. This tests whether the occlusion training transfers to real target-modality occlusion rather than merely saving compute.