A segmentation model trained on MRI from one set of scanners can lose more than 20% of its accuracy on a scanner it has never seen. The AMOS abdominal benchmark reports exactly this drop, and it falls hardest on small organs such as the adrenal glands and duodenum (AMOS). Relabeling data for every new scanner is not realistic. Expert voxel-level contouring is slow enough that even AMOS needed a four-stage loop of radiologists to produce its labels. Retraining a deployed model on the new site also causes it to forget the old one: Regulating AI Adaptation finds that local fine-tuning recovers up to 0.23 AUC at the new hospital but costs an average of 0.176 AUC at the original hospital. A cheaper route is to re-render the scans we already have labels for so they look as if they came from the new scanner, then train one segmenter on the original and re-rendered data together. That route has a known failure mode. Image translators trained without paired data tend to move, shrink, or blur small structures, so the old label no longer matches the new image. The segmenter is then trained on wrong answers exactly where it is weakest.
This note proposes a scanner-to-scanner translator designed so that the label stays true after translation. It has two parts. The first is attention masking: inside the generator, a pixel is allowed to exchange information only with pixels of the same organ in the middle transformer layers. This keeps one organ's appearance from bleeding into its neighbor. The second is a pair of adversarial critics, which are networks trained to tell real target-scanner images from translated ones. A global critic checks the whole slice against the target scanner's overall intensity and layout. A local critic inspects full-resolution crops centered on small-organ boundaries. The two parts address complementary weaknesses. Masking works on a coarse token grid that is too blunt for an adrenal gland, and the full-resolution local critic compensates at the fine scale. The claim is that together they produce translated images whose source labels remain accurate. That accuracy, rather than visual realism, should determine how much of the unseen-scanner gap is closed on small organs.
Core Claim
The quantity that governs whether translation helps segmentation is label fidelity: how well the source mask still delineates each organ in the translated image. Unconstrained translators trade label fidelity for target realism, and they do so disproportionately on small organs. We predict that organ-isolated attention combined with a boundary-centered local critic raises small-organ label fidelity. We further predict that this gain, not global realism scores, is what carries through to unseen-scanner Dice. If the prediction holds, a single segmenter trained on source data plus translated data should recover a large share of the AMOS unseen-MRI-scanner gap. It should do so without any target labels and without degrading source scanners, which removes the forgetting trade-off that Wu et al. document.
Relation to Prior Work
Each strand of prior work plays a distinct role. The closest existing system is MSA-GAN (Alam et al.). It synthesizes pneumoconiosis lesions in 2D chest X-rays using a mask-disentangling generator and local and global critics, and uses the results as augmentation. It supplies the dual-critic design. It differs from this proposal in three ways: it pastes new pathology into a fixed domain, it never faces a scanner shift, and its local critic looks at lesion crops rather than label boundaries. Two further works motivate the label-scarcity angle and expose its weak point. Robust Histopathology Image Analysis: To Label or to Synthesize? builds synthetic nucleus layouts and refines them with a single-scale adversarial refiner, reweighting samples by how real the discriminator judges them. PathoGAN derives tumor masks by "healing" diseased MRI slices. Its authors attribute its boundary outliers to single-scale discriminators and call for multi-scale ones. Our design takes up that call directly.
The masking mechanism comes from natural-image layout generation. Freestyle Layout-to-Image Synthesis hard-masks cross-attention logits outside each object. DenseDiffusion shows that self-attention must be constrained too, and it also documents the constraint that coarse attention grids miss thin structures. Seg2Any finds that isolating self-attention only in middle transformer layers stops attribute bleeding while outer layers keep global coherence. We reuse that layer-selective rule. Organ masks play the role of entity masks, and the "attribute" being isolated is scanner-specific tissue appearance. None of these works uses adversarial critics or medical scanner shift. Finally, TransNetR and Double Encoder-Decoder Networks pursue robustness through segmenter architecture. They are complementary: our change is to the training data, and it can be applied to any segmenter.
Research Program
The translator takes a source MRI slice stack together with its organ label map and outputs the same anatomy in the appearance of a target scanner. We use three adjacent slices to limit inter-slice flicker. The translator is trained without paired data. It is a transformer generator operating on tokens of 8×8 pixels over 256×256 slices. In a contiguous band of middle layers, a self-attention logit between tokens $i$ and $j$ is set to $-\infty$ unless both tokens share an organ label, following the hard-masking rule of Freestyle Layout-to-Image Synthesis. Background forms its own group. Boundary tokens that cover several organs are assigned to all of the organs they touch. Outer layers attend freely so that bias fields and global intensity remain consistent. Which layers form the middle band is a tuned choice, and we sweep it rather than assume Seg2Any's layer range transfers.
The training objective has three parts:
$$ \mathcal{L} = \mathcal{L}_{\text{adv}}^{\text{global}} + \lambda_{\ell}\,\mathcal{L}_{\text{adv}}^{\text{local}} + \lambda_{c}\,\mathcal{L}_{\text{cycle}} $$
Here $\mathcal{L}_{\text{adv}}^{\text{global}}$ is the adversarial loss from a critic that sees the downsampled whole slice. $\mathcal{L}_{\text{adv}}^{\text{local}}$ is the adversarial loss from a critic that sees full-resolution crops. $\mathcal{L}_{\text{cycle}}$ is a translate-back reconstruction loss. $\lambda_{\ell}$ and $\lambda_{c}$ are weights. On translated images, the local critic's crops are sampled centered on boundaries of small organs taken from the source labels. On real target images, which have no labels, crops are sampled at matched anatomical positions using a coarse source-trained segmenter. Weighting the local and global losses against each other is a known source of instability in dual-critic training, so $\lambda_{\ell}$ is tuned on a held-out source-scanner validation split and never on target labels.
After translation, a 3D-UNet is trained on the union of source scans and translated scans, all carrying source labels. We use 3D-UNet because AMOS reports it as the strongest baseline on CT (89.04% mDice versus 86.32% for Swin-UNetr, AMOS). Alongside segmentation accuracy we measure label fidelity directly. For each translated image, we compare its source label against the prediction of an oracle segmenter trained on real labeled target-scanner data. That comparison gives a per-organ fidelity Dice. We can then test whether fidelity or realism predicts downstream gains.
Baselines
The naive reference point is a 3D-UNet trained only on the source scanners and tested on the held-out MRI scanner. This is the configuration AMOS describes as dropping "over 20%" on unseen MRI scanners. The upper reference point is the same network trained with real labels from the target scanner.
The strongest competitor we expect is aggressive intensity augmentation: random gamma, bias-field, contrast, noise, and histogram-matching transforms applied to source scans. It needs no generator and directly attacks the appearance differences behind scanner effects. It is run with the identical 3D-UNet, schedule, and hyperparameter search budget as our method.
The strongest generative competitor is an unconstrained unpaired translator with a single critic, the CycleGAN-style approach used as a comparator in MSA-GAN. We also run MSA-GAN-style dual critics without attention masking. The file reports no result on AMOS for these methods. The only relevant reported comparison is on the H1 chest X-ray task, where MSA-GAN augmentation reached 88.79% classification accuracy versus 86.57% for CycleGAN and 86.54% for standard augmentation (Alam et al.). That task is not comparable, so all AMOS numbers will be produced here under one protocol. We also include a target self-training baseline: pseudo-labels on the target scans, filtered by Monte Carlo Dropout uncertainty in the style of Can et al.. Finally, as an annotation-cost comparator, we fine-tune the source model on $k$ labeled target scans for $k \in \{1, 3, 5\}$, measuring both target gain and source retention.
Ablations isolate the key ingredients. We remove the masking, remove the local critic, move the local crops to random rather than boundary-centered locations, and mask all layers instead of the middle band. All generators receive the same GPU-hour budget and the same number of translated training scans.
Our method counts as beating the field only if it clears two bars. First, it must exceed the tuned intensity-augmentation baseline on mean unseen-scanner Dice, and on adrenal and duodenum Dice specifically, across all held-out-scanner splits with non-overlapping seed intervals. Second, it must stay within one Dice point of source-only performance on source scanners.
Why not just…?
Why not just use heavy intensity augmentation? This is the real threat. MRI scanner shift is partly a global intensity remapping that augmentation can cover. Where augmentation cannot help is texture and boundary sharpness that depend on local structure, which is where AMOS shows small organs failing. The comparison is set up to settle this. If augmentation matches our method on small organs, the generative machinery is unjustified and we will report that result.
Why not just use an unconstrained CycleGAN-type translator? The concern is that it improves realism while damaging label fidelity, and the PathoGAN authors' own diagnosis of boundary outliers from single-scale critics points that way (Andermatt et al.). This proposal does not assume the damage occurs; it measures it. The per-organ fidelity Dice will show directly whether unconstrained translation corrupts small-organ labels.
Why not just fine-tune on a few labeled target scans? A few scans may well beat us on the target. However, Wu et al. report large source-site losses under local fine-tuning, and each new scanner needs fresh expert contours. We report both sides of that trade, and we also test fine-tuning mixed with source replay, since replay might remove the forgetting. If $k=1$ replay fine-tuning matches us, the honest conclusion is that one labeled scan beats label-free translation.
Why not just self-train on target pseudo-labels? On small organs under a greater than 20% shift, the initial pseudo-labels are poor, and uncertainty filtering risks reinforcing systematic errors. The head-to-head per-organ comparison, including a combination of pseudo-labeling with our translations, will show whether the two approaches are redundant or additive.
Proposed Experiments
All experiments use AMOS MRI in a leave-one-scanner-out protocol. Target-scanner labels are used only for evaluation and for the fidelity oracle. A second setting adds AMOS CT as extra source data, since joint CT and MRI training is reported to add 2.14% mDice on MRI. Metrics are per-organ Dice and 95th-percentile Hausdorff distance on the target scanner, the same metrics on source scanners to measure forgetting, and per-organ label fidelity of the translations.
Three checks decide the question. The first is whether, across all methods and ablations, small-organ label fidelity correlates with small-organ target Dice more strongly than a global realism score such as the global critic's accuracy does. If realism predicts better than fidelity, the core claim fails. The second is whether removing masking lowers fidelity specifically on organs adjacent to larger ones, for example the adrenal gland beside the kidney or liver. That is the pattern bleeding would produce. The third is whether the full method beats tuned intensity augmentation on adrenal and duodenum Dice.
A negative result on the third check with a positive result on the first would still be informative. It would say that label fidelity matters, but that augmentation already preserves it perfectly because it never moves anatomy.
References
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Regulating AI Adaptation: An Analysis of AI Medical Device Updates
- Multiscale attention generative adversarial networks for lesion synthesis in chest X-ray images
- Robust Histopathology Image Analysis: To Label or to Synthesize?
- Pathology Segmentation Using Distributional Differences to Images of Healthy Origin
- Freestyle Layout-to-Image Synthesis
- Dense Text-to-Image Generation with Attention Modulation
- Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Control
- TransNetR: Transformer-based Residual Network for Polyp Segmentation with Multi-Center Out-of-Distribution Testing
- Double Encoder-Decoder Networks for Gastrointestinal Polyp Segmentation
- Learning to Segment Medical Images with Scribble-Supervision Alone