Relabel the Translated Image, Not Just the Real One

Cross-modality segmentation often has to work without any labels in the new modality. A common example is learning to outline a tumor on T2 MRI when masks exist only for contrast-enhanced T1 MRI. The standard recipe has two steps. First, an unpaired image-to-image translator, a generator trained without matched image pairs to make source images look like target-modality images, converts each labeled source scan into a fake target scan. Second, a segmenter trains on these fakes, and each fake simply inherits the human mask drawn on its source scan. That inheritance step is almost never questioned. Yet translators are known to shift intensities near boundaries, deform small structures, and hide information in high frequencies to satisfy their round-trip losses. The inherited mask can therefore be correct for the source image and slightly wrong for the image the segmenter actually sees.

We propose to treat the label of a translated image as something to generate, not something to inherit. The idea comes from machine translation. There, Leveraging Synthetic Targets for Machine Translation showed that students trained on a teacher model's outputs beat students trained on human references, by +3.3 to +5.2 BLEU in low-resource settings. The explanation offered is that machine targets are more consistent and better matched to what the student can learn. Our claim is that the analogous move helps visual cross-modality adaptation. A segmentation teacher should re-annotate each translated image near the boundaries of its inherited mask, and the student should train on these image-consistent synthetic masks. This should help for two separable reasons: it repairs misalignment the translator introduced, and it removes boundary inconsistencies between human annotators. We will measure both reasons separately.

Core Claim and How It Differs From Prior Work

Existing work at this intersection always edits the image side and never the label side of the synthetic pair. SynSeg-Net (SynSeg-Net) trains the translator and segmenter jointly but applies the source ground truth unchanged to every translated image. TarGAN (TarGAN) adds a region-of-interest stream and a shape controller so that translated organs keep their shape. In both, the effort goes into making the image honor the old mask. The two-stage CrossMoDA pipeline of Li et al. (Unsupervised Cross-Modality Domain Adaptation…) inherits masks onto 2D CycleGAN outputs and names self-training on real T2 scans as future work. DEST (Partially Supervised Unpaired Multi-Modal Learning…) generates pseudo-labels on unlabeled real pixels by averaging model snapshots, and it has no image translator at all.

Our proposal differs from all of these in where the synthetic target goes. It replaces part of the human mask on the synthetic image. It does not add pseudo-labels to real target images, though we test that combination too.

ACAL (Augmented Cyclic Adversarial Learning) plays a motivating role. It showed that judging translation by task semantics instead of pixel reconstruction lifted SVHN→MNIST accuracy from 45.5% to 93.9% in a low-resource setting, evidence that the task model is the right judge of what a translation preserved. ACAL uses that judge to shape the generator through categorical predictions. We use a dense segmentation judge to rewrite the training target.

The machine translation papers supply both the hypothesis and its expected limits. Hrinchuk et al. report a ceiling effect: the synthetic-target advantage shrinks as the student approaches the teacher's capacity. GigaST shows that small students saturate on large pseudo-labeled corpora. We therefore predict gains concentrated in small students and small labeled sets, and we test this prediction directly.

Research Program

Let $x_S$ be a labeled source scan, $y_S$ its human mask, and $\tilde{x} = G(x_S)$ its translation into the target modality by a fixed unpaired translator $G$. A teacher segmenter $f$ is first trained on pairs $(\tilde{x}, y_S)$ in the usual way and stabilized by snapshot averaging in the manner of DEST. We then build a synthetic target $\tilde{y}$ for each translated image. It keeps the human mask everywhere except within a band of width $r$ voxels around the mask boundary. Inside that band, it takes the teacher's label wherever the teacher's confidence exceeds a threshold $\tau$:

$$\tilde{y}(v) = \begin{cases} \arg\max_c f_c(\tilde{x})(v) & \text{if } v \in B_r(y_S) \text{ and } \max_c f_c(\tilde{x})(v) > \tau \\ y_S(v) & \text{otherwise} \end{cases}$$

Here $v$ indexes voxels, $c$ indexes classes, and $B_r(y_S)$ is the band of voxels within distance $r$ of the boundary of $y_S$. The band keeps the teacher from inventing or erasing whole structures. This guards against the bias amplification and error cascades that the synthetic-target literature warns about. Under this rule, the human label still decides topology and the teacher only decides where the boundary sits in the translated image. A fresh student then trains on $(\tilde{x}, \tilde{y})$.

We study three settings of the band. Setting $r=0$ recovers standard inheritance. Small $r$ allows only boundary correction. Setting $r=\infty$ amounts to full sequence-level distillation as in Hrinchuk et al. These settings let us trace how much of the gain comes from local alignment versus wholesale replacement of the target.

The program has three analyses. The first measures how wrong inherited masks actually are. On the Brain TICV data used by SynSeg-Net, where 20 subjects have both MRI and CT, the true target-modality boundary is available. There we can directly quantify the misalignment between translated images, their inherited masks, and their teacher-relabeled masks.

The second separates the two hypothesized mechanisms. The relabeling procedure runs on the source modality alone, with no translation involved, and the resulting ceT1 segmenter is compared against one trained on human masks. Any gain there comes purely from annotator inconsistency. Gains that appear only after translation come from translator misalignment.

The third repeats Hrinchuk et al.'s switch experiment for segmentation. Students train first on inherited masks and then switch to synthetic masks, and vice versa. This tests whether synthetic targets act as an easier curriculum or as a better final objective.

Baselines

The naive reference point is training directly on source ceT1 scans with cropping and heavy intensity and spatial augmentation, with no translation. Li et al. show this alone is strong on CrossMoDA. Their reported numbers are 0.408 VS Dice without cropping, rising to 0.740 VS Dice and 0.731 cochlea Dice with cropping and augmentation, using translated training data. Any method that cannot beat crop-plus-augmentation at matched training budget is not worth its generator.

The standard pipeline baseline is translate-then-inherit: the same translator, the same segmenter architecture, and $r=0$. This is exactly our method with the key ingredient removed, so it is also the main ablation. The strongest reported system on our primary benchmark is Li et al.'s four-model ensemble of 2.5D and 3D networks trained on translated images with inherited masks. It reaches 0.794 VS Dice and 0.741 cochlea Dice on 32 CrossMoDA 2021 validation scans. We will compare against it in two ways. First, single model against single model, holding augmentation, cropping, and architecture identical to theirs. Second, ensemble against ensemble, with our relabeled targets fed into the same four architectures.

On the SynSeg-Net tasks, the reference is SynSeg-Net's joint training. The paper reports median Dice 0.919 on MRI→CT splenomegaly against 0.880 for the two-stage pipeline, and 0.966 against 0.952 on TICV. These numbers are available only if we can access the same data. If we cannot, we report only our re-run of both pipelines.

Self-training on real target scans is a separate baseline, discussed below. It has no reported result found on Lacuna for CrossMoDA, so we will implement it ourselves using DEST-style snapshot pseudo-labels.

All methods share the same translator checkpoint, data splits, voxel spacing, number of training iterations, and a hyperparameter search of equal size. For our method, the band width and threshold count against that search budget, not on top of it. Beating the baselines means two things: a VS Dice gain over translate-then-inherit that exceeds the seed-to-seed standard deviation over five seeds, and a lower average surface distance. A gain on top of self-training is the stronger claim we care about.

Why not just…?

Why not just self-train on the real T2 scans? This is the most serious threat. It is the next step Li et al. themselves propose, and it targets the real target distribution rather than a synthetic one. Our view is that the two approaches fix different errors. Self-training corrects for the gap between translated and real images, while relabeling corrects the mismatch between a translated image and its own label. Self-training on real scans also starts from a teacher trained on misaligned pairs. That said, if self-training alone matches self-training plus relabeling, the relabeling contribution is not real. The 2×2 experiment below (relabeling on or off, self-training on or off) settles this directly.

Why not just fix the translator so the mask stays valid? TarGAN's shape controller and ACAL's task-level cycle loss both push the generator to preserve structure, and TarGAN reports a segmentation score of 65.79% against 29.23% for StarGAN on CT→T1w translation. A better translator could make relabeling unnecessary. We will train a structure-preserving translator, a CycleGAN with an added segmentation-consistency term, and apply relabeling on top of it. If the misalignment measured on paired TICV data falls to the level of annotator noise and relabeling adds nothing, we will conclude that the generator-side fix suffices.

Why not just soften the mask boundaries? Label smoothing or down-weighting the loss near boundaries is far cheaper and also tolerates boundary noise. The difference is that smoothing is symmetric and blind to the image, whereas relabeling moves each boundary in the direction that specific translated image demands. We include boundary-weighted loss with the same band width $r$ as a baseline. If it matches relabeling, the honest reading is that we found boundary noise but no benefit from synthetic targets as such. In that case the paper would be reframed as a measurement study.

Proposed Experiments and Decisive Checks

The primary benchmark is CrossMoDA. We use the 2021 split, with 105 labeled ceT1 scans, 105 unlabeled T2 scans, and 32 validation scans, so that results compare directly with Li et al. We use the 2023 multi-site release, with 226 labeled and 295 unlabeled scans, to test robustness to scanner variation. On 2023 we also apply relabeling on top of IntraStyler-style diversified translations (IntraStyler). We report Dice and average symmetric surface distance for vestibular schwannoma and cochlea. Secondary benchmarks are CHAOS 2019, as used by TarGAN, and, where accessible, the SynSeg-Net MRI/CT spleen and TICV tasks. The paired TICV subjects provide the misalignment diagnostic.

The first decisive check is the 2×2 design described above. Relabeling should add at least as much over self-training alone as it adds over plain inheritance. Otherwise the idea is redundant with self-training.

The second check is the mechanism split. On paired TICV, Dice between the inherited mask and the true CT mask should fall below Dice between the relabeled mask and the true CT mask. On source-only ceT1, the relabeling gain should be measurably smaller than the post-translation gain. Together these show that translator misalignment, and not annotator style, drives most of the effect.

The third check is the ceiling prediction carried over from Hrinchuk et al. We sweep student size from a small 2D network up to the teacher's 3D architecture, and labeled-set size from 10% to 100% of source scans. The advantage should shrink as the student approaches the teacher. If it instead grows or stays flat, the machine translation analogy is wrong and the gain has a different cause.

The last check is the switch experiment, which indicates whether synthetic masks are best used as a curriculum or as the final training objective.