Carrying Scribble Labels Across Imaging Modalities with a One-to-Many Translator

Medical image segmentation models trained on one kind of scanner image, such as MRI, often fail on another, such as CT. On whole-heart data an unadapted MR-trained model scores only 17.2% Dice on CT (SIFA). The standard fix is to translate labeled source images into the look of the target modality and train on the translated images. Every strong method of this kind assumes the source images carry dense, voxel-by-voxel expert outlines, which take hours per scan to draw. Separately, Gotkowski et al. showed that a few scribbled lines per scan get a standard 3D nnU-Net within about four Dice points of dense supervision, but only when training and testing stay inside one modality. This proposal asks what happens when both savings are needed at once: the source modality has only scribbles and the target modality has no labels at all.

The idea is to translate each scribbled source scan into many plausible target-modality renderings rather than one. The anatomy and the scribbles stay fixed, and only the target-specific appearance is sampled from a learned distribution. A segmenter trained with a loss restricted to scribbled voxels then sees each scribble under the range of contrasts real target scans show. The same scribbles also anchor the translator's anatomy encoder, so geometry cannot drift away from the thin labels. In two sentences: scribbles are too sparse to survive a translator that distorts geometry or collapses target appearance to a single average, so we pair a geometry-locked, appearance-stochastic translator with partial-loss training. We then test, against cost-matched alternatives, whether the one-to-many sampling is what makes scribble-only cross-modality transfer work.

Core Claim

Cross-modality adaptation from scribble-only source labels can approach dense-label adaptation on cardiac MR→CT and bSSFP→LGE MRI. This is not yet known: the file reports no result for scribble-source cross-modality adaptation, and we found no reported result on Lacuna for it. The proposal makes two specific bets. The first is that the gain over a deterministic translator comes from sampling learned target appearance. Hand-designed intensity curves should not substitute for this, because CT-versus-MR differences and LGE enhancement patterns are not monotone grayscale remappings. The second is that thin scribbles make geometric fidelity the binding constraint. A translator whose anatomy encoder is supervised only through scribbles must still keep boundaries aligned to within a few voxels, or the transferred scribbles label the wrong tissue.

This differs from the closest work along three axes. MT-UDA and Meta-hallucinator attack label scarcity in MR→CT transfer, but they assume a few fully labeled volumes, and their translators are deterministic: a CycleGAN in MT-UDA, learned but non-sampled transformations in Meta-hallucinator. SGDR splits images into an anatomy code and a style code, but it anchors anatomy with a dense source segmentation loss and does not sample styles from a prior. Gotkowski et al. supply the label-efficient training recipe but never cross a modality boundary. We are combining a one-to-many translator with scribble-only anchoring and partial-loss training, and we evaluate the result against cost-matched target annotation. None of these papers does that.

Research Program

The translator follows the content–style design. A shared anatomy encoder maps an image to a spatial "content" map. A style encoder maps it to a low-dimensional appearance code, and a decoder renders content plus style in the target modality. This is the disentangled family used by SGDR and the variational models VAE-NAM and Domain-Specific Variational Information Bound. Style enters through adaptive instance normalization, which rescales feature channels per style code.

The appearance code is trained against a Gaussian prior, so at training time we can draw fresh target appearances for any source content. Two terms keep the decoder from ignoring the sampled code. One reconstructs the code from the generated image, the remedy Augmented CycleGAN uses against latent neglect and hidden-noise "steganography." The other is a diversity term on outputs. We deliberately avoid Augmented CycleGAN's full two-generator, two-encoder, four-discriminator setup, because the file flags its optimization fragility.

Geometry is locked in two ways. A small segmentation head on the content map is trained with partial cross-entropy and Dice loss on scribbled source voxels only, replacing SGDR's dense loss with the Gotkowski objective. A patch-level real/synthetic mixing discriminator, borrowed from SynthMix, penalizes local structural noise that a global discriminator ignores.

The downstream segmenter is a standard 3D nnU-Net with partial loss. For each scribbled source volume we generate $K$ target renderings with independent style samples, and the scribbles are copied unchanged onto every rendering. Unlabeled real target volumes are used only by the translator's discriminators, never by the segmenter. This keeps the segmenter simple, so any gain can be attributed to the training images.

We will also measure whether the samples are trustworthy, not just whether Dice goes up. Diversity is real if sampled renderings cover the spread of real target intensity statistics, measured as per-structure intensity histograms compared to the real target scans. Geometry is preserved if an oracle target segmenter agrees with the source dense masks on the translated images. The oracle is trained on dense target labels and used only for this audit. Dense source masks are likewise held out from training and used only for evaluation.

Baselines

The naive reference point is nnU-Net with partial loss trained on scribbled source scans and applied directly to target scans with no adaptation. It shows how much of the 17.2% source-only collapse (SIFA) persists under scribbles. We also train the same network on scribbled source scans with heavy Bézier-curve intensity augmentation, as in Dual Normalization. That method reports 54.44% Dice on BraTS from a T2 source without target data, though on a different task.

The strongest deterministic competitor is a scribble-adapted pipeline. A CycleGAN (or SynthMix's translator) translates scribbled source scans once, and nnU-Net with partial loss is trained on the result. This is exactly our system with $K=1$ and no latent, so the comparison is clean: same segmenter, same scribbles, same unlabeled target pool, same training epochs and translator hyperparameter search budget. We also re-run MT-UDA and SIFA with their dense source loss swapped for partial loss on scribbles. There is no reported scribble-source result for either, so we produce one.

The dense-label methods serve as reference ceilings. On MM-WHS MR→CT, SynthMix reports 82.5% Dice with 3.8 mm ASSD and SGDR reports 80.18% Dice, both with full dense source labels. With only four densely labeled MR scans, MT-UDA reports 61.6% Dice and 5.9 mm ASD, and Meta-hallucinator reports a further 6.7 Dice points over MT-UDA. On MSCMR bSSFP→LGE, SGDR reports 77.91% Dice. Our method counts as a success if it beats the deterministic scribble pipeline by a margin outside seed-to-seed variance (three seeds, paired by volume) on both tasks. It must also match or beat MT-UDA's 61.6% at four labeled scans while using scribbles on all source scans at comparable or lower drawing cost.

Three ablations isolate the key ingredients. The first removes the scribble-anchored content head. The second keeps stochastic sampling but sets the style code to its prior mean (deterministic-but-same-architecture). The third replaces learned style sampling with Bézier remapping of a single deterministic translation.

Why not just…?

Why not just scribble a few target scans and train nnU-Net with partial loss there? This is the most serious threat. Gotkowski et al. show that scribbles alone reach 0.813 mean Dice against a 0.856 dense bound within a modality, so a handful of target scribbles might beat any adaptation. We do not argue around this; we measure it. We train target-domain nnU-Net with partial loss on 1, 2, 4 and 8 scribbled target volumes and report how many target volumes match our adapted model. If two scribbled target scans suffice, the honest conclusion is that adaptation is only worth it when target annotation is impossible, for example a new scanner with no annotator access. The paper would then be reframed around that regime.

Why not just apply Bézier intensity augmentation on top of one deterministic translation? This gets cheap diversity without a latent space. Its single-source generalization result (54.44% on BraTS) is far below translation-based adaptation, but that comparison is across tasks, so it is not decisive. The third ablation answers it directly. If Bézier-on-CycleGAN matches learned sampling, the stochastic translator is unnecessary, and we will report that.

Why not translate target scans back to the source modality at test time, LMIDiffusion style? LMIDiffusion reports 0.88 Dice versus CycleGAN's 0.85 on IXI PDw→T1w. That is a much smaller physics gap than MR→CT, and it pays hundreds of denoising steps per scan. We include target→source translation with our translator run in reverse as a baseline. MR→CT is precisely where the file notes rigid mutual-information guidance is untested.

Proposed Experiments

The main experiment runs on MM-WHS 2017 MR→CT (four structures) and MSCMR bSSFP→LGE. Scribbles on MR come from ScribbleBench's interior and border scribble generator applied to the dense masks; MSCMR is already part of ScribbleBench. Metrics are Dice and ASSD per structure, with ASSD reported separately because thin-label methods tend to fail at boundaries first.

The first decisive check is the $K$-sweep ($K = 1, 2, 4, 8, 16$). If Dice saturates at $K=1$, the one-to-many claim fails. The second is the diversity audit: learned samples should cover real target intensity spread better than Bézier remaps, and gains should concentrate on structures with high real-target appearance variance, such as myocardium in LGE. The third is the geometry audit with the oracle segmenter. Removing the scribble-anchored content head should visibly worsen oracle–mask agreement and ASSD, which would confirm that sparse labels make geometric drift the dominant error.

The fourth check is the cost-matched curve against target-domain scribbling described above. We use labeled-voxel fraction as the annotation-cost proxy, since the file reports no timing numbers, and we state that limitation plainly.