Generalizing to Unseen Scanners from Scribbles Alone

Segmentation networks for medical images are usually trained on scans from one hospital, with every organ traced by an expert. When they are moved to a different scanner or imaging modality, they often fail badly. The best current defense is to train on one site while randomly changing image appearance, so the network learns to ignore scanner style. The strongest versions of this trick change each organ's brightness independently, and to do that they need a complete outline of every organ in every training scan. That outline is exactly what hospitals cannot afford to produce. Cheaper training methods learn from scribbles, a few quick strokes drawn inside each structure, but they have only ever been tested on scans from the same scanner they were trained on.

This note proposes to test whether per-organ appearance augmentation can be run when organ boundaries are only roughly known. The idea is to replace the hard organ masks with soft, smoothly varying region weights. These weights come from the scribbles and from the network's own current predictions, and they fade to neutral in the uncertain zone around organ boundaries, so the augmentation never creates a sharp artificial edge. The concrete question is simple: trained only on scribbles from one domain, can a network reach the cross-modality accuracy that dense-mask methods report, and is the per-organ part of the augmentation actually what gets it there?

Core Claim

Two lines of work have never been joined. Single-source domain generalization, meaning training on one site and testing on unseen sites with no target data, has three main augmentation methods. SLAug remaps global intensity with random Bézier curves and also shifts each organ's intensity separately using ground-truth masks. StyCona splits images by singular value decomposition into "style" (the singular values) and "content" (the rank-one components) and mixes them across images. FreeSDG mixes low- and high-frequency image bands and adds a reconstruction pretext task. All three assume dense masks on the training domain.

The scribble methods make the opposite assumption. CycleMix combines mix augmentation with consistency losses. The nnU-Net partial-loss baseline of Gotkowski et al. computes the loss only on scribbled voxels and ignores the rest. Both are evaluated only on test data from the training distribution.

The closest bridge is scribble-based domain adaptation via co-segmentation. It still needs 150 densely labeled source volumes, and it uses target-domain images during training, so it adapts to a known target rather than generalizing to an unseen one.

Our claim is that SLAug's mask-dependent component, its per-organ location-scale shift, can be rewritten to depend only on a soft partition of the image. Under scribble supervision, that soft version should recover most of the cross-domain benefit the component gives under dense supervision. The claim fails if per-organ augmentation adds nothing beyond global augmentation once training uses scribbles. The experiments are built to detect that outcome directly.

The Mechanism

Let $x$ be a source image and let $g$ be SLAug's random global intensity curve. For each class $k$, including background, draw a random scale $a_k$ and offset $b_k$. Let $w_k(x)\ge 0$, with $\sum_k w_k = 1$ at every voxel, be a soft weight saying how much each voxel belongs to class $k$. The augmented image is

$$\tilde{x} = \sum_k w_k \odot \big(a_k\, g(x) + b_k\big).$$

When $w_k$ is a hard ground-truth mask, this reduces to SLAug's local augmentation. When $w_k$ is smooth, the intensity shift changes gradually across uncertain boundary zones instead of jumping.

The file flags the main risk explicitly. Applying a per-class shift along a one-voxel-wide scribble, or along a wrong boundary, plants a fake intensity edge inside an organ. The network would then learn that fake edge as if it were real anatomy.

The weights are built in two phases to limit this risk. Early in training, before the network's predictions can be trusted, $w_k$ comes from random-walk seed expansion of the scribbles, the same propagation that Can et al. used for relabeling. This is followed by a Gaussian smoothing whose width is a tuned hyperparameter. Later, $w_k$ switches to the temperature-softened posterior of an exponential-moving-average teacher network. Scribbled voxels are clamped to weight one, and voxels where the teacher is uncertain (measured by Monte Carlo dropout variance, again following Can et al.) are pushed toward uniform weights. A uniform weight cancels the per-class shift at that voxel and leaves only the global curve.

The segmentation loss remains the plain partial loss of Gotkowski et al.. The soft regions shape only the augmentation, never the labels. This keeps the method from becoming hidden pseudo-label training, a distinction the ablations below test directly. SLAug's gradient-saliency fusion is included only as an optional component, because the file notes its per-iteration computational cost.

Research Program

The work has three parts.

First, we build scribble versions of two standard cross-domain benchmarks. MS-CMR cardiac MRI (bSSFP to LGE and LGE to bSSFP) already comes with scribbles, as used by CycleMix. For abdominal CT-to-MRI we use AMOS, which has 500 CT and 100 MRI scans from eight scanners and a reported drop of more than 20% mDice on unseen MRI scanners (AMOS). We generate scribbles for AMOS with the ScribbleBench protocol, so they match the annotation density Gotkowski et al. used. We also train on the SLAug CT-to-MRI organ subset (liver, kidneys, spleen) so results can be compared with published dense-mask numbers.

Second, we run a controlled grid that crosses three factors. Supervision is either dense masks or scribbles. Augmentation is nnU-Net default, global-only, oracle per-organ, pseudo-label per-organ, or soft-region per-organ. Evaluation is in-domain or cross-domain.

Third, we measure calibration under shift (Gap 4 in the file). Nobody has characterized what partial-loss training does to model confidence on unseen scanners, and Rao showed that augmentations that delete or misattribute anatomy distort calibration. Fake edges from bad region estimates are a plausible route to the same failure.

Baselines

The naive reference point is nnU-Net trained with partial loss on scribbles, using its default augmentation. This is the method from Embarrassingly Simple Scribble Supervision, which reports a mean Dice of 0.813 across ScribbleBench versus 0.856 for dense supervision. It is the strongest scribble method in the file and beats CycleMix (0.559) and ShapePU (0.369) by wide margins. However, all of those numbers are in-domain, and no reported result was found on Lacuna for any scribble-trained model tested on an unseen scanner or modality. We run it ourselves with the same backbone, training length, and preprocessing as every other arm.

The strongest existing domain-generalization method is SLAug trained with dense masks. It reports an average Dice of 88.63% on abdominal CT to MRI, against 86.31% for the previous best method (CSDG) and a 90.85% supervised upper bound. It reports 87.67% Dice on cardiac LGE to bSSFP (SLAug). StyCona reports 73.39% DSC for bSSFP to LGE and 81.59% for LGE to bSSFP (StyCona).

The dense-mask SLAug run is the ceiling we are trying to approach with about 1% of the labeled voxels. We reproduce it inside our own pipeline rather than relying on the published figures, because backbone differences alone could explain gaps of a few Dice points.

We define success concretely. On CT to MRI, the soft-region method trained on scribbles should close at least half of the gap between the scribble nnU-Net baseline and our reproduced dense SLAug. It should also beat the best scribble-trained global-only arm by a margin larger than the standard deviation across three seeds.

Two ablations isolate the key ingredient. The first is oracle per-organ augmentation: the per-class shift uses dense ground-truth masks while the loss still sees only scribbles. This measures the most that region-aware augmentation could ever add under scribble supervision. The second sets the soft weights to hard argmax masks from the same teacher. This tests whether smoothness, rather than region estimation itself, is what prevents fake-edge damage.

All arms share one hyperparameter search budget, tuned on a held-out portion of the source domain only. No target-domain data is used for model selection.

Why not just…?

Why not just use nnU-Net with partial loss, since it already comes within 4 Dice points of dense supervision? That gap was measured in-domain, and AMOS shows that even densely trained 3D U-Nets lose more than 20% mDice on unseen MRI scanners. The pL arm in our grid answers this directly. If it already matches dense SLAug across domains, the proposal has no reason to exist, and we would report that.

Why not just apply global appearance augmentation, which needs no masks at all? This is the most serious threat. SLAug's global Bézier curves, StyCona's style mixing, and FreeSDG's frequency mixing (FreeSDG) all run without any labels. FreeSDG's ablation, which drops from 0.795 to 0.720 Dice on LES-AV, shows that its full pipeline matters, but it says nothing about per-organ augmentation. The file contains no evidence that the local component is necessary.

The oracle ablation settles the question. If oracle per-organ augmentation does not beat global-only augmentation under scribble supervision, then the soft-region method has nothing to recover. In that case we would reframe the contribution as an empirical result that scribble-trained domain generalization only needs global augmentation.

Why not pseudo-label the scribbles first and then run ordinary SLAug on the pseudo-masks? This two-stage pipeline is simple and may well be competitive. Its weakness is that it commits to hard boundaries from a first-stage model, so the per-organ shift places sharp intensity steps exactly where that model is wrong. Our pseudo-label arm and our hard-argmax ablation test this. If they match the soft version, the smoothing adds nothing and the simpler pipeline should be preferred.

Why not prompt a foundation segmentation model instead? The adapted Segment Anything Model (SAM) reaches 79.54% average Dice across unseen prostate sites (SAM for single-source generalization). However, it needed decoder fine-tuning on dense medical masks, and it depends on a coarse upstream network to produce its box prompts. Under scribble training and domain shift, that upstream network is the very component that degrades. We include a scribble-trained version of this auto-prompting pipeline on the MS-CMR cardiac task as a secondary comparison.

Why not synthesize new cross-domain scans with a scribble-conditioned diffusion model? ScribbleGen gained 0.8 mIoU on natural images, needed a realism curriculum to avoid hurting performance in low-data regimes, and was never tested on medical images or under domain shift. Intensity remapping cannot invent anatomy, whereas a generator can. Calibrated intensities matter for diagnosis (Azizi et al.), which makes invented anatomy a real risk. For these reasons we treat diffusion synthesis as out of scope rather than as a competitor we can fairly tune.

Decisive Checks

Each benchmark direction (AMOS CT to MRI, SLAug CT to MRI, bSSFP to LGE, LGE to bSSFP) is reported with Dice and 95th-percentile Hausdorff distance on the target domain, averaged over three seeds. On AMOS, voxel-wise expected calibration error is also reported on the target domain.

The proposal rests on three outcomes. First, oracle per-organ augmentation beats global-only augmentation under scribble supervision. Second, the soft-region method recovers most of the oracle's gain without any dense masks. Third, hard-argmax and two-stage pseudo-label variants do measurably worse, especially on calibration and boundary distance, which is where fake edges should show up.

A further check addresses whether the method generalizes beyond its development settings. We run the in-domain ScribbleBench tasks, including BraTS and KiTS, to confirm the method does not collapse outside the cardiac setting the way CycleMix did (Gotkowski et al.).