Training a model to outline organs or tumors in 3D scans normally requires a specialist to trace every boundary, which takes hours per scan. Two cheaper substitutes exist. The first is scribbles: a few strokes drawn through the inside of each structure. The second is unlabeled scans, which every hospital has in bulk and which can train a network through self-supervised pretraining, meaning the model learns from a made-up task such as filling in hidden parts of an image. The two have been studied separately, and both have mostly been tested on scans from the same hospital the training data came from. That is the wrong test. Models trained at one site lose more than 20% of their accuracy on MRI from scanners they have not seen (AMOS).
This note proposes a specific claim about how the two signals fit together. Scribbles avoid edges, so a scribble-trained model never sees where a structure ends. Scanner shift mostly changes intensity and contrast, which is exactly what defines those edges. Unlabeled scans from the deployment hospital contain both missing pieces. We predict that self-supervised pretraining on a pool of unlabeled scans that includes the target scanner pays off more for scribble-trained models than for densely trained ones. We also predict the payoff will be larger on unseen sites than on the training site, and will show up mainly in boundary error rather than overlap. If that is true, a hospital could deploy a segmentation model with scribbles drawn at one site, a handful of scribbled scans at its own site, and its own archive of unlabeled scans, with no dense masks anywhere.
Core Claim
Let $G(s, d)$ be the improvement in Dice from replacing random initialization with self-supervised initialization. Here $s$ is the supervision type (scribble or dense) and $d$ is the test set (in-domain or a held-out scanner). Dice is the standard overlap score between predicted and true masks. The claim is a difference-in-differences:
$$ \big[G(\text{scribble}, \text{OOD}) - G(\text{scribble}, \text{ID})\big] \;>\; \big[G(\text{dense}, \text{OOD}) - G(\text{dense}, \text{ID})\big] \;>\; 0, $$
where OOD means out-of-distribution (an unseen scanner or site) and ID means in-domain. We expect the same ordering, more strongly, for 95th-percentile Hausdorff distance (95HD), the near-worst boundary error. The claim is falsifiable in a simple way. If pretraining gives scribble models and dense models the same gain, or gives no extra gain off-site, then pretraining is just a generic initialization trick. In that case the pairing with weak labels has no special value.
The second, more practical claim isolates where the gain comes from. Including the unlabeled target-site scans in the pretraining pool should matter more than any choice of pretext task. Pretraining on source-site scans alone should recover only part of the off-site gain.
Why this gap is open
The closest work fixes one or two of these constraints and leaves the rest in place. Scribble-based Domain Adaptation via Co-segmentation adapts vestibular schwannoma segmentation from contrast-enhanced T1 to T2 MRI. It uses scribbles on 30 target scans and reaches 83.4% Dice, above a target model trained on dense masks (81.6%). It motivates our setting: a few target scribbles are worth more than they look. It also depends on 150 densely annotated source volumes and uses no self-supervision. We remove the dense source and replace it with scribbles plus pretraining.
Embarrassingly Simple Scribble Supervision for 3D Medical Segmentation supplies the method we build on. Its nnU-Net with a partial loss (a loss computed only on scribbled voxels) reaches a mean Dice of 0.813 across seven 3D tasks, against 0.856 with dense masks. It also constrains our design. Specialized scribble methods tuned on cardiac data collapse elsewhere (0.05 Dice on brain tumors, 0.36 on abdominal CT), so we will not build on cardiac-specific architectures such as ScribFormer. ScribbleBench trains from scratch and tests only in-domain, which are the two variables we change.
Unsupervised and source-free adaptation, such as ProtoContra-SFDA, aligns unlabeled target features to a source model trained on dense masks. It reaches 86.1% Dice from MRI to CT. Single-source generalization, such as Domain Game, limits the cross-site drop on prostate MRI to about 4.7%, again with dense source labels. Both show that off-site robustness can be engineered. Neither asks what happens when the source labels are themselves weak.
On the pretraining side, three results shape the choice of pretext task:
- Pathology features learned in-domain beat ImageNet features (Wang et al.).
- Domain-specific self-supervision stays stable with 1% labels (Li et al.).
- Brightness and contrast distortion is the best augmentation for grayscale radiographs (Van der Sluijs et al.).
MixSSL warns that contrastive learning degrades when different patients' scans look alike. That favors a reconstruction-style pretext task over a pure contrastive one, at higher compute cost.
Research Program
The system has three parts. Every part uses existing components; what is new is the experimental design. First, we pretrain a 3D nnU-Net encoder-decoder with a masked-reconstruction pretext task: hide random 3D blocks and reconstruct them. This follows the inpainting family represented by paired cell inpainting. Input-side brightness and contrast distortion forces the reconstruction to rely on anatomy rather than raw intensity. The pretraining pool is all unlabeled scans from every site in the benchmark, including the held-out target scanners. This is legitimate because a deploying hospital has its own unlabeled archive.
Second, we fine-tune with the ScribbleBench partial loss on scribbles from the source site. Optionally we add scribbles on a small number $k$ of target-site scans, with $k \in \{0, 2, 5\}$. Third, we test a "keep the pretext alive" variant that continues the reconstruction loss on unlabeled target scans during fine-tuning. This addresses the concern from the small-sample literature that fine-tuning on a few labels makes the model forget general features and overfit to quirks of the labeled scans.
We vary three factors: initialization (random, ImageNet-style, source-only pretraining, pooled pretraining), supervision (dense or scribble, at 10, 20, and 50 source volumes), and $k$. We report Dice and 95HD both in-domain and on held-out sites. Scribbles are synthetic, generated as in ScribbleBench, so supervision is controlled exactly. We treat scribbled-voxel fraction as the annotation-cost proxy, because the file gives no clinician timing data.
Baselines
The naive reference point is nnU-Net with partial loss trained from random initialization on source-site scribbles, which is exactly the ScribbleBench recipe. Its reported in-domain result is a mean Dice of 0.813 across seven tasks, against 0.856 for dense supervision (ScribbleBench). We found no reported result on Lacuna for this baseline on held-out scanners. Our first deliverable is therefore that missing number.
The strongest augmentation-only competitor is the same recipe with heavy intensity, contrast, and random-convolution augmentation. It gets the same number of training steps and the same hyperparameter search budget as our method. This is the most serious threat, because intensity robustness is exactly what pretraining is supposed to add.
The strongest existing weak-target adaptation method is co-segmentation domain adaptation (Scribble-based DA), whose reported figure is 83.4% Dice on the vestibular schwannoma T1-c to T2 task. We run it two ways. The first gives it its dense source masks, as an upper reference that uses far more labels. The second replaces the dense source with scribbles, as a matched comparison. Beating the matched version, and coming within its error of the dense-source version while using no dense masks at all, would count as success.
For unlabeled-target adaptation we run ProtoContra-SFDA (reported 86.1% Dice on CHAOS MRI to MICCAI 2015 CT) on top of a scribble-trained source model. We also run iterative pseudo-label self-training on target scans with uncertainty filtering, following Can et al.. The dense-supervision model is the ceiling.
The ablation that isolates the key ingredient compares source-only pretraining against pooled pretraining, holding the number of pretraining scans fixed by subsampling. Total pretraining plus fine-tuning compute is matched across arms. The augmentation-only arm gets the pretraining compute as extra supervised epochs.
Why not just…?
Why not just use heavy intensity augmentation instead of pretraining? This is a real threat. Van der Sluijs et al. show that intensity distortion carries much of the benefit in radiographs, and our pretext task uses the same distortions. Augmentation only simulates shift, though. Domain Game's authors note that hand-designed augmentations miss shifts driven by contrast dynamics or susceptibility effects (Domain Game), whereas pooled pretraining sees real target scans. The compute-matched augmentation arm settles it. If that arm matches pooled pretraining off-site, we will report that the pretext task adds nothing beyond augmentation and drop the claim.
Why not just self-train on unlabeled target scans with pseudo-labels? Pseudo-labels from a scribble model inherit its boundary errors, and iterative pipelines drift when early labels are systematically wrong (Can et al.). Pretraining never consumes the model's own predictions. The pseudo-label baseline and the combination of both methods are both run. If self-training alone closes the 95HD gap, pretraining is redundant.
Why not just densely annotate two target scans? Co-segmentation domain adaptation already found that 30 scribbled target scans beat dense target training (Scribble-based DA). ScribbleBench puts scribbles within about 4 Dice points of dense masks. We include a cost-matched arm that trades target scribbles for one or two densely annotated target scans at equal labeled-voxel budget. This arm tests directly whether scribble breadth beats mask depth.
Why not just use a large general or frozen foundation encoder? The file's evidence points against generic features: Wang et al. report the in-domain advantage, and Dai et al. find that alignment with medical terminology matters more than parameter scale. SpurCon's reliance on a frozen BiomedCLIP is noted as unable to adapt to unseen modalities such as specialized MRI sequences (SpurCon). The ImageNet-initialized arm stands in for this alternative.
Proposed Experiments
The primary testbed is AMOS: 500 CT and 100 MRI scans from two centers and eight scanners. It reports 89.04% mean Dice for a 3D U-Net on CT and a drop of more than 20% on MRI from unseen scanners. We hold out scanners in leave-one-scanner-out splits, with the held-out scanner's images (not labels) in the pretraining pool. For a second anatomy with a known site gap, we use the six-institution prostate MRI setting from Domain Game.
For a tumor target, we use BraTS, including BraTS-Africa as the resource-limited held-out cohort. ScribbleBench shows this is where specialized scribble methods collapse, and Domain Game reports a 10.5% Dice gain there with dense labels. The vestibular schwannoma T1-c to T2 task allows direct comparison to the 83.4% co-segmentation figure. ACDC is used only as a sanity check, given the documented cardiac overfitting.
Three outcomes decide the main question. The first is the sign and size of the difference-in-differences in the Core Claim, with confidence intervals from repeated splits. The second is whether the pooled-versus-source-only ablation accounts for most of the off-site gain. The third is whether 95HD improves more than Dice under scribble supervision.
A secondary analysis asks how the number of target scribbled scans $k$ trades against pretraining. If pooled pretraining with $k = 0$ matches random initialization with $k = 5$, the unlabeled archive substitutes for target annotation. That is the practical result a hospital needs. A negative result on any of the three primary checks is publishable as well, since no reported result exists for scribble-trained 3D models under scanner shift.
References
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Scribble-Based Domain Adaptation via Co-segmentation
- Embarrassingly Simple Scribble Supervision for 3D Medical Segmentation
- ScribFormer: Transformer Makes CNN Work Better for Scribble-based Medical Image Segmentation
- Source-Free Domain Adaptation for Medical Image Segmentation via Prototype-Anchored Feature Alignment and Contrastive Learning
- Domain Game
- Weakly Supervised Prostate Tma Classification Via Graph Convolutional Networks
- Boosting automatic COVID-19 detection performance with self-supervised learning and batch knowledge ensembling
- Exploring Image Augmentations for Siamese Representation Learning with Chest X-Rays
- Mix-Up Self-Supervised Learning for Contrast-Agnostic Applications
- Learning unsupervised feature representations for single cell microscopy images with paired cell inpainting
- Learning to Segment Medical Images with Scribble-Supervision Alone
- Pathologyvlm: a large vision-language model for pathology image understanding
- SpurCon: Weighted Supervised Contrastive Learning for Mitigating Spurious Cues in Medical Imaging