Doctors rarely have time to trace tumor outlines voxel by voxel, so a practical alternative is to train segmentation networks from scribbles: a few quick strokes marking "definitely tumor" and "definitely healthy." This works surprisingly well overall. A plain 3D nnU-Net (a self-configuring U-Net pipeline) that computes its loss only on scribbled voxels comes within about four Dice points of full supervision. The remaining error sits where scribbles say nothing and image contrast says little: the blurry margin where an infiltrating tumor fades into normal tissue. Intensity-based tools such as conditional random fields cannot settle this band, because they rely on sharp intensity edges that low-contrast margins do not have.
We propose to settle the band with a different question: can this tissue be predicted from its surroundings by a model that only knows what healthy tissue looks like? We train a healthy-tissue inpainting model (a generator that fills a masked-out region with plausible healthy anatomy given the surrounding scan) using only regions the annotator already scribbled as healthy. We then probe small balls in the unlabeled band. If masking out a ball and asking the model to fill it in reproduces what is actually there, the tissue behaves like healthy tissue. If the reconstruction fails beyond what the model's own sampling spread would explain, it does not. These per-voxel predictability scores are calibrated against the scribbles in the same scan and become soft targets for the boundary band. The output is still an ordinary nnU-Net, with no extra annotation, and the generative model is used only during training.
Core Claim
Healthy-context predictability is a boundary signal that does not depend on intensity gradients, and it is available at no extra annotation cost under scribble supervision. Adding it as soft supervision on the unlabeled band will narrow the boundary error (95th-percentile Hausdorff distance and boundary Dice) on low-contrast lesion tasks. Specifically, we expect gains on brain tumor, kidney tumor and liver tumor, and no gain on high-contrast cardiac structures. The gain should exceed what matched extra scribbling time or non-generative predictability scores provide.
Relation to Prior Work
Two papers motivate the proposal by locating the gap. Embarrassingly Simple Scribble Supervision for 3D Medical Segmentation shows that nnU-Net with a partial loss reaches 0.813 mean Dice on ScribbleBench against 0.856 for dense labels, and that low-contrast tumor tasks account for most of the residual gap. Learning to Segment Medical Images with Scribble-Supervision Alone propagates scribbles with random walks and CRFs; it degrades only slightly on cardiac MRI but depends on edge-snapping that fails without contrast.
The closest work, which this proposal must clearly differ from, derives masks by "healing" an image and subtracting. Pathology Segmentation using Distributional Differences to Images of Healthy Origin uses a CycleGAN-style generator that emits a label map. COIN learns a perturbation that flips a disease classifier and thresholds the difference. Both use image-level labels and a single global counterfactual. Their boundaries are blurry: the first reaches 72.9% Dice versus 86.3% fully supervised on BraTS 2017 and reports Hausdorff outliers at low-contrast transitions, and the second inherits the classifier's attribution blur. Our signal differs in three ways. It is local, asking about one small ball given its real context rather than producing a whole healed image. It is calibrated against the annotator's own scribbles in each volume rather than thresholded globally. And it supervises only the band scribbles leave undetermined, so it refines a strong segmenter rather than replacing it.
Generator Versus Segmentor and Adversarial Pseudo Healthy Synthesis Needs Pathology Factorization constrain the design. The first needs dense masks to keep its segmentor from collapsing. The second shows unpaired translation hides pathology and distorts anatomy, scoring identity 0.80 and healthiness 0.83 against 0.88 and 0.99 with paired masks. We therefore avoid adversarial whole-image translation entirely and train the inpainter only on voids whose contents are known to be healthy.
The method comes from the diffusion inpainting line: Denoising Diffusion Models for 3D Healthy Brain Tissue Inpainting, the BraTS 2023 inpainting challenge, and fastWDM3D, which makes many 3D inpaintings affordable. We deliberately do not use the hemispheric mirroring of BrainNormalizer, because it cannot transfer to kidney or liver.
Boundary architectures such as reverse attention (Yu et al., MISNet) and edge branches (DB-MSMUNet) all need dense contour targets. We supply a target rather than an architecture.
Research Program
The inpainter is a 3D wavelet-diffusion model built on the fastWDM3D recipe. During training, void masks are sampled only inside background-scribbled regions, so every training target is healthy tissue, while the context may contain tumor, as it will at test time. A second variant pretrains on BraTS 2023 inpainting data, which has healthy voids, and then fine-tunes on the scribble-derived voids. This lets us separate "healthy prior from the target dataset's scribbles" from "healthy prior from external data."
For each training volume, we first train nnU-Net with the partial loss on scribbles. The band is the set of voxels where this model's softmax lies between 0.1 and 0.9, or which lie within a fixed distance of its predicted surface. At sampled band voxels $v$ we mask a ball $B(v)$ of radius $r$, draw $K$ inpaintings $\hat{x}^{(k)}_{B(v)}$, and compute a predictability score
$$ s(v) = \frac{\lVert x_{B(v)} - \bar{x}_{B(v)} \rVert}{\sigma_{B(v)} + \epsilon}. $$
Here $x_{B(v)}$ is the observed intensity in the ball, $\bar{x}_{B(v)}$ is the mean of the $K$ samples, $\sigma_{B(v)}$ is their spread, and $\epsilon$ is a small constant. Dividing by the spread keeps naturally variable healthy tissue, such as cortical folds or bowel, from being flagged.
We compute $s$ on scribbled voxels too. On those voxels we fit a per-volume logistic map $q(v)=\sigma((s(v)-\tau)/T)$, choosing $\tau$ and $T$ to separate foreground from background scribbles. Volumes whose scribbled voxels are not separable, with AUROC below a set threshold, contribute no band targets, which is an explicit abstain rule. The segmenter is then retrained with
$$ L = L_{\text{pL}} + \lambda \sum_{v \in \text{band}} c(v)\, \mathrm{BCE}\big(p(v), q(v)\big), $$
where $p(v)$ is the network's foreground probability. The confidence $c(v)=|2q(v)-1|$ down-weights voxels the score cannot decide. The score is computed once offline, with a single refresh after the retrained model redraws the band. Inference cost is unchanged.
Baselines
The naive reference is nnU-Net trained with the partial loss on scribbles alone, run exactly as in Embarrassingly Simple Scribble Supervision on its ScribbleBench scribbles. It is also the strongest existing method: it reports 0.813 mean Dice across seven tasks against 0.856 for dense labels, and it beats ShapePU (0.369) and CycleMix (0.559). Per-task Dice for BraTS2020, KiTS2023 or LiTS under partial loss is not in our file (no reported result found on Lacuna for per-task ScribbleBench values), so we rerun it ourselves with identical folds, scribbles, preprocessing and epochs. Beating it means a statistically significant reduction in HD95 and an increase in boundary Dice on at least two of the three tumor tasks, with no loss of volumetric Dice. Dense nnU-Net is reported as the ceiling.
A self-training baseline gives the same training schedule a non-generative band target: pseudo-labels from Monte Carlo dropout thresholded on the band, as in Can et al.. This tests whether any second-round band supervision helps, regardless of where it comes from. The CRF baseline applies dense CRF band targets with hyperparameters tuned on the same validation scribbles.
A global-counterfactual baseline replaces our local score with a COIN-style difference map. A classifier is trained on scribble-derived patch labels, and its counterfactual is calibrated and injected through the same loss. This isolates "local probing" from "generative healing" in general.
The key ablations hold everything fixed except the scorer. In one, the diffusion inpainter is replaced by a non-generative predictor: a masked autoencoder trained on the same healthy voids, and an even simpler local-statistics predictor that fills the ball with neighborhood mean and variance. If the simple predictor matches the diffusion model, the generative prior is not what matters. Other ablations remove the normalization by $\sigma$, remove the per-volume calibration, and vary probe radius $r$.
Every method gets the same GPU budget for the segmenter. The offline scoring compute is reported separately.
Why not just…?
Why not just use nnU-Net with the partial loss and stop? It is the real threat. It already closes most of the gap, and our method may add little on tasks where its convolutional prior suffices. The direct comparison above settles this. We also include ACDC as a high-contrast control, where we predict no gain. A gain there would suggest we are adding generic regularization rather than resolving ambiguity.
Why not just snap the boundary with a CRF or random walk? Agarwal et al. found that dense CRFs on CT raised precision but sharply cut recall. Can et al. report strong hyperparameter sensitivity. Both effects follow from relying on intensity edges that infiltrative margins lack. Our CRF baseline uses the same band and loss, so any difference reflects the signal itself.
Why not just spend the effort on more scribbles near the boundary? This is the most honest competitor. We simulate boundary-focused scribbles drawn from dense labels at a matched annotation cost, estimated from stroke length, and train partial-loss nnU-Net on them. If two extra boundary strokes per volume match our gain, the method is not worth its compute, and we will report it that way.
Why not just heal the whole image and subtract? CycleGAN subtraction reached 72.9% Dice with Hausdorff outliers in Andermatt et al.. Xia et al. show that unpaired translation hides pathology. The COIN-style baseline above tests the strongest version of this alternative under our calibration.
Proposed Experiments
The main study runs on the ScribbleBench tumor tasks BraTS2020, KiTS2023 and LiTS, with ACDC as a high-contrast control, using five-fold cross-validation and the published scribbles. We report Dice, HD95, and normalized surface Dice at 1–2 mm, computed both overall and restricted to the dense-label boundary band.
The first decisive check is mechanistic. On dense-labeled validation volumes, we measure the AUROC of $s(v)$ for tumor versus healthy within the band, and compare it against network softmax, MC-dropout uncertainty and local intensity contrast. If $s$ is not more discriminative than these in the band, the proposal fails at its premise, before any segmentation training. The second check is the scorer ablation: diffusion versus masked autoencoder versus local statistics. The third is the matched-annotation-cost comparison.
A secondary test addresses the mask dependency of pseudo-healthy inpainting. We feed our predicted masks, the partial-loss masks, and ground-truth masks to the BraTS 2023 inpainting pipeline. The challenge's top median SSIM of 0.91 was obtained with ground-truth voids. Here we measure how much of that quality survives when the void is estimated from scribbles.
References
- Embarrassingly Simple Scribble Supervision for 3D Medical Segmentation
- Learning to Segment Medical Images with Scribble-Supervision Alone
- Pathology Segmentation Using Distributional Differences to Images of Healthy Origin
- COIN: Counterfactual Inpainting for Weakly Supervised Semantic Segmentation for Medical Images
- Generator Versus Segmentor: Pseudo-healthy Synthesis
- Adversarial Pseudo Healthy Synthesis Needs Pathology Factorization
- Denoising Diffusion Models for 3D Healthy Brain Tissue Inpainting
- The Brain Tumor Segmentation (BraTS) Challenge 2023: Local Synthesis of Healthy Brain Tissue via Inpainting
- fastWDM3D: Fast and Accurate 3D Healthy Tissue Inpainting
- BrainNormalizer: Anatomy-Informed Pseudo-Healthy Brain Reconstruction from Tumor MRI via Edge-Guided ControlNet
- Parallel Res2Net-based Network with ReverseAttention for Polyp Segmentation
- Multi-scale information sharing and selection network with boundary attention for polyp segmentation
- DB-MSMUNet: Dual Branch Multi-Scale Mamba UNet for Pancreatic CT Scans Segmentation
- Weakly Supervised Lesion Co-Segmentation on Ct Scans