Learning Where Anatomical Edges Are to Fill in Scribble Labels in 3D

Training a 3D medical segmentation model normally needs an expert to outline every organ or tumor boundary voxel by voxel. A cheaper option is scribbles: a few quick strokes inside each structure and in the background. The strongest current recipe for scribbles is deliberately plain. It trains the standard nnU-Net pipeline and computes the loss only on the scribbled voxels, a "partial loss." On ScribbleBench, a seven-task benchmark, it reaches a mean Dice of 0.813 against 0.856 for full dense labels (Embarrassingly Simple Scribble Supervision for 3D Medical Segmentation). The remaining gap is concentrated where scribbles carry no information at all, which is the exact location of the boundary between two structures. That is the one thing an annotator skips when drawing strokes instead of outlines.

Image restoration research has a long-standing answer to "where are the boundaries when the data doesn't say." Reconstruction models that recover a structural skeleton first (edges, gradients, edge-preserving smoothed images) and fill texture second generalize better, because boundaries are more stable across scenes and degradations than raw intensities. We propose to borrow this idea in a specific way. A 3D network learns, from the unlabeled scans alone, to predict the edge structure of a clean volume from a heavily degraded copy of it. The edges it can predict reliably are treated as anatomical. The edges it cannot predict are treated as noise or texture. This learned edge map then decides where the scribble-trained segmenter may change its label across the roughly 95% of voxels that nobody annotated.

Core Claim

Scribble supervision fails at boundaries because partial-loss training gives unlabeled voxels no signal. Earlier fixes injected edge information from outside the network. One used a hand-tuned post-hoc CRF, a conditional random field that smooths labels according to image intensities (Learning to Segment Medical Images with Scribble-Supervision Alone). Another used a frozen natural-image edge detector (Boundary Perception Guidance: A Scribble-Supervised Semantic Segmentation Approach). Both sources break on low-contrast CT and MRI.

Our claim is that an edge map learned by degradation-robust reconstruction on the target scans themselves supplies the missing signal. That map is coupled to the segmenter as a training loss, not a post-processing step. It should close a substantial part of the 4.3-point ScribbleBench gap. It should also generalize across all seven tasks with a single fixed hyperparameter setting, rather than overfitting to cardiac MRI the way specialized scribble methods do (Revisiting 3D Medical Scribble Supervision: Benchmarking Beyond Cardiac Segmentation).

What would be new is the combination of three things: a learned structure prior in place of an external detector, trained from reconstruction on the same unlabeled volumes; reliability gating that comes from the reconstruction itself; and evaluation across ScribbleBench rather than one anatomy.

Research Program

Structure head. The backbone is the nnU-Net partial-loss model, unchanged. We add a second decoder head, the structure head. For each training volume we build a degraded copy using noise, anisotropic downsampling and random masked blocks. The head is trained to predict the gradient magnitude of an edge-preserving smoothed version of the clean volume. The target follows StructureFlow's argument that edge-preserved smooth images carry richer structural context than sparse binary edges (Ren et al.). The continuous-gradient choice follows Hou et al., who avoid thresholded binary edges.

The rationale is that a network can only reproduce an edge from a degraded input if the edge is supported by surrounding anatomical context. Scanner noise and fine texture are not recoverable from such an input and are averaged away. We train two heads with different degradation seeds and use their disagreement as a per-voxel reliability score $r_v$ between 0 and 1. This mirrors adaptive conditioning that down-weights edge reliance where edges are untrustworthy (Hong et al.).

Coupling loss. The structure prior enters segmentation training through one loss over pairs of neighboring unlabeled voxels:

$$ \mathcal{L}_{\text{couple}} = \sum_{(u,v)} r_{uv}\, \exp\!\left(-\frac{e_{uv}}{\sigma}\right) \lVert p_u - p_v \rVert^2 \;-\; \lambda \sum_v r_v\, e_v\, \max_c \lVert \nabla p_{v,c} \rVert . $$

The symbols are defined as follows. $p_v$ is the segmenter's class-probability vector at voxel $v$. $e_{uv}$ is the predicted edge strength between neighbors $u$ and $v$, and $e_v$ is the predicted edge strength at voxel $v$. $r_{uv}$ is the minimum reliability of the two voxels. $\sigma$ sets how strongly an edge blocks label propagation. $\lambda$ weights the second term.

The first term spreads scribble labels freely across regions with no edge and stops at learned edges. The second term rewards label transitions that fall on reliable edges. It is written so that the gradient flows only through the dominant class channel at each voxel. This is a 3D port of Boundary Perception Guidance's max-pooling gradient routing, which keeps background edges from pulling on every class. The structure head receives no gradient from the segmentation loss, so segmentation errors cannot reshape the edges to match themselves. This guards against the confirmation-bias collapse seen in iterative pseudo-labeling.

One setting for all tasks. Hand-tuned CRF kernels are the known failure point of edge-aware scribble methods. We therefore tune $\sigma$ and $\lambda$ once, on ACDC only. All seven ScribbleBench tasks then run with that frozen setting and nnU-Net's own automatic configuration.

Baselines

Do-nothing reference and primary competitor. This is nnU-Net with partial loss on ScribbleBench scribbles, exactly as in Embarrassingly Simple Scribble Supervision for 3D Medical Segmentation. It reports a mean Dice of 0.813 across ScribbleBench. We rerun it ourselves on the same splits and scribbles, with the same epoch budget as our method plus the extra epochs our structure head consumes, so that compute is matched.

Dense-supervision upper bound. Reported at 0.856 mean Dice in the same paper, this fixes the size of the gap we are trying to close. Beating the primary competitor means a mean Dice gain that is significant across five seeds. It also means improvement or no loss on each of the seven tasks, since a gain confined to cardiac data is exactly the pattern ScribbleBench exposed. We additionally require a clear improvement on a surface-distance boundary metric on BraTS2020 and KiTS2023. No reported per-task boundary numbers for partial loss were found on Lacuna, so we compute them ourselves.

CRF refinement. This is partial loss followed by DenseCRF post-processing, reproducing the style of Learning to Segment Medical Images with Scribble-Supervision Alone. The Revisiting paper reports that partial loss surpassed DenseCRF across the seven datasets. No aggregate DenseCRF figure was found on Lacuna. We give the CRF the same tune-on-ACDC-only budget as our method.

Direct port of Boundary Perception Guidance to 3D. This keeps our coupling loss but replaces the learned edge map with a classical Sobel or Canny gradient magnitude, computed slice-wise or in 3D. In the original method, Boundary Perception Guidance reaches 76.0% mIoU on PASCAL VOC 2012 against a 76.4% fully supervised baseline, but it has not been tested on medical volumes.

Ablations isolating the key ingredient. The first keeps the structure head as an auxiliary multitask output with no coupling loss, which tests whether gains come merely from a richer representation. The second removes the reliability weights $r$. The third replaces the learned target with a classical edge-preserving filter's gradient, computed without any network.

Context only. ShapePU (0.369) and CycleMix (0.559) mean ScribbleBench Dice, from the same benchmark papers, are cited only to show where complex methods land. They are not competitors we expect to be close.

Why not just…?

Why not just keep nnU-Net with partial loss? It is already within 4.3 points of dense labels. This is the most serious threat, and the proposal is built around it. Partial loss places no constraint on where the boundary lies between two scribbles. The residual gap should therefore be boundary error, not interior error. We test this before building anything: we split partial-loss errors into voxels within a few millimeters of the true boundary and voxels farther away. If most of the gap is not boundary error, the premise fails and we stop. If it is, our method must recover it with frozen hyperparameters on all seven tasks. Otherwise the benchmark's lesson, that plain methods win, stands.

Why not just refine with a CRF or GrabCut? The evidence in the file runs against it. In the RECIST co-segmentation work, a densely connected CRF raised precision but cut recall and lost to a learned decoder (Weakly Supervised Lesion Co-Segmentation on Ct Scans). Can et al. found CRF hyperparameters fragile and credited uncertainty filtering, not the CRF, for most of their gains. ScribbleBench found partial loss beating DenseCRF outright. Our first coupling term is CRF-like in form, so the fair question is whether learned, reliability-weighted edges trained into the network beat intensity kernels applied afterwards. The CRF baseline, run under the same single-task tuning budget, answers that.

Why not just use image gradients or a classical edge-preserving filter as the edge map? This is a real alternative, and it could plausibly match us on high-contrast CT organs. We include it as an ablation rather than arguing it away. Our hypothesis is that the learned map wins specifically where raw gradients are dominated by noise and fine texture: infiltrative brain tumor margins and kidney tumors next to kidney tissue of similar intensity. If the classical-filter ablation matches the learned map on BraTS2020 and KiTS2023, the reconstruction component is unnecessary. We would then report a simpler method.

Why not just pretrain with masked reconstruction and fine-tune? Generic self-supervised pretraining might explain any gain. The multitask ablation, with structure head present but no coupling, and a masked-reconstruction pretraining variant of the partial-loss model separate "better features" from "edges telling unlabeled voxels where to stop."

Why not prompt a foundation model such as SAM with the scribbles? The indexed resource evaluating SAM zero-shot on medical images (Generalist_Vision_Foundation_Models_for_Medical_Imaging) points to weak handling of 3D context and CT intensity units without adaptation. No reported ScribbleBench number for SAM was found on Lacuna. We run slice-wise SAM prompted with scribble points as a secondary reference on the three CT tasks. We do not treat it as the main comparison.

Proposed Experiments

The main experiment covers all seven ScribbleBench tasks: ACDC, MSCMR, AMOS2022, WORD, LiTS, BraTS2020 and KiTS2023. It uses the benchmark's interior scribbles and five seeds, and reports Dice alongside a boundary surface metric. The decisive check has two parts. The mean Dice gap to 0.856 must shrink relative to our partial-loss rerun. No task may regress, with hyperparameters tuned on ACDC alone.

A second check sweeps scribble density downward. The coupling loss should matter more as labeled voxels become scarcer. If the gain stays flat as scribbles thin out, the mechanism is not doing what we claim.

A third check validates the structure head directly. On held-out dense labels, the head's high-reliability edges should overlap true anatomical boundaries more than Sobel edges do, especially on BraTS2020.

A final stress test repeats the main comparison using ScribbleBench's border scribbles. If coupling adds nothing once annotators already trace boundaries, the method is confirmed to substitute specifically for boundary annotation. That is the practical claim for easing the expert bottleneck.

Image-level-only supervision, where conditional diffusion reaches 0.563 Dice on BraTS (Conditional Diffusion Models for Weakly Supervised Medical Image Segmentation), is a natural follow-up. It is deliberately out of scope, because its gap is dominated by localization rather than boundary placement.