Local Region Fitting in Feature Space for Scribble-Supervised 3D Segmentation

Expert-drawn scribbles, a few quick strokes inside each organ or tumor, cost a tiny fraction of the time a radiologist needs to trace a full 3D mask. The most reliable way to learn from them turns out to be almost trivial. A standard 3D segmentation network trained with a "partial loss", meaning ordinary cross-entropy and Dice computed only on the scribbled voxels, beats every specialized scribble method across seven diverse CT and MRI tasks. It still trails full supervision by about four Dice points. The reason is structural. The partial loss sends no training signal at all to the unlabeled voxels, and those are where boundaries live. We propose to supply that missing signal with a classical idea from active contours: a good segmentation splits each small neighborhood into an "inside" and an "outside" that are each internally homogeneous. We compute that homogeneity on the network's own learned features, not on raw intensities, so the objective still works where CT and MRI edges are faint.

The core claim is that a local region-fitting energy is the right way to regularize unlabeled voxels under scribble supervision. The energy is a Chan-Vese-style objective with neighborhood-level statistics, computed on stop-gradient network features and added to an otherwise unchanged 3D partial-loss pipeline. It is right in a specific sense. It should close a meaningful part of the gap to full supervision across non-cardiac tasks, and it should do so without the recall collapse that pairwise-smoothness methods such as dense CRFs cause on heterogeneous lesions. CRFs are graphical models that encourage neighboring voxels with similar intensity to share a label. Because the energy is built only from Gaussian convolutions, it adds negligible cost and nothing at inference time.

Where the gap is, and what is already done

The motivating evidence comes from Embarrassingly Simple Scribble Supervision for 3D Medical Segmentation and Revisiting 3D Medical Scribble Supervision: Benchmarking Beyond Cardiac Segmentation. These papers built ScribbleBench, a set of seven 3D tasks (LiTS, BraTS2020, AMOS2022, KiTS2023, WORD, MSCMR, ACDC) with simulated interior and border scribbles. On it, nnU-Net with partial loss reaches 0.813 mean Dice against 0.856 for dense supervision. CycleMix (0.559) and ShapePU (0.369) collapse, and some specialized methods fall to 0.05 Dice on brain tumors. These results constrain the design heavily. Any new ingredient must be generic, must add few hyperparameters, and must not rely on the cardiac-tuned heuristics that failed.

Most scribble methods that target boundaries fall into that trap. ScribFormer uses a CNN–Transformer–attention-map triple branch whose self-attention acts as an implicit shape prior. ScribbleVC fuses class embeddings with visual features and mixes confidence-gated pseudo-labels from two decoders. Both are 2D, both are evaluated mainly on cardiac data, and neither has an explicit boundary objective. Learning to Segment Medical Images with Scribble-Supervision Alone expands scribbles with random walks, refines them with offline dense CRFs and uncertainty masking, and gets within 2.9 to 4.5 Dice of full supervision on ACDC and prostate MRI. Its authors report CRF hyperparameter sensitivity and explicitly call for constraints built into training instead of offline loops. That is the direction we take.

The closest cross-domain work is Boundary Perception Guidance. It sharpens scribble-supervised boundaries using an off-the-shelf natural-image edge detector, reaching 76.0% mIoU on PASCAL VOC with CRF. External edge maps are exactly what fails in low-contrast CT and MRI. Our energy uses no external detector and no edges, only region statistics. CaPro filters prompts for a frozen 2D SAM using a topological codebook learned from MNIST digits. That design is specific to thin curvilinear structures, so it does not address solid organs or tumors.

The method itself comes from two places. CNN-based Semantic Segmentation using Level Set Loss showed that the Chan-Vese energy can be made differentiable and trained end to end, gaining about 4 mIoU on PASCAL VOC with essentially no training overhead. It uses global intensity statistics and dense labels. Active Contours Driven by Local Rayleigh Distribution Fitting Energy for Ultrasound Image Segmentation supplies the second idea: fitting statistics in local windows rather than globally, which handles intensity inhomogeneity. Voxel-level feature metric learning in MetricUNet supports using learned features rather than intensities, but MetricUNet needs dense contours to choose its sampled voxels. As far as the file shows, no one has added a variational region prior to the unlabeled voxels of a 3D partial-loss scribble pipeline. Doing so in feature space with local statistics is the specific new combination.

Core Claim and Mechanism

Let $p_c(u)$ be the network's softmax probability of class $c$ at voxel $u$. Let $g(u)$ be a feature vector at $u$, taken with stop-gradient from the penultimate decoder layer of an exponential-moving-average copy of the network. Let $K_\sigma$ be a 3D Gaussian window of width $\sigma$. The local mean of class $c$ around $u$ is

$$m_c(u) = \frac{\sum_{w} K_\sigma(u-w)\, p_c(w)\, g(w)}{\sum_{w} K_\sigma(u-w)\, p_c(w) + \epsilon},$$

and the region energy is

$$E = \sum_{u \in \mathcal{U}} \sum_c p_c(u)\, \lVert g(u) - m_c(u) \rVert^2 .$$

Here $\mathcal{U}$ is the set of unscribbled voxels and $\epsilon$ is a small constant. The training loss is the unchanged nnU-Net partial loss plus $\lambda E$, with $\lambda$ ramped up over training.

Gradients reach only the predictions, never the features. The features therefore cannot collapse to make the energy trivially small. The partial loss keeps the scribbled voxels pinned, which limits the confirmation bias of self-training. Both the numerator and the denominator of $m_c$ are Gaussian convolutions, so the cost is a handful of separable 3D blurs per class.

The hypothesis that separates this energy from CRF-style objectives concerns boundary length. Pairwise smoothness terms reward short boundaries, so they tend to shrink irregular or heterogeneous lesions, which would explain the recall drop that Weakly Supervised Lesion Co-Segmentation on Ct Scans observed with dense CRFs. A region-fitting term has no length penalty. It moves a boundary only when doing so makes both sides of a neighborhood more homogeneous. We expect it to improve boundary precision without systematic under-segmentation, and the experiments measure this directly.

Research Program

The work has three parts. The first is building the energy into the public nnU-Net pipeline used by the ScribbleBench authors, keeping preprocessing, architecture, schedule, and scribble simulation identical. The second is a feature-source study that compares three choices for $g$: raw image intensity, which is effectively a local 3D version of Kim et al.'s loss; stop-gradient EMA decoder features; and features from the model's own partial-loss-pretrained encoder, frozen. The third is a diagnostic analysis of where the gains appear. We bin voxels by distance to the nearest scribble and to the true boundary, and we record volume ratios (predicted over true) per structure to test the no-shrinkage hypothesis. Across all tasks there is a single $\lambda$ and a single $\sigma$ in physical millimeters, chosen on the ACDC and AMOS validation folds only and then frozen. Refusing per-task tuning answers the benchmark's central criticism of over-engineering.

A secondary regime tests clinical metadata as supervision. On NIH DeepLesion, the RECIST long and short diameter lines that oncologists draw routinely are treated as interior scribbles. The line endpoints serve as boundary-adjacent evidence, and the region beyond a dilated bounding ellipse is treated as background scribble. This tests whether the energy turns routine clinical annotations into masks without the GrabCut initialization that Agarwal et al. depend on.

Baselines

The naive reference point is nnU-Net with partial loss, exactly as in Embarrassingly Simple Scribble Supervision for 3D Medical Segmentation. It is also the strongest existing method on this question: it reports 0.813 mean Dice on ScribbleBench, the best scribble-supervised result in the file. We rerun it with the same splits, the same nnU-Net configuration, the same epoch count and the same seeds, so the only difference in our method is the added term. Dense full supervision serves as the ceiling, reported at 0.856 mean Dice by the same authors, and we rerun it under our setup.

We cite the specialized methods at their ScribbleBench numbers rather than retuning them, since that paper already ran them: CycleMix at 0.559 and ShapePU at 0.369. On ACDC alone, the reported ScribFormer Dice is 0.888 from ScribFormer. Because it was reported under a 2D protocol, we treat it as context and not as a like-for-like comparison.

The most dangerous simple competitor is one round of self-training. We train nnU-Net with partial loss, predict dense pseudo-labels on the training volumes, and retrain on them with the same total epoch budget as our method. We also run an offline dense-CRF pseudo-label pipeline in the style of Learning to Segment Medical Images with Scribble-Supervision Alone, giving it the same small hyperparameter search budget on the same two validation tasks as our $\lambda$ and $\sigma$.

Two ablations isolate the key ingredient. The first computes the energy on raw intensities, the local 3D analogue of CNN-based Semantic Segmentation using Level Set Loss; it tests whether learned features matter. The second computes it with global rather than local statistics, taking $\sigma\to\infty$; it tests whether locality matters for heterogeneous tumors.

We will count the proposal as beating the strongest baseline if three conditions hold. Averaged over seeds, it must recover at least 1.5 of the 4.3 Dice points between partial loss and full supervision, pre-registered as mean Dice at or above 0.828. It must reduce mean 95th-percentile Hausdorff distance. And it must not fall more than 0.5 Dice below partial loss on any single task. It must also beat the self-training baseline, or the claim reduces to "more training signal helps."

Why not just…?

Why not just keep nnU-Net with partial loss? It is the reference to beat, and the evidence says it is strong. Its gap to full supervision is still real, and by construction it gives no gradient to the unlabeled voxels, where boundary errors live. If our distance-binned analysis shows the 4.3-point gap sits in voxels far from scribbles and near true boundaries, a targeted term there is justified. If the gap turns out to be spread uniformly, the premise weakens, and we will report that.

Why not just self-train on the partial-loss model's own pseudo-labels? This is the most serious threat. It is simple, fully generic, and also sends signal to unlabeled voxels. Its weakness is that it copies the first model's blobby boundaries and reinforces them, the confirmation bias noted throughout the scribble literature. Our energy instead asks whether a boundary splits the local features cleanly, which is independent evidence. The matched-budget self-training baseline settles the question. If it matches our method, we will say the region term adds nothing beyond more signal.

Why not just expand scribbles offline with CRFs or random walks? The file's evidence runs against it. Learning to Segment Medical Images with Scribble-Supervision Alone reports strong hyperparameter sensitivity. Weakly Supervised Lesion Co-Segmentation on Ct Scans found that dense CRFs sharply cut lesion recall. Revisiting 3D Medical Scribble Supervision reports partial loss outperforming DenseCRF baselines across 3D tasks. We still run the CRF baseline and compare volume ratios, since recall loss is the specific failure our energy is designed to avoid.

Why not just prompt SAM with the scribbles? On non-natural structures, CaPro reports vanilla SAM at 15.60% mIoU on OCT1600 and 24.03% on OCTA-500. Only heavy domain-specific prompt engineering rescued it, and only for curvilinear shapes in 2D. No reported result was found on Lacuna for SAM on ScribbleBench, so we treat SAM as out of scope rather than claim to have beaten it.

Proposed Experiments and Decisive Checks

The main experiment runs all methods on the seven ScribbleBench tasks with three seeds. We report Dice, 95th-percentile Hausdorff distance, surface Dice at a 1 mm tolerance, precision, recall, and volume ratio, per task and as a mean. A scribble-density sweep, thinning the simulated scribbles to half and a quarter, tests whether the energy matters more as supervision gets sparser, as it should if its value lies in reaching unlabeled regions.

The DeepLesion RECIST experiment compares against the reported 90.3% mean Dice of the co-segmentation model in Weakly Supervised Lesion Co-Segmentation on Ct Scans, evaluated on its 1,000 manually masked lesions. We will state plainly that our setting removes GrabCut and adds 3D context, so the comparison is indicative rather than controlled.

Three outcomes would decide the claim. The first is whether feature-space local fitting beats both the intensity and the global ablations on the tumor tasks (LiTS, BraTS, KiTS), where heterogeneity is greatest. The second is whether volume ratios stay near one while CRF pseudo-labels shrink lesions. The third is whether the gain over partial loss survives against matched-budget self-training. Positive answers on all three would show that a cheap variational region prior, not more architecture, is what the scribble regime lacks. Failure on the third would redirect the work toward understanding why plain self-training is enough.