Training Segmentation Models with Sparse Annotations

This direction is merged into another direction: sparse annotation efficiency. Its last version stays readable and citable here.

Version 1 of 1108 papers114 conceptsfrozen 2026-10-04

The standard recipe for training deep learning models for image segmentation is "pixel-perfect" supervision: an expert meticulously traces the boundaries of every object of interest. In medical imaging or specialized computer vision, this process is a massive bottleneck. A single 3D CT scan can take a radiologist hours to annotate, yet a model might need thousands of such scans to generalize.

Research into sparse and weak supervision suggests that this exhaustive effort is largely redundant. By using "cheap" labels—such as scribbles, points, or bounding boxes—researchers have found they can achieve segmentation performance nearly indistinguishable from models trained on dense masks, often reducing human labor by over 90%.

The Redundancy of the Pixel

The core intuition behind this research is that most pixels in a dense mask provide very little new information to a neural network. Once a model understands the "core" appearance of a tumor and the general "background" of healthy tissue, the exact placement of the boundary voxels becomes a refinement task rather than a fundamental learning task.

Shengqi Zhu and colleagues demonstrated this early on with "label strokes," showing that sparse lines across street scenes could train robust parsers with only marginal performance degradation. Similarly, in the medical domain, Michael Goetz introduced DALSA, which uses quick scribbles in unambiguous areas to slash labeling time by 70x while using domain adaptation to correct for the statistical bias of avoiding "hard" boundary pixels.

From Seeds to Masks: Propagation and Expansion

If the model is only given a few points or a scribble, how does it learn to predict a full, dense mask? The dominant strategy is to treat sparse annotations as "seeds" that expand to fill the object.

  1. Graphical Propagation: Methods like ScribbleSup by Di Lin and Jifeng Dai alternate between training a CNN and propagating labels through a graphical model (like a Markov Random Field). The CNN learns the appearance, while the graph ensures spatial consistency, pushing the labels from the scribbles to the object boundaries.
  2. Region Growing: Xinggang Wang and Zilong Huang developed Deep Seeded Region Growing (DSRG). This integrates classical image processing—starting at a high-confidence seed and "growing" the region to similar nearby pixels—directly into the deep learning loss function.
  3. Geometric Priors: For 3D volumes, Shuwei Zhai showed in PA-Seg that as few as seven points (one foreground, six on the background faces) are sufficient. By using geodesic distances, the model can infer the 3D extent of an organ from these minimal anchors.

Leveraging Consistency and Uncertainty

When dense labels are missing, models must rely on internal logic to stay "honest." This is often achieved through consistency regularization.

Issam Laradji and Pau Rodríguez demonstrated this for COVID-19 CT segmentation. By enforcing that a model’s prediction for an image must be spatially equivariant (e.g., if you rotate the image, the predicted mask should rotate identically), the model learns to respect the underlying structure of the data even when it only has a single "point" click for supervision.

Another approach is Adversarial Erasing, pioneered by Yunchao Wei. To force a model to find the entire object rather than just the most discriminative part (like a bird's beak), the system iteratively "erases" the parts it has already found, forcing the network to discover the remaining extent of the object to satisfy the image-level label.

The Power of Strong Baselines

A surprising finding in recent years is that complex, specialized algorithms for weak supervision might be over-engineered. Karol Gotkowski and Paul F Jaeger introduced ScribbleBench, showing that an "embarrassingly simple" baseline—a standard nnU-Net trained only on the pixels covered by scribbles—often outperforms state-of-the-art weakly supervised methods.

This suggests that the most important factor isn't necessarily a complex label-propagation math, but rather a high-capacity model and a training framework that doesn't "overfit" to the sparsity of the input. As Guotai Wang emphasizes with the PyMIC toolkit, the goal is "annotation-efficiency": moving the focus from the precision of a single label to the volume and diversity of the dataset that sparse labeling enables.

Papers in this version

Showing 60 of 108 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_training_segmentation_models_with_sparse_68679ec5,
  title = {Training Segmentation Models with Sparse Annotations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/training-segmentation-models-with-sparse-annotations/rdr_68679ec51fa64d028d5999110f082c71},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_training_segmentation_models_with_sparse_68679ec5_v1,
  title = {Training Segmentation Models with Sparse Annotations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/training-segmentation-models-with-sparse-annotations/rdr_68679ec51fa64d028d5999110f082c71/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}