Learning 3D Medical Segmentation from Sparse Annotations

297 concepts246 papers2 proposals

Deep learning has revolutionized medical image analysis, but its success heavily relies on massive amounts of voxel-by-voxel annotations. In 3D domains like Computed Tomography (CT) and Magnetic Resonance Imaging (MRI), this requirement creates a severe volumetric annotation bottleneck. As noted by Amber L. Simpson et al. in A large annotated medical image dataset for the development and evaluation of segmentation algorithms, manually delineating complex anatomical structures across hundreds of individual 2D slices requires hours of expert radiologist labor per scan, making full supervision highly unscalable.

To bypass this bottleneck, a prominent research direction focuses on training high-fidelity 3D segmentation models using sparse or weak annotations—such as a few points, scribbles, or single 2D slices—instead of exhaustive volumetric masks. The core challenge is bridging the information gap between sparse human inputs and dense 3D outputs.

Propagating Sparse Seeds via Geometric and Shape Priors

The most intuitive way to overcome sparse labeling is to automatically expand minimal user inputs into dense pseudo-labels before or during network training.

For instance, Shuwei Zhai et al. demonstrate in PA-Seg: Learning from Point Annotations for 3D Medical Image Segmentation using Contextual Regularization and Cross Knowledge Distillation that a model can achieve near fully supervised performance using only seven point annotations per object (one foreground, six background). This works by leveraging geodesic seed expansion. Instead of calculating naive Euclidean distances, geodesic metrics incorporate both spatial distance and underlying image gradients. The expansion treats the image intensity changes as topographical terrain: the label flows easily through homogenous regions (valleys) but halts abruptly at sharp intensity variations (ridges), effectively mapping out the boundaries of an organ from a single point.

[User Point Click] ──> [Geodesic Distance Field] ──> [Gradient-Bounded Expansion] ──> [Dense Pseudo-Mask]

Where intensity boundaries are blurry or corrupted by artifacts, geometric expansion alone can fail. To counter this, the authors of Weakly Supervised Volumetric Segmentation via Self-taught Shape Denoising Model introduce a self-taught shape denoising network. By training a secondary network to learn global 3D shape priors from the model's own historical best predictions, the system learns what an anatomical structure should look like. It can then filter out irregular, noisy boundary leaks caused by weak slice-level scribbles, enforcing global anatomical plausibility.

Enforcing Multi-View and Spatial Consistency

When a network is trained on sparse annotations like scribbles, it risks overfitting entirely to the labeled voxels while ignoring the surrounding unannotated tissue. To prevent this, researchers enforce consistency constraints across different network views, scales, and feature spaces.

In Scribble-based 3D Multiple Abdominal Organ Segmentation via Triple-branch Multi-dilated Network with Pixel- and Class-wise Consistency, Meng Han et al. tackle this with a multi-dilated network architecture (TDNet). The framework processes the input volume through three parallel branches, each utilizing different dilation rates to alter the network's receptive field.

Why does this multi-scale approach work? By forcing the network to maintain strict pixel- and class-wise consistency across branches with narrow and wide viewing ranges, the model is penalized if its predictions change based on context alone. This forces the feature extractors to decouple the semantic identity of a tissue from the local density of the manual scribbles, ensuring robust generalization to completely unlabeled slices.

                      ┌──> Branch 1 (Low Dilation)  ──┐
                      │                               │
[Sparse Input Volume] ├──> Branch 2 (Mid Dilation)  ──┼──> [Consistency Loss Minimization]
                      │                               │
                      └──> Branch 3 (High Dilation) ──┘

Taking consistency down to the voxel level, Simon Reiß et al. propose a graph-constrained contrastive regularization method (Con2R) in Graph-Constrained Contrastive Regularization for Semi-weakly Volumetric Segmentation. Con2R constructs a spatial graph over the volume, pulling the latent representations of neighboring voxels closer together if they share similar intensities and predicted semantics. This contrastive alignment creates a continuous feature manifold, allowing the sparse supervisory signal to naturally propagate through the graph structure to fill in unannotated gaps.

Bridging the 2D-to-3D Dimensionality Gap

Clinicians are highly proficient at interacting with 2D cross-sections but struggle with the cognitive load of annotating in native 3D space. Consequently, a major subfield involves translating sparse 2D interactions into dense 3D segmentations.

One strategy is cross-dimensional projection. For example, a framework for 3D Arterial Segmentation via Single 2D Projections and Depth Supervision in Contrast-Enhanced CT Images utilizes only a single randomly selected 2D projection per training volume. The model maps the 2D supervision back into the 3D tensor by using the volume's raw intensity data as a depth guide, effectively back-projecting the sparse 2D label along lines of consistent contrast.

Alternatively, researchers are adapting 2D interactive foundation models to handle 3D propagation. In Accelerating Volumetric Medical Image Annotation via Short-Long Memory SAM 2, Yuwen Chen et al. address the problem of "mask drift"—a failure mode where a 2D segmentation mask gradually deforms and errors out as it is propagated slice-by-slice through a 3D volume.

The authors resolve this by introducing a dual-memory bank (Short-Long Memory) to the Segment Anything Model 2 (SAM 2). The short-term memory maintains local continuity between adjacent slices, while the long-term memory stores high-confidence historical anchors from across the entire volume. If the mask begins to drift due to a sudden change in tissue appearance, the long-term memory acts as a global structural correction mechanism, rectifying the propagation path without requiring further manual clicks.

Self-Training and Prototype Rectification

In semi-supervised setups where a few volumes are fully labeled and the rest are completely unlabeled, models rely on self-training to generate pseudo-labels on the unlabeled cohort, as explored by Holger R Roth et al. in Unsupervised Segmentation of 3D Medical Images Based on Clustering and Deep Representation Learning. However, standard self-training suffers from confirmation bias: the model reinforces its own incorrect pseudo-labels over time.

To break this feedback loop, the author of Leveraging labelled data knowledge: A cooperative rectification learning network for semi-supervised 3D medical image segmentation introduces the Cooperative Rectification Learning Network (CRLN).

[Fully Labeled Scans] ──> Extract Clean Class Prototypes ──┐
                                                           ▼
[Unlabeled Scans]     ──> Generate Pseudo-Labels ──> [Adaptive Rectification] ──> Clean Training Signal

CRLN distills the knowledge from the small, fully labeled dataset into a set of clean, representative feature "prototypes" for each tissue class. When the network evaluates an unlabeled volume and generates a noisy pseudo-label, the system measures the distance between the generated features and the clean prototypes. If a pseudo-labeled voxel deviates significantly from its corresponding class prototype, it is adaptively rectified or down-weighted. This ensures that the model's self-training loop is continuously anchored by verified anatomical knowledge, preventing error accumulation and successfully dissolving the volumetric annotation bottleneck.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 239 papers
  2. Version 1 · 2026-10-04 · 298 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_manual_annotation_bottleneck_volumetric__772eece3,
  title = {manual annotation bottleneck (volumetric annotation bottleneck)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manual-annotation-bottleneck-volumetric-annotation-bottleneck/rdr_772eece360ad4a6a912eb2e15af1d7bd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_manual_annotation_bottleneck_volumetric__772eece3_v2,
  title = {manual annotation bottleneck (volumetric annotation bottleneck)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manual-annotation-bottleneck-volumetric-annotation-bottleneck/rdr_772eece360ad4a6a912eb2e15af1d7bd/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake