3D Medical Image Segmentation with Sparse Annotation via Cross-Teaching Between 3D and 2D Networks

Heng Cai, Lei Qi, Qian Yu, Yinghuan Shi, Yang Gao

TL;DR

Training 3D medical segmentation models usually requires experts to painstakingly label every single slice of a scan. This paper introduces a 3D-2D Cross-Teaching framework that achieves state-of-the-art results using only 16% of the labeled slices. By making a 3D network and two 2D networks "teach" each other through clever pseudo-label selection, the system reaches performance levels comparable to models trained on 100% of the data.

What Problem Does This Paper Address?

Deep learning has revolutionized medical imaging, but it has a "data hunger" problem. For 3D tasks like heart or organ segmentation, a single patient volume might contain hundreds of slices. Asking a radiologist to provide pixel-perfect labels for every slice (dense annotation) is expensive and often unfeasible in clinical practice.

Researchers have tried "weak" annotations like bounding boxes or scribbles, but these often lose the precise object boundaries critical for surgery or diagnosis. A better middle ground is sparse annotation: labeling only a few representative slices. However, training on sparse data is difficult because the "supervision signal" is too thin—the model doesn't see enough examples to generalize well to the unlabeled gaps between slices.

Methods

The authors propose a framework based on the principle of co-training, where different models provide "pseudo-labels" for each other on unlabeled data. Their specific innovation is the 3D-2D Cross-Teaching architecture:

  1. Cross Annotation: Instead of labeling random slices, they label slices from two different planes (transverse and coronal). This creates a "cross" shape in 3D space, capturing more anatomical diversity.
  2. Diverse Perspectives: They use one 3D network (V-Net) and two 2D networks (U-Nets). The 3D network excels at understanding inter-slice context, while the 2D networks focus on high-resolution intra-slice patterns.
  3. Pseudo-Label Selection: To ensure the models don't "teach" each other mistakes, they developed two strategies:
  4. Hard-Soft Confidence Threshold (for the 3D network): The model estimates its own accuracy based on the sparse ground truth. If it's performing well ("reliable"), it uses a lower ("soft") threshold to keep more labels. If it's struggling, it uses a "hard" threshold to only pass on its most certain predictions.
  5. Consistent Label Fusion (for 2D networks): Since the two 2D networks look at different planes, the framework only trusts pseudo-labels where both 2D networks agree.
  6. Mixed Supervision: The final loss function combines the sparse ground-truth labels with these filtered pseudo-labels to train all networks simultaneously.

Datasets

The researchers evaluated their method on the MMWHS (Multi-Modality Whole Heart Segmentation) dataset.

  • Composition: 20 cardiac CT images covering seven heart substructures.
  • Setup: They split the data into 12 volumes for training, 4 for validation, and 4 for testing.
  • Sparsity: While the "fully supervised" baseline used 96 labeled slices per volume, the proposed method used only 16 slices per volume.

Key Insights and Ideas

  • Dimensionality as Diversity: In traditional co-training, you often use two identical models with different initializations. Here, the "view difference" comes from the architecture itself (3D vs. 2D). The 3D model sees the "forest" (volume), while the 2D models see the "trees" (slices), making their errors less likely to overlap.
  • Dynamic Trust: The "Hard-Soft" thresholding recognizes that a model's confidence isn't always a proxy for accuracy. By using the sparse ground truth to estimate a "pseudo-accuracy," the system dynamically decides how much to trust the model's predictions during the teaching phase.
  • Efficiency: The fact that the model outperforms the fully supervised version (82.67% vs 81.69% Dice score) suggests that the cross-teaching mechanism might actually act as a form of regularization, preventing the model from over-fitting to specific slices.

Key Results

The method was compared against several State-of-the-Art (SOTA) semi-supervised methods like Mean Teacher (MT) and Cross-Pseudo Supervision (CPS).

  • Accuracy: The proposed method achieved a Dice score of 82.67%. This outperformed the best SOTA semi-supervised method (CPS at 77.19%) by a significant margin.
  • Efficiency: Remarkably, the method slightly surpassed the fully-supervised upper bound (81.69%) despite using roughly 1/6th of the manual labeling effort.
  • Boundary Precision: Qualitative results (visual scans) showed that the proposed method produced much smoother and more anatomically accurate boundaries compared to other weak-supervision methods, which often produced "noisy" or fragmented segments.

Limitations and Future Work

  • Dataset Size: The study was conducted on a relatively small dataset (20 volumes). While common in medical imaging research, testing on larger, more heterogeneous datasets would be necessary to prove its robustness.
  • Plane Dependency: The method relies on labeling specific planes (transverse and coronal). In some medical conditions, these planes might not be the most informative, and an automated way to select the "best" slices to annotate could be a valuable extension.
  • Computational Overhead: Running three networks (one 3D, two 2D) simultaneously during training increases the computational cost compared to a single-model setup, though this is usually acceptable if it saves hundreds of hours of expert labor.

Summary updated

Abstract

Medical image segmentation typically necessitates a large and precisely annotated dataset. However, obtaining pixel-wise annotation is a labor-intensive task that requires significant effort from domain experts, making it challenging to obtain in practical clinical scenarios. In such situations, reducing the amount of annotation required is a more practical approach. One feasible direction is sparse annotation, which involves annotating only a few slices, and has several advantages over traditional weak annotation methods such as bounding boxes and scribbles, as it preserves exact boundaries. However, learning from sparse annotation is challenging due to the scarcity of supervision signals. To address this issue, we propose a framework that can robustly learn from sparse annotation using the cross-teaching of both 3D and 2D networks. Considering the characteristic of these networks, we develop two pseudo label selection strategies, which are hard-soft confidence threshold and consistent label fusion. Our experimental results on the MMWHS dataset demonstrate that our method outperforms the state-of-the-art (SOTA) semi-supervised segmentation methods. Moreover, our approach achieves results that are comparable to the fully-supervised upper bound result.
BibTeX
@inproceedings{cai2023d,
  author = {Heng Cai and Lei Qi and Qian Yu and Yinghuan Shi and Yang Gao},
  title = {{3D Medical Image Segmentation with Sparse Annotation via Cross-Teaching Between 3D and 2D Networks}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  doi = {10.1007/978-3-031-43898-1\_59},
}

Loading related content...

Figure