3D Medical Image Segmentation with Sparse Annotation via Cross-Teaching between 3D and 2D Networks

Heng Cai, Lei Qi, Qian Yu, Yinghuan Shi, Yang Gao

Abstract

Medical image segmentation typically necessitates a large and precisely annotated dataset. However, obtaining pixel-wise annotation is a labor-intensive task that requires significant effort from domain experts, making it challenging to obtain in practical clinical scenarios. In such situations, reducing the amount of annotation required is a more practical approach. One feasible direction is sparse annotation, which involves annotating only a few slices, and has several advantages over traditional weak annotation methods such as bounding boxes and scribbles, as it preserves exact boundaries. However, learning from sparse annotation is challenging due to the scarcity of supervision signals. To address this issue, we propose a framework that can robustly learn from sparse annotation using the cross-teaching of both 3D and 2D networks. Considering the characteristic of these networks, we develop two pseudo label selection strategies, which are hard-soft confidence threshold and consistent label fusion. Our experimental results on the MMWHS dataset demonstrate that our method outperforms the state-of-the-art (SOTA) semi-supervised segmentation methods. Moreover, our approach achieves results that are comparable to the fully-supervised upper bound result.

Introduction

Deep learning has revolutionized medical imaging, but its success is often predicated on the availability of massive, densely annotated datasets. For 3D volumetric tasks, such as heart or organ segmentation, a single patient volume can contain hundreds of slices. Requiring radiologists to provide pixel-perfect labels for every slice is economically and practically unfeasible in many clinical settings.

While weak annotations like bounding boxes or scribbles offer a faster alternative, they often fail to capture the precise anatomical boundaries necessary for surgical planning or diagnostic accuracy. Sparse annotation—labeling only a small subset of representative slices—presents a compelling middle ground. However, the resulting "thin" supervision signal makes it difficult for standard models to generalize across the unlabeled gaps. This paper introduces a 3D-2D Cross-Teaching framework designed to overcome these limitations by leveraging the complementary strengths of different network architectures.

Methodology

The proposed framework utilizes a co-training principle where multiple models provide pseudo-labels for one another on unlabeled data. The core innovation lies in the 3D-2D Cross-Teaching architecture, which employs one 3D network (V-Net) and two 2D networks (U-Nets).

Cross Annotation Strategy

To maximize anatomical diversity with minimal effort, the researchers utilize a "cross annotation" strategy. Instead of labeling random slices, they label slices from two orthogonal planes (transverse and coronal). This creates a cross-shaped sampling in 3D space, ensuring the model receives supervision across different orientations.

Visualization of the sparse cross-annotation strategy, showing labeled slices in different planes.

Pseudo-Label Selection

To prevent the propagation of errors between models, two specific selection strategies were developed:

  1. Hard-Soft Confidence Threshold (3D Network): The 3D network estimates its own reliability based on its performance on the sparse ground truth. If reliability is high, a "soft" (lower) threshold is used to retain more pseudo-labels; if low, a "hard" (higher) threshold ensures only the most certain predictions are passed to the 2D networks.
  2. Consistent Label Fusion (2D Networks): Because the two 2D networks observe the volume from different planes, the framework only accepts pseudo-labels where both 2D models agree, effectively filtering out view-specific noise.

Results and Discussion

The framework was evaluated on the MMWHS (Multi-Modality Whole Heart Segmentation) dataset, using only 16 labeled slices per volume (approximately 16% of the data used in fully supervised baselines).

A comprehensive overview of the segmentation results, including ground truth (a), various sparse annotation patterns (b-e), and the consistency mapping (f) used for label fusion.

Performance Metrics

The proposed method achieved a Dice score of 82.67%, significantly outperforming state-of-the-art semi-supervised methods like Cross-Pseudo Supervision (CPS), which reached 77.19%. Remarkably, the method slightly surpassed the fully-supervised upper bound of 81.69%, suggesting that the cross-teaching mechanism may provide a regularizing effect that prevents overfitting to specific slices.

Conclusion

By combining the "forest-level" context of 3D networks with the "tree-level" resolution of 2D networks, this framework demonstrates that high-fidelity medical segmentation is possible with a fraction of the manual labeling effort. This approach significantly lowers the barrier for deploying high-performance AI tools in clinical environments where expert time is at a premium.

Summary updated

Abstract

Medical image segmentation typically necessitates a large and precisely annotated dataset. However, obtaining pixel-wise annotation is a labor-intensive task that requires significant effort from domain experts, making it challenging to obtain in practical clinical scenarios. In such situations, reducing the amount of annotation required is a more practical approach. One feasible direction is sparse annotation, which involves annotating only a few slices, and has several advantages over traditional weak annotation methods such as bounding boxes and scribbles, as it preserves exact boundaries. However, learning from sparse annotation is challenging due to the scarcity of supervision signals. To address this issue, we propose a framework that can robustly learn from sparse annotation using the cross-teaching of both 3D and 2D networks. Considering the characteristic of these networks, we develop two pseudo label selection strategies, which are hard-soft confidence threshold and consistent label fusion. Our experimental results on the MMWHS dataset demonstrate that our method outperforms the state-of-the-art (SOTA) semi-supervised segmentation methods. Moreover, our approach achieves results that are comparable to the fully-supervised upper bound result.
BibTeX
@inproceedings{cai2023d,
  author = {Heng Cai and Lei Qi and Qian Yu and Yinghuan Shi and Yang Gao},
  title = {{3D Medical Image Segmentation with Sparse Annotation via Cross-Teaching between 3D and 2D Networks}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  doi = {10.1007/978-3-031-43898-1\_59},
}
Figure