SGTC: Semantic-Guided Triplet Co-training for Sparsely Annotated Semi-Supervised Medical Image Segmentation

Ke Yan, Qing Cai, Fan Zhang, Ziyan Cao, Zhi LiuAAAI 2025 · DOI

TL;DR

This paper introduces SGTC, a novel framework for semi-supervised medical image segmentation that significantly reduces the annotation burden by only requiring three orthogonal slices per volume. SGTC leverages CLIP's text-embedding capabilities for semantic guidance and a triplet co-training strategy to improve segmentation accuracy and robustness, outperforming existing methods on challenging benchmarks.

What Problem Does This Paper Address?

Accurate segmentation of anatomical structures and pathologies in medical images is crucial for diagnosis, treatment planning, and research. However, existing deep learning methods often require a large number of meticulously annotated medical images, which are expensive and time-consuming to create, especially for 3D volumetric data. While semi-supervised learning offers a solution by utilizing unlabeled data, many approaches still demand dense annotations for a limited set of labeled volumes. Even worse, some methods that attempt to reduce annotation by using only one or two slices per volume struggle with semantic understanding and fail to capture the full 3D spatial information, leading to suboptimal segmentation. This paper addresses the critical need for effective medical image segmentation methods that can achieve high performance with minimal annotation effort, specifically by leveraging sparse annotations from three orthogonal slices.

Methods

The SGTC framework comprises two main components:

  • Semantic-Guided Auxiliary Learning Mechanism (SGAL): This mechanism aims to enhance the quality of pseudo-labels and enable semantic-aware segmentation. It leverages the pre-trained CLIP model to incorporate text representations. Specifically, it uses an "Adapter" module to fine-tune CLIP's text encoder to better understand clinical semantics from medical prompts. These text embeddings are then combined with image features to guide the segmentation network.
  • Triple-View Disparity Training Strategy (TVDT): This strategy tackles the challenge of sparse annotations by employing a triplet of sub-networks, each trained on different orthogonal views (sagittal, coronal, axial). Each sub-network is supervised by two of the three available orthogonal slices. This approach encourages disagreement among the sub-networks, forcing them to learn complementary knowledge and improving overall robustness. For unlabeled data, the sub-networks engage in co-training, where the predictions of one network serve as pseudo-labels for the others.

The overall loss function combines a supervised loss (weighted cross-entropy and Dice loss) for labeled data and an unsupervised loss (weighted cross-entropy loss) for unlabeled data, with a dynamic weighting parameter to balance their contributions during training.

Datasets

The proposed SGTC framework was evaluated on three public medical image segmentation datasets:

  • LA2018: Gadolinium-enhanced MR imaging scans for left atrium segmentation.
  • KiTS19: Abdominal CT scans for kidney and kidney tumor segmentation.
  • LiTS: CT dataset focused on liver and liver tumor segmentation.

The paper uses standard splits for training and testing, with a focus on evaluating performance under a 10% labeled data setting.

Key Insights and Ideas

  • Minimal Annotation, Maximum Impact: The core idea that only three orthogonal slices per volume are sufficient for high-quality semi-supervised segmentation is a significant advancement, drastically reducing the annotation bottleneck.
  • CLIP for Clinical Semantics: Leveraging CLIP's text-embedding capabilities, even with a fine-tuned adapter, proves effective in injecting semantic understanding into medical image segmentation, going beyond purely visual features.
  • Disagreement is Key: The triple-view disparity training strategy, by forcing sub-networks to learn from different orthogonal views and then co-train, highlights the benefit of encouraging diversity and complementary learning among models.
  • Orthogonal Views are Better than Parallel: Visualizations and quantitative results suggest that using three orthogonal slices provides a more comprehensive representation of the 3D volume compared to using three parallel slices or just two orthogonal slices.

Key Results

  • State-of-the-Art Performance: SGTC consistently outperforms seven state-of-the-art semi-supervised segmentation methods across the LA2018, KiTS19, and LiTS datasets under a 10% labeled data setting. For instance, on the LA2018 dataset, SGTC achieved a Dice score of 0.847 ± 0.044, surpassing the second-best method by 6.3%.
  • Improved Robustness: The triple-view disparity training strategy, especially with the SCA (sagittal, coronal, axial) annotation, leads to better shape-related performance and more comprehensive supervision signal distribution.
  • Semantic Guidance Benefits: The integration of semantic cues from CLIP significantly improves the network's ability to perceive subtle changes and boundary information, leading to more accurate segmentation.
  • Text Prompt Effectiveness: Combining visual and textual modalities through carefully designed prompts yields substantial performance gains compared to using only image features.

Limitations and Future Work

  • Limited Foreground Information: The authors acknowledge that SGTC's performance may degrade when the selected sparse slices contain limited foreground information.
  • Future Directions: The paper suggests optimizing the approach to address the limitation of sparse slices with limited foreground information. Further research could explore more advanced prompt engineering for CLIP or investigate alternative strategies for handling challenging cases with minimal visible structures in the annotated slices.

Summary updated

BibTeX
@inproceedings{yan2025sgtc,
  author = {Ke Yan and Qing Cai and Fan Zhang and Ziyan Cao and Zhi Liu},
  title = {{SGTC: Semantic-Guided Triplet Co-training for Sparsely Annotated Semi-Supervised Medical Image Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i9.32986},
}
Figure