Semi-supervised Contrastive Learning for Label-efficient Medical Image Segmentation

Xinrong Hu, Dewen Zeng, Xiaowei Xu, Yiyu Shi

Abstract

Deep learning methods for medical image segmentation are often limited by the high cost and domain expertise required for pixel-wise annotations. While self-supervised contrastive learning has shown promise in learning representations from unlabeled data, it often ignores the small amount of labeled data available in semi-supervised settings. This paper proposes a novel supervised local contrastive loss that leverages limited pixel-wise annotations during the pre-training phase. By forcing pixels with the same label to cluster together in the embedding space, the method improves segmentation performance. To manage the computational complexity of pixel-wise comparisons, the authors introduce downsampling and block division strategies. Evaluation on MRI and CT datasets demonstrates that this approach consistently outperforms state-of-the-art contrastive and semi-supervised techniques.

Introduction

Training accurate semantic segmentation models requires dense, pixel-level annotations. In the medical domain, this process is exceptionally expensive, requiring expert physicians to manually outline organs or tumors slice by slice. To mitigate this, researchers have turned to semi-supervised learning and contrastive pre-training.

A common paradigm involves a two-step pre-training phase: first learning global (image-level) features and then local (pixel-level) features using only unlabeled data. The actual labels are typically introduced only during a final fine-tuning stage. This paper identifies a missed opportunity: by ignoring available labels during contrastive pre-training, models fail to explicitly group pixels of the same anatomical structure together in the latent space.

Methodology

The proposed framework utilizes a standard 2D U-Net backbone and introduces a hybrid pre-training approach that integrates labels earlier in the pipeline.

Contrastive Learning Strategies

The framework consists of three main components as illustrated below:

  1. Global Contrastive Learning (Self-Supervised): The encoder is trained on unlabeled data to recognize that different augmentations of the same image slice share similar high-level features.
  2. Supervised Local Contrastive Learning: This is the core innovation. Using the labeled subset of data, the decoder is trained with a loss function that forces feature embeddings of pixels with the same class label (positive pairs) to be close together, while pushing pixels with different labels (negative pairs) apart.
Overview of the contrastive learning framework, comparing (a) global contrastive, (b) self-supervised local contrastive, and (c) the proposed supervised local contrastive learning.

Efficiency and Optimization

Computing a contrastive loss between every pixel in high-resolution maps leads to $O(h^4)$ computational complexity. To address this, the authors propose two selection strategies:

  • Downsampling: Computing the loss using features separated by a fixed stride, leveraging the high spatial redundancy in medical images.
  • Block Division: Splitting the feature map into smaller grids and computing the contrastive loss only within local blocks to save memory.

Additionally, the authors explicitly exclude background pixels from the positive set to focus the model's representational power on complex foreground anatomy.

Key Results

The framework was evaluated on two public datasets: the Hippocampus (MRI) dataset and the MMWHS (CT) cardiac dataset. The proposed method consistently outperformed training from scratch and state-of-the-art semi-supervised methods across various label scarcity settings (5% to 40% labeled data).

  • Quantitative Performance: On the MMWHS dataset with 40% labeled data, the global+local(block) method achieved a Dice score of 0.764, significantly higher than the state-of-the-art self-supervised contrastive method (0.730) and the random initialization baseline (0.715).
  • Qualitative Analysis: t-SNE visualizations confirmed that the supervised local contrastive loss creates clearly separated, tightly clustered feature embeddings for different anatomical classes, whereas baseline methods resulted in scattered and overlapping clusters.

Conclusion

By incorporating limited label information directly into the contrastive pre-training phase, this method provides a superior initialization for medical image segmentation tasks. While the supervised local contrastive loss is memory-intensive and requires careful tuning of block sizes and strides, its ability to learn class-specific local features significantly reduces the annotation burden for high-performance medical AI models.

Summary updated

Abstract

The success of deep learning methods in medical image segmentation tasks heavily depends on a large amount of labeled data to supervise the training. On the other hand, the annotation of biomedical images requires domain knowledge and can be laborious. Recently, contrastive learning has demonstrated great potential in learning latent representation of images even without any label. Existing works have explored its application to biomedical image segmentation where only a small portion of data is labeled, through a pre-training phase based on self-supervised contrastive learning without using any labels followed by a supervised fine-tuning phase on the labeled portion of data only. In this paper, we establish that by including the limited label in formation in the pre-training phase, it is possible to boost the performance of contrastive learning. We propose a supervised local contrastive loss that leverages limited pixel-wise annotation to force pixels with the same label to gather around in the embedding space. Such loss needs pixel-wise computation which can be expensive for large images, and we further propose two strategies, downsampling and block division, to address the issue. We evaluate our methods on two public biomedical image datasets of different modalities. With different amounts of labeled data, our methods consistently outperform the state-of-the-art contrast-based methods and other semi-supervised learning techniques.
BibTeX
@inproceedings{hu2021semi,
  author = {Xinrong Hu and Dewen Zeng and Xiaowei Xu and Yiyu Shi},
  title = {{Semi-supervised Contrastive Learning for Label-efficient Medical Image Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  doi = {10.1007/978-3-030-87196-3\_45},
}
Figure