Semi-supervised Contrastive Learning for Label-Efficient Medical Image Segmentation

Xinrong Hu, Dewen Zeng, Xiaowei Xu, Yiyu Shi

TL;DR

Deep learning for medical image segmentation is bottlenecked by the high cost of acquiring pixel-wise annotations. This paper introduces a semi-supervised contrastive learning framework that leverages limited labeled data during the pre-training phase via a novel supervised local contrastive loss. By using downsampling and block division to keep computational costs in check, the method significantly outperforms existing self-supervised pre-training approaches on medical datasets.

What Problem Does This Paper Address?

Training accurate semantic segmentation models requires dense, pixel-level annotations. In the medical domain, this is exceptionally expensive because it requires expert physicians to manually outline organs or tumors slice by slice.

To get around this, researchers use semi-supervised learning and contrastive learning. A common recent approach is a two-step pre-training phase: first, the model learns global (image-level) features using unlabeled data, and then it learns local (pixel-level) features, again using only unlabeled data. The actual labels are only introduced at the very end during a fine-tuning stage.

The authors identify a missed opportunity here: if you have a small amount of labeled data available anyway, why not use it during the contrastive pre-training phase? By ignoring these labels early on, existing methods fail to explicitly force the model to group pixels of the same anatomical structure together in the latent space.

Methods

The authors use a standard 2D U-Net backbone and propose a hybrid pre-training framework:

  1. Global Contrastive Learning (Self-Supervised): Similar to standard frameworks like SimCLR, the encoder is trained on unlabeled data to recognize that augmented versions of the same image slice are similar, capturing high-level global features.
  2. Supervised Local Contrastive Learning: This is the core novelty. The decoder is attached, and the model is trained using the labeled subset of the data. The loss function forces the feature embeddings of pixels with the same class label (positive pairs) to be close together, and pixels with different class labels (negative pairs) to be pushed apart.
  3. Efficiency Hacks: Computing a contrastive loss between every single pixel in a high-resolution feature map results in an explosive $O(h^4)$ computational complexity, which easily causes Out-Of-Memory (OOM) errors. To fix this, the authors propose two feature selection strategies:
  4. Downsampling: Computing the loss using features separated by a fixed stride (e.g., a stride of 4). This works because neighboring pixels in medical images are highly redundant.
  5. Block Division: Splitting the feature map into smaller grids (e.g., 16x16 blocks) and computing the contrastive loss only within each local block, then averaging the results.

Furthermore, the authors explicitly exclude background pixels from the "positive" set. Since medical images are mostly background, comparing background pixels to other background pixels wastes computational capacity.

Datasets

The methods were evaluated on two public medical image segmentation datasets of different modalities:

  • Hippocampus (MRI): From the MICCAI 2018 Medical Segmentation Decathlon. Contains 260 3D volumes annotated for the anterior and posterior hippocampus.
  • MMWHS (CT): From the MICCAI 2017 challenge. Contains 20 3D cardiac CT volumes annotated for seven different heart structures (left ventricle, right atrium, myocardium, etc.).

Key Insights and Ideas

  • Labels in Pre-training Matter: Injecting even a small fraction of labeled data into the contrastive pre-training phase creates much tighter, class-specific clusters in the embedding space than purely self-supervised pre-training. It provides a vastly superior initialization for the fine-tuning stage.
  • Background is Noise: By excluding background pixels from the positive contrastive set, the model avoids wasting representational power on empty space and focuses entirely on distinguishing complex foreground anatomy.
  • Local Redundancy is a Feature, Not a Bug: The success of the downsampling and block division strategies proves that you don't need to compare every single pixel to learn good local representations. Medical images have high spatial redundancy; skipping pixels saves massive amounts of memory without sacrificing segmentation accuracy.

Key Results

The proposed framework consistently outperformed training from scratch, standard data augmentation (Mixup), and state-of-the-art semi-supervised contrastive methods across various label scarcity settings (5%, 10%, 20%, and 40% labeled data).

  • Quantitative: On the complex MMWHS (CT) dataset with 40% labeled data, the proposed global+local(block) method achieved a Dice score of 0.764. For comparison, the state-of-the-art self-supervised contrastive method achieved 0.730, and a standard random initialization baseline achieved only 0.715.
  • Qualitative: t-SNE visualizations of the latent space showed that the authors' supervised local contrastive loss creates clearly separated, tightly clustered feature embeddings for different anatomical classes. In contrast, baseline methods yielded scattered and overlapping clusters.

Limitations and Future Work

  • Limitations: The authors acknowledge that the supervised local contrastive loss is highly memory-intensive. The block size and stride parameters must be carefully manually tuned based on the hardware's maximum memory to avoid OOM errors. Additionally, if a block contains entirely background pixels, it is skipped entirely, which might discard some useful contextual boundary information.
  • Future Work: While not explicitly detailed in a dedicated section, the natural next steps for this research would involve extending the framework from 2D slices to native 3D volumes (which would further exacerbate the memory challenges but potentially improve context) and exploring dynamic, anatomy-aware block sizing.

Summary updated

Abstract

The success of deep learning methods in medical image segmentation tasks heavily depends on a large amount of labeled data to supervise the training. On the other hand, the annotation of biomedical images requires domain knowledge and can be laborious. Recently, contrastive learning has demonstrated great potential in learning latent representation of images even without any label. Existing works have explored its application to biomedical image segmentation where only a small portion of data is labeled, through a pre-training phase based on self-supervised contrastive learning without using any labels followed by a supervised fine-tuning phase on the labeled portion of data only. In this paper, we establish that by including the limited label in formation in the pre-training phase, it is possible to boost the performance of contrastive learning. We propose a supervised local contrastive loss that leverages limited pixel-wise annotation to force pixels with the same label to gather around in the embedding space. Such loss needs pixel-wise computation which can be expensive for large images, and we further propose two strategies, downsampling and block division, to address the issue. We evaluate our methods on two public biomedical image datasets of different modalities. With different amounts of labeled data, our methods consistently outperform the state-of-the-art contrast-based methods and other semi-supervised learning techniques.
BibTeX
@inproceedings{hu2021semi,
  author = {Xinrong Hu and Dewen Zeng and Xiaowei Xu and Yiyu Shi},
  title = {{Semi-supervised Contrastive Learning for Label-Efficient Medical Image Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  doi = {10.1007/978-3-030-87196-3\_45},
}

Loading related content...

Figure