VoCo: A Simple-yet-Effective Volume Contrastive Learning Framework for 3D Medical Image Analysis

Linshan Wu, Jia-Xin Zhuang, Hao ChenCVPR 2024 · OpenReview

Abstract

Self-Supervised Learning (SSL) has demonstrated promising results in 3D medical image analysis. However, the lack of high-level semantics in pre-training still heavily hinders the performance of downstream tasks. We observe that 3D medical images contain relatively consistent contextual position information, i.e., consistent geometric relations between different organs, which leads to a potential way for us to learn consistent semantic representations in pre-training. In this paper, we propose a simple-yet-effective Volume Contrast (VoCo) framework to leverage the contextual position priors for pre-training. Specifically, we first generate a group of base crops from different regions while enforcing feature discrepancy among them, where we employ them as class assignments of different regions. Then, we randomly crop sub-volumes and predict them belonging to which class (located at which region) by contrasting their similarity to different base crops, which can be seen as predicting contextual positions of different sub-volumes. Through this pretext task, VoCo implicitly encodes the contextual position priors into model representations without the guidance of annotations, enabling us to effectively improve the performance of downstream tasks that require high-level semantics. Extensive experimental results on six downstream tasks demonstrate the superior effectiveness of VoCo.

Introduction

Self-supervised learning (SSL) is vital for 3D medical imaging where expert annotations are expensive and scarce. Current SSL paradigms typically rely on information reconstruction (e.g., Masked Autoencoders) or instance-level contrastive learning (e.g., MoCo). While effective, these methods often focus on low-level textures or local features, failing to capture the high-level semantic structures inherent in human anatomy.

Unlike natural images, 3D medical scans exhibit highly consistent contextual positions. For instance, the liver is consistently located to the right of the stomach, and the kidneys maintain fixed relative positions to the spine. This "spatial prior" provides a natural, global coordinate system that can be leveraged to learn semantic representations. We propose VoCo (Volume Contrast), a framework that treats these consistent anatomical layouts as implicit semantic labels for pre-training.

The VoCo Framework

VoCo introduces a novel pretext task: Contextual Position Prediction. The framework consists of two primary components designed to encode anatomical priors into the model's feature space.

1. Contextual Position Prediction

The input volume is partitioned into a grid of $n$ non-overlapping base crops (e.g., a $4 \times 4 \times 1$ grid). These crops serve as "prototypes" for specific anatomical regions. During training, a sub-volume is randomly cropped from the original image. The model is then tasked with predicting which base regions this random crop overlaps with.

Instead of a hard classification, VoCo uses soft labels based on the percentage of spatial overlap. The prediction is generated by calculating the cosine similarity between the feature representation of the random crop ($p$) and the features of the base crops ($q_i$). This approach allows the model to learn a continuous representation of 3D space.

2. Volume Contrast for Regularization

To ensure that the base crop features are discriminative and to prevent "feature collapse" (where all features become identical), a regularization loss ($L_{reg}$) is applied. This loss encourages the features of different base crops to be as orthogonal as possible, forcing the model to learn distinct representations for different anatomical zones.

The VoCo architecture featuring the Prediction and Regularization branches.

Experimental Results

The effectiveness of VoCo was evaluated across six diverse downstream tasks, including organ segmentation (BTCV, LiTs, MSD Spleen, MM-WHS), brain tumor segmentation (BraTS 21), and COVID-19 classification (CC-CCII). Pre-training was performed on a combined dataset of approximately 1,600 CT scans.

VoCo consistently outperformed state-of-the-art SSL methods. On the BTCV dataset, VoCo achieved a Dice Score of 83.85%, surpassing the previous best method (GL-MAE) by 1.84%. Similar gains were observed in classification tasks, where VoCo reached 90.83% accuracy on the CC-CCII dataset. Notably, even when pre-trained on CT data, VoCo showed superior performance on MRI datasets (BraTS), indicating that the learned anatomical relationships generalize across modalities.

Performance comparison of VoCo against baseline and state-of-the-art methods across six datasets.

Conclusion

VoCo demonstrates that the consistent spatial layout of human anatomy can serve as a powerful "global coordinate system" for self-supervised learning. By predicting the relative positions of sub-volumes through a contrastive approach, the model learns high-level semantic features that are highly beneficial for downstream medical analysis tasks. VoCo is computationally efficient, avoiding the need for large batch sizes or complex clustering, making it a practical and effective solution for 3D medical SSL.

Summary updated

Abstract

Self-Supervised Learning (SSL) has demonstrated promising results in 3D medical image analysis. However, the lack of high-level semantics in pre-training still heavily hinders the performance of downstream tasks. We observe that 3D medical images contain relatively consistent contextual position information, i.e., consistent geometric relations between different organs, which leads to a potential way for us to learn consistent semantic representations in pre-training. In this paper, we propose a simple-yet-effective Volume Contrast (VoCo) framework to leverage the contextual position priors for pre-training. Specifically, we first generate a group of base crops from different regions while enforcing feature discrepancy among them, where we employ them as class assignments of different regions. Then, we randomly crop sub-volumes and predict them belonging to which class (located at which region) by contrasting their similarity to different base crops, which can be seen as predicting contextual positions of different sub-volumes. Through this pretext task, VoCo implicitly encodes the contextual position priors into model representations without the guidance of annotations, enabling us to effectively improve the performance of downstream tasks that require high-level semantics. Extensive experimental results on six downstream tasks demonstrate the superior effectiveness of VoCo.
BibTeX
@inproceedings{wu2024voco,
  author = {Linshan Wu and Jia-Xin Zhuang and Hao Chen},
  title = {{VoCo: A Simple-yet-Effective Volume Contrastive Learning Framework for 3D Medical Image Analysis}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=DE11kMbapd},
}
Figure