Self-supervised learning for medical image classification: a systematic review and implementation guidelines

Shih-Cheng Huang, Anuj Pareek, Malte E. K. Jensen, Matthew P. Lungren, Serena Yeung, Akshay S. ChaudhariNpj Digit. Medicine 2023 · DOI

TL;DR

Self-supervised learning (SSL) offers a promising path to overcome the data scarcity and annotation costs in medical imaging AI. This systematic review synthesizes existing SSL strategies for medical image classification, providing a comprehensive overview and practical guidelines for researchers, highlighting that combined SSL approaches and end-to-end fine-tuning generally yield superior results.

What Problem Does This Paper Address?

The field of medical image analysis has seen significant advancements with deep learning, but its widespread adoption is hampered by the need for massive, meticulously labeled datasets. Creating these datasets is time-consuming, expensive, and requires specialized expertise, making it a bottleneck for developing robust AI models across diverse clinical applications. Furthermore, supervised models often struggle to generalize to new datasets or tasks, limiting their real-world utility. Self-supervised learning (SSL) presents a compelling solution by enabling models to learn valuable representations from abundant unlabeled medical data, thereby reducing reliance on manual annotations and potentially improving generalization. This paper addresses the need for a structured understanding of SSL in medical image classification, aiming to consolidate terminology, review existing approaches, and offer practical guidance for future research.

Methods

This paper presents a systematic review of self-supervised learning (SSL) applied to medical image classification. The authors followed the PRISMA guidelines for their search strategy, which involved querying PubMed, Scopus, and ArXiv for studies published between January 2012 and May 2021. They defined four main categories of SSL strategies:

  • Innate Relationship SSL: Pre-training on hand-crafted tasks that leverage the internal structure of data without additional labels (e.g., predicting image rotation, solving jigsaw puzzles).
  • Generative SSL: Learning the data distribution to reconstruct original inputs or create synthetic data (e.g., autoencoders, GANs).
  • Contrastive SSL: Learning by contrasting positive pairs (different augmentations of the same image) with negative pairs, aiming to minimize distance in latent space for positive pairs.
  • Self-Prediction SSL: Masking or augmenting parts of an image and reconstructing the original based on the unaltered portions (inspired by masked language modeling in NLP).

The review also categorizes fine-tuning strategies into "end-to-end fine-tuning" and "extract features for classifier."

Datasets

The review does not introduce new datasets. Instead, it systematically analyzes existing research that utilized various medical imaging datasets across different modalities and clinical domains. The paper mentions the use of publicly available datasets like CheXpert, RSPECT, RadFusion, and MIMIC-CXR for chest imaging studies, and highlights the general challenge of data availability for multi-modal approaches involving radiology reports due to privacy concerns.

Key Insights and Ideas

  • SSL is a rapidly growing field in medical imaging: The number of publications on SSL for medical image classification has seen a significant increase, indicating its rising importance.
  • Combined SSL strategies show promise: The review suggests that combining different SSL approaches (e.g., contrastive and generative) tends to outperform individual strategies on downstream tasks.
  • End-to-end fine-tuning often leads to better performance: For medical images, fine-tuning the entire pre-trained model (encoder and classifier) generally yields better results compared to training a classifier on extracted features.
  • Medical images present unique challenges for SSL: Unlike natural images, medical images often have high inter-class visual similarities and subtle pathological cues that can be obscured or altered by standard augmentation techniques used in SSL. This necessitates tailored augmentation strategies.
  • SSL is particularly valuable in low-data regimes: The benefits of SSL are most pronounced when labeled data is scarce, making it a crucial tool for improving model performance and generalization.
  • Clinical relevance is key: Innate relationship pre-text tasks that are clinically relevant showed greater performance improvements over transformation-based pre-text tasks.

Key Results

The systematic review analyzed 79 studies. Key findings include:

  • Performance Improvement: The majority of self-supervised pre-trained models showed relative performance improvements of 0.216-32.6% AUROC, 0.440-29.2% accuracy, and 0.137-14.3% F1 score compared to models without SSL pre-training.
  • Prevalence of Contrastive Learning: Contrastive learning was the most frequently used SSL strategy, appearing in 44 out of 79 studies.
  • Dominant Modalities and Domains: Radiology was the most explored clinical domain (47/79 studies), with a strong focus on chest imaging (pneumonia, COVID-19, etc.). Pathology was the second most prevalent domain (12/79 studies).
  • Fine-tuning Strategy: End-to-end fine-tuning generally led to better performance for medical images compared to training a classifier on extracted features.
  • Comparison of SSL Strategies: Figure 4c indicates that combined SSL strategies tend to outperform other individual categories, though publication bias might play a role.

Limitations and Future Work

Limitations:

  • Publication Bias: The review acknowledges that studies with positive results are disproportionately reported, potentially leading to an overestimation of SSL benefits.
  • Limited Comparability: The included studies used diverse imaging modalities, performance metrics, and objectives, making direct statistical comparison of SSL strategies challenging.
  • Subjectivity in Categorization: Categorizing SSL strategies could involve subjectivity, especially for novel or hybrid approaches.
  • Scope: The review focused solely on medical image classification, excluding other important tasks like segmentation, regression, and registration.

Future Work:

  • Direct Comparisons: Future research should conduct direct comparisons of different SSL strategies using standardized terminology and metrics to facilitate discovery of optimal approaches.
  • Tailored Augmentations: Developing augmentation strategies specifically designed for the unique properties of medical images is crucial.
  • Leveraging Clinical Information: Exploring the use of patient demographics or medical history to define positive pairs in contrastive learning could enhance performance.
  • Multi-modal Learning: Further investigation into incorporating additional modalities (e.g., radiology reports) into SSL frameworks is warranted.
  • Federated Learning: Utilizing federated learning to train SSL models across multiple institutions without sharing patient data could address privacy concerns and enable larger-scale training.
  • Benchmarking: Researchers should systematically test new SSL strategies on multiple datasets and ideally across different modalities and medical imaging domains.

Summary updated

BibTeX
@article{huang2023self,
  author = {Shih-Cheng Huang and Anuj Pareek and Malte E. K. Jensen and Matthew P. Lungren and Serena Yeung and Akshay S. Chaudhari},
  title = {{Self-supervised learning for medical image classification: a systematic review and implementation guidelines}},
  journal = {Npj Digit. Medicine},
  year = {2023},
  doi = {10.1038/s41746-023-00811-0},
}
Figure