Self-Supervised Pretraining Enables High-Performance Chest X-Ray Interpretation Across Clinical Distributions

Niveditha S. Iyer, Aditya Gulati, Oishi Banerjee, Cécile Logé, Maha Farhat, Agustina Saenz, Pranav RajpurkarmedRxiv 2022 · DOI

Abstract

Chest X-rays (CXRs) are a rich source of information for physicians – essential for disease diagnosis and treatment selection. Recent deep learning models aim to alleviate strain on medical resources and improve patient care by automating the detection of diseases from CXRs. However, shortages of labeled CXRs can pose a serious challenge when training models. Currently, models are generally pretrained on ImageNet, but they often need to then be finetuned on hundreds of thousands of labeled CXRs to achieve high performance. Therefore, the current approach to model development is not viable on tasks with only a small amount of labeled data. An emerging method for reducing reliance on large amounts of labeled data is self-supervised learning (SSL), which uses unlabeled CXR datasets to automatically learn features that can be leveraged for downstream interpretation tasks. In this work, we investigated whether self-supervised pretraining methods could outperform traditional ImageNet pretraining for chest X-ray interpretation. We found that SSL-pretrained models outperformed ImageNet-pretrained models on thirteen different datasets representing high diversity in geographies, clinical settings, and prediction tasks. We thus show that SSL on unlabeled CXR data is a promising pretraining approach for a wide variety of CXR interpretation tasks, enabling a shift away from costly labeled datasets.

Introduction

The standard paradigm in medical AI relies heavily on transfer learning from natural image datasets like ImageNet. However, a significant "domain gap" exists between natural images (e.g., animals, objects) and medical imaging (e.g., grayscale, high-resolution radiographs). Bridging this gap typically necessitates large volumes of labeled medical data for effective finetuning. Obtaining these labels is a major bottleneck, requiring extensive time from board-certified radiologists. This study explores Self-Supervised Learning (SSL) as a solution, leveraging unlabeled images and radiology reports to learn domain-specific features without the need for manual annotation.

Methodology

The researchers evaluated seven SSL methods against a standard ImageNet-pretrained ResNet-50 baseline. These methods were categorized into two primary groups:

  1. Image-only models: (MedAug, S2MTS2, MoCo-CXR) which learn structural patterns from X-ray images alone.
  2. Multimodal models: (ConVIRT, REFERS, GLoRIA, etc.) which learn by aligning X-ray images with their corresponding free-text radiology reports.

Performance was assessed through Linear Probing (evaluating inherent feature quality with a frozen backbone) and Finetuning (evaluating maximum task-specific potential).

Experimental Setup and Datasets

The study's robustness stems from its evaluation across 13 diverse datasets, covering various clinical settings and global geographies including the USA, Brazil, China, Spain, and Vietnam.

Summary of the 13 datasets used for evaluation, showing diversity in geography, setting, and predictive tasks.

Results and Discussion

The findings demonstrate that SSL-pretrained models consistently outperform ImageNet-pretrained models across all 13 evaluation datasets when finetuned.

Multimodal Advantage

Multimodal models (image + text) consistently emerged as the top performers. The alignment of visual features with semantic information from radiology reports provides a superior training signal compared to image-only methods. For instance, on the NIH ChestX-ray14 dataset, models like ConVIRT showed significant AUC improvements over the baseline.

Robustness to Distribution Shift

A critical challenge in medical AI is maintaining performance across different clinical environments. SSL-pretrained models showed greater resilience to distribution shifts, maintaining high performance across diverse global populations compared to the ImageNet baseline.

AUC performance comparison across multiple datasets and pathologies. SSL models consistently outperform the supervised ResNet50 baseline (grey).

Feature Quality

Linear probing results indicated that SSL models learn more relevant representations of medical anatomy. In many cases, the frozen features of SSL models yielded better results than the fully finetuned ImageNet baseline, suggesting that SSL captures the underlying "language" of medical imaging more effectively.

Limitations and Future Directions

Despite these advancements, certain tasks remain challenging. Models still struggle with subjective assessments, such as determining the severity of pulmonary edema, and complex spatial tasks, like the precise localization of central lines. Future research should focus on integrating clinical metadata and scaling SSL to even larger, more diverse unlabeled datasets to further improve spatial reasoning and clinical utility.

Conclusion

Self-supervised pretraining on domain-specific data significantly enhances the performance and generalizability of chest X-ray interpretation models. By reducing the dependency on large-scale labeled datasets, SSL paves the way for more accessible and robust AI tools in clinical practice.

Summary updated

Abstract

Abstract Chest X-rays (CXRs) are a rich source of information for physicians – essential for disease diagnosis and treatment selection. Recent deep learning models aim to alleviate strain on medical resources and improve patient care by automating the detection of diseases from CXRs. However, shortages of labeled CXRs can pose a serious challenge when training models. Currently, models are generally pretrained on ImageNet, but they often need to then be finetuned on hundreds of thousands of labeled CXRs to achieve high performance. Therefore, the current approach to model development is not viable on tasks with only a small amount of labeled data. An emerging method for reducing reliance on large amounts of labeled data is self-supervised learning (SSL), which uses unlabeled CXR datasets to automatically learn features that can be leveraged for downstream interpretation tasks. In this work, we investigated whether self-supervised pretraining methods could outperform traditional ImageNet pretraining for chest X-ray interpretation. We found that SSL-pretrained models outperformed ImageNet-pretrained models on thirteen different datasets representing high diversity in geographies, clinical settings, and prediction tasks. We thus show that SSL on unlabeled CXR data is a promising pretraining approach for a wide variety of CXR interpretation tasks, enabling a shift away from costly labeled datasets.
BibTeX
@misc{iyer2022self,
  author = {Niveditha S. Iyer and Aditya Gulati and Oishi Banerjee and Cécile Logé and Maha Farhat and Agustina Saenz and Pranav Rajpurkar},
  title = {{Self-Supervised Pretraining Enables High-Performance Chest X-Ray Interpretation Across Clinical Distributions}},
  year = {2022},
  doi = {10.1101/2022.11.19.22282519},
  howpublished = {medRxiv},
}
Figure