Local Contrastive Learning for Medical Image Recognition

S. A. Rizvi, Ruixiang Tang, X. Jiang, X. Ma, X. HuarXiv 2023 · DOI · arXiv

Introduction

Deep learning has significantly advanced medical imaging, yet it remains constrained by a significant "data hunger" problem. Training robust models typically requires vast quantities of images meticulously labeled by expert radiologists. While self-supervised models like CLIP have attempted to mitigate this by learning from raw radiology reports, they often focus on "global" image features. In clinical practice, however, the diagnostic signal frequently resides in subtle, localized anomalies—such as a faint shadow in a lung apex—that global models may overlook or dismiss as noise. Furthermore, the "black box" nature of these models poses a challenge for clinical adoption, as clinicians require interpretable evidence for AI-generated diagnoses.

The LRCLR Framework

To address these challenges, researchers introduced Local Region Contrastive Learning (LRCLR). This plug-and-play framework is designed to enhance existing vision-language models by enabling them to "zoom in" on clinically relevant local regions without requiring manual bounding box annotations during training.

The LRCLR architecture consists of an image encoder with local region selection, a text encoder, and a cross-modality transformer to align local patches with descriptive text.

The framework operates through two primary mechanisms:

  1. Unsupervised Local Region Selection: Leveraging the internal self-attention mechanism of a Vision Transformer (ViT), the model identifies significant local patches. Rather than relying solely on the final layer, LRCLR calculates a cumulative attention map by multiplying attention matrices across all encoder layers ($a_{final} = \prod a_l$). This captures the propagation of information through the network, identifying regions most critical to the overall image context.
  2. Cross-Modality Transformer: Selected local regions are processed alongside text tokens from radiology reports. This allows the model to explicitly learn the correspondence between specific medical phrases (e.g., "pleural effusion") and their corresponding visual manifestations in the image.

The training objective combines a global contrastive loss (matching the entire image to the full report) with a local contrastive loss (aligning specific patches with relevant text segments).

Experimental Results

The model was fine-tuned on the MIMIC-CXR dataset, comprising over 377,000 chest X-rays and reports. Evaluation was conducted on the CheXpert test set to measure zero-shot performance across 14 medical observations, where the model must identify pathologies without having seen explicit labels during training.

Comparison of zero-shot performance (Mean AUC) between the CheXzero baseline and CheXzero augmented with LRCLR across various medical findings.

Integrating LRCLR into the CheXzero baseline resulted in a significant performance boost, raising the average AUC from 76.24 to 78.66. The most substantial improvements were observed in localized pathologies that are often difficult for global models to detect:

  • Fractures: AUC increased by 27.2 points (from 44.9 to 72.1).
  • Pneumothorax: AUC increased by 13.0 points (from 62.3 to 75.3).
  • No Finding: AUC increased by 7.6 points (from 75.7 to 83.3).

Beyond quantitative metrics, LRCLR provides inherent interpretability. The attention weights generated during the cross-modality alignment act as heatmaps, allowing radiologists to see exactly which parts of an X-ray influenced the model's prediction, such as highlighting rib regions for fractures or lower lung areas for opacities.

Conclusion

LRCLR demonstrates that local alignment is a powerful tool for improving the accuracy and transparency of medical AI. By repurposing the internal attention of transformers for unsupervised region selection, the framework achieves state-of-the-art zero-shot performance while providing the explainability necessary for clinical trust. Future research will explore extending this modular approach to other imaging modalities, such as MRI and CT scans, and refining region selection in the presence of complex, overlapping pathologies.

Summary updated

Abstract

The proliferation of Deep Learning (DL)-based methods for radiographic image analysis has created a great demand for expert-labeled radiology data. Recent self-supervised frameworks have alleviated the need for expert labeling by obtaining supervision from associated radiology reports. These frameworks, however, struggle to distinguish the subtle differences between different pathologies in medical images. Additionally, many of them do not provide interpretation between image regions and text, making it difficult for radiologists to assess model predictions. In this work, we propose Local Region Contrastive Learning (LRCLR), a flexible fine-tuning framework that adds layers for significant image region selection as well as cross-modality interaction. Our results on an external validation set of chest x-rays suggest that LRCLR identifies significant local image regions and provides meaningful interpretation against radiology text while improving zero-shot performance on several chest x-ray medical findings.
BibTeX
@misc{rizvi2023local,
  author = {S. A. Rizvi and Ruixiang Tang and X. Jiang and X. Ma and X. Hu},
  title = {{Local Contrastive Learning for Medical Image Recognition}},
  year = {2023},
  eprint = {2303.14153},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.14153},
}
Figure