OTCXR: Rethinking Self-Supervised Alignment using Optimal Transport for Chest X-ray Analysis

Vandan Gorade, Azad Singh, Deepak MishraWACV 2025

TL;DR

This paper introduces OTCXR, a novel self-supervised learning (SSL) framework for chest X-ray analysis that leverages Optimal Transport (OT) to achieve "dense semantic invariance." By focusing on aligning fine-grained features across different views, OTCXR significantly improves representation learning for medical imaging tasks, especially in low-data regimes.

What Problem Does This Paper Address?

Analyzing chest X-rays (CXRs) requires a deep understanding of subtle anatomical structures and pathological indicators. Traditional self-supervised learning (SSL) methods, while effective in many domains, often struggle with CXRs because they tend to treat images as collections of isolated features, missing crucial local details and global contextual dependencies. This can lead to representations that overlook clinically relevant patterns. Furthermore, CXRs exhibit significant heterogeneity (variations in appearance) and similarity (different conditions looking alike), making accurate feature alignment a challenge. The paper aims to bridge this gap by developing an SSL approach that can capture precise anatomical and pathological attributes, regardless of variations in viewpoints, orientations, or imaging conditions.

Methods

The core of OTCXR is its novel integration of Optimal Transport (OT) into an SSL framework. Instead of relying on traditional invariance-based alignment, OTCXR formulates dense semantic invariance as an OT problem. This means it learns to align features at a fine-grained level, considering the spatial relationships between pixels.

Key components of OTCXR include:

  • Dense Feature Maps: The framework operates on dense feature maps (outputs of CNN encoders before pooling) rather than just pooled features. This preserves spatial information, which is vital for CXR analysis.
  • Optimal Transport (OT) for Alignment: OT is used to find the optimal "transport plan" that minimizes the cost of transforming features from one image view to another. This allows for a more granular and accurate estimation of feature alignment.
  • Cross-Viewpoint Semantics Infusion Module (CV-SIM): This module enhances the OT approach by capturing global semantic relationships across different viewpoints. It uses a multi-head cross-view attention mechanism to refine feature representations.
  • Variance and Covariance Regularization: Inspired by VICReg, OTCXR incorporates variance and covariance regularizations to prevent model collapse and ensure that learned representations are informative and less redundant.

Datasets

The paper uses the following datasets for training and evaluation:

  • NIH Chest X-ray14: A large dataset with over 112,000 CXR images labeled with 14 thoracic pathologies. Used for pre-training and evaluation.
  • VinBig-CXR: A dataset containing 18,000 CXR images annotated with 14 pathologies. Used for evaluation.
  • RSNA Pneumonia Dataset: A dataset of 30,000 chest radiographs annotated for pneumonia. Used for evaluation.
  • SIIM-ACR Pneumothorax Dataset: Used for downstream segmentation evaluation.

Key Insights and Ideas

  • Dense Semantic Invariance is Key for CXRs: The paper argues that for medical imaging like CXRs, simply enforcing invariance at a global level isn't enough. Capturing fine-grained, dense semantic relationships across different views is crucial for accurate diagnosis.
  • Optimal Transport as a Powerful Alignment Tool: OT proves to be a flexible and effective mechanism for aligning these dense features, going beyond simple cosine similarity or contrastive losses.
  • CV-SIM Enhances Global Context: The CV-SIM module is essential for enriching the feature representations by incorporating global contextual dependencies, which OT alone might miss.
  • Regularization Prevents Collapse and Retains Information: The inclusion of variance and covariance regularization is vital for maintaining representation diversity and preventing information loss, a common pitfall in SSL.

Key Results

OTCXR consistently outperforms existing state-of-the-art SSL methods and even supervised baselines across various chest X-ray classification tasks on the NIH, VinBig-CXR, and RSNA datasets.

  • Classification Performance: Under semi-supervised linear evaluation (both frozen and fine-tuning protocols), OTCXR achieves the highest AUC scores. For example, on the NIH dataset with 1% labeled data, OTCXR achieved an AUC of 68.8%, outperforming the next best SSL method (DenseCL) by 1.1%.
  • Transferability: OTCXR demonstrates strong transferability to other datasets and tasks, including segmentation. It achieved the highest Dice coefficient (0.586) on the SIIM-ACR Pneumothorax segmentation task after fine-tuning representations pre-trained on NIH.
  • Qualitative Results: Diagnostic heat maps generated by OTCXR show more precise alignment with ground truth bounding boxes, indicating better interpretability and identification of diagnostically significant regions compared to baseline methods.

Limitations and Future Work

  • Sensitivity to Marginal Constraints: The authors note that the OT solution's sensitivity to marginal constraints derived from CV-SIM could be a limitation, potentially impacting performance on rare or subtle pathologies where global context might be less informative.
  • Generalizability Across Imaging Protocols: The framework's effectiveness might vary across different imaging protocols or equipment, potentially requiring fine-tuning for diverse clinical settings.
  • Computational Overhead: While minimal, OTCXR does introduce a slight increase in memory requirements due to the additional heads (Rs and Rt) compared to some existing SSL methods.

Future work could explore extending OTCXR to other medical imaging modalities and investigating its performance in more challenging scenarios with even scarcer labeled data.

Summary updated

BibTeX
@inproceedings{gorade2025otcxr,
  author = {Vandan Gorade and Azad Singh and Deepak Mishra},
  title = {{OTCXR: Rethinking Self-Supervised Alignment using Optimal Transport for Chest X-ray Analysis}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure