Uni-Mlip: Unified Self-supervision for Medical Vision Language Pre-training

Ameera Bawazir, Kebin Wu, Wenbin Li

Abstract

Recent advancements in vision-language pre-training via contrastive learning have significantly improved performance across computer vision tasks. However, in the medical domain, obtaining multimodal data is often costly and challenging due to privacy, sensitivity, and annotation complexity. To mitigate data scarcity while boosting model performance, we introduce Uni-Mlip, a unified self-supervision framework specifically designed to enhance medical vision-language pre-training. Uni-Mlip seamlessly integrates cross-modality, uni-modality, and fused-modality self-supervision techniques at the data-level and the feature-level. Additionally, Uni-Mlip tailors uni-modal image self-supervision to accommodate the unique characteristics of medical images. Our experiments across datasets of varying scales demonstrate that Uni-Mlip significantly surpasses current state-of-the-art methods in three key downstream tasks: image-text retrieval, image classification, and visual question answering (VQA).

Introduction

Vision-Language Pre-training (VLP) models like CLIP have revolutionized general computer vision, but the medical domain faces a unique "data desert." Multimodal medical data is scarce, expensive to annotate, and fraught with privacy concerns. Furthermore, standard self-supervised learning (SSL) techniques often fail when applied to medical imaging. In natural images, changing brightness or contrast rarely alters the semantic category. In medical imaging, however, absolute pixel intensity and fine-grained local details are primary indicators of disease. Standard augmentations can inadvertently "wash out" these diagnostic signals, leading to poor model performance.

To address these challenges, we introduce Uni-Mlip, a unified framework that integrates self-supervision at three distinct levels: cross-modal, uni-modal, and fused-modal.

Methodology

Uni-Mlip maximizes information extraction from limited data through a multi-tiered self-supervision strategy:

  1. Cross-modal SSL: In addition to standard Image-Text Contrastive (ITC) loss, we introduce Feature-level Perturbation (FP). By applying DropBlock to image features and Dropout to text features, we force the model to maintain semantic alignment even when the representation space is noisy.
  2. Uni-modal SSL (Image): We incorporate a SimCLR-style contrastive loss. To solve the "intensity washout" problem, we propose a two-stage training trick: we first train without SimCLR, then freeze the mean and variance in Batch Normalization (BN) layers while keeping weights and biases learnable. This allows the model to handle strong augmentations without corrupting the global intensity statistics essential for diagnosis.
  3. Fused-modal SSL: We utilize Masked Language Modeling (MLM), where the model predicts missing text tokens by observing both partial text and corresponding image features through a multimodal fusion transformer.
The Uni-Mlip architecture integrates multiple levels of self-supervision, including feature perturbation and masked language modeling.

Experimental Setup

The model was pre-trained on PMC-OA, a dataset of 1.65 million medical image-caption pairs. Evaluation was conducted across several benchmarks:

  • Retrieval: ROCO (Radiology Objects in COntext).
  • Classification: NIH (ChestX-ray8), CheXpert, and MIMIC-CXR.
  • VQA: VQA-RAD and SLAKE.

Results and Discussion

Uni-Mlip consistently outperformed previous state-of-the-art (SOTA) models like PMC-CLIP and M3AE across all tasks.

Retrieval Performance

On the ROCO dataset, Uni-Mlip achieved a significant 3.17% improvement in Recall@10 for image-to-text retrieval compared to the previous best. This demonstrates the effectiveness of aligning features at multiple levels of granularity.

Evaluation results showing Uni-Mlip outperforming various baselines across different Recall metrics for retrieval tasks.

Efficiency and VQA

The framework shows significant promise in low-data regimes. When fine-tuned on only 10% of the MIMIC dataset, Uni-Mlip reached an AUC of 79.1%, a 1.7% gain over PMC-CLIP. In VQA tasks, the model showed substantial gains in closed-ended questions, improving accuracy by 1.48% on VQA-RAD and 1.44% on Slake over the baseline.

Conclusion

The success of Uni-Mlip highlights that the failure of standard SSL in medical imaging is often due to how augmentations interact with Batch Normalization. By freezing BN statistics, we can leverage robust contrastive learning without losing diagnostic integrity. Future work will explore scaling to larger backbones and testing on specialized modalities like 3D MRI volumes.

Summary updated

Abstract

Recent advancements in vision-language pre-training via contrastive learning have significantly improved performance across computer vision tasks. However, in the medical domain, obtaining multimodal data is often costly and challenging due to privacy, sensitivity, and annotation complexity. To mitigate data scarcity while boosting model performance, we introduce \textbf{Uni-Mlip}, a unified self-supervision framework specifically designed to enhance medical vision-language pre-training. Uni-Mlip seamlessly integrates cross-modality, uni-modality, and fused-modality self-supervision techniques at the data-level and the feature-level. Additionally, Uni-Mlip tailors uni-modal image self-supervision to accommodate the unique characteristics of medical images. Our experiments across datasets of varying scales demonstrate that Uni-Mlip significantly surpasses current state-of-the-art methods in three key downstream tasks: image-text retrieval, image classification, and visual question answering (VQA).
BibTeX
@misc{bawazir2024uni,
  author = {Ameera Bawazir and Kebin Wu and Wenbin Li},
  title = {{Uni-Mlip: Unified Self-supervision for Medical Vision Language Pre-training}},
  year = {2024},
  eprint = {2411.15207},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.15207},
}
Figure