The Role of Local Alignment and Uniformity in Image-Text Contrastive Learning on Medical Images

Philip Müller, Georgios Kaissis, Daniel RueckertarXiv 2022 · DOI · arXiv

Abstract

Contrastive learning models like CLIP often lack the local awareness necessary for medical imaging tasks such as lesion detection or organ segmentation. While previous methods like LoVT introduced complex local alignment losses, this work demonstrates that the primary benefit of these losses stems from enforcing local uniformity. By replacing alignment with a simple Gaussian uniformity regularizer, the authors achieve state-of-the-art performance on localized tasks, simplifying the pretraining process while maintaining high performance.

Introduction

General-purpose vision-language models rely on global contrastive losses to align entire images with text descriptions. However, medical applications require fine-grained, local awareness. Previous state-of-the-art methods addressed this by introducing local contrastive losses that match image patches to specific sentences in radiology reports. While effective, these methods are computationally intensive and theoretically opaque. This research explores whether the complexity of local alignment is necessary or if a simpler principle—uniformity—is the true driver of success.

The Uniformity Hypothesis

Contrastive learning forces can be categorized into two components: Alignment, which pulls related representations together, and Uniformity, which ensures representations are spread evenly across the feature space to avoid collapse.

We hypothesize that while global alignment is essential, the "local" version of alignment (patch-to-sentence) is largely redundant. Instead, the critical contribution of local losses is local uniformity—preventing the model from collapsing all local features into a single global representation. To test this, we introduced LoVT-uni-gauss, a model that replaces complex alignment machinery with a Gaussian uniformity loss acting as a regularizer.

Impact of different uniformity regularizers on the mean per-sample uniformity of image features.

As shown in Figure 1, the introduction of Gaussian and Cross-Entropy uniformity regularizers significantly increases the mean per-sample uniformity, ensuring that different regions of a chest X-ray maintain distinct feature representations.

Experimental Setup

The models were pretrained on 30% of the MIMIC-CXRv2 dataset. Evaluation was conducted on a rigorous suite of 18 localized tasks across five public datasets, including RSNA Pneumonia Detection, SIIM Pneumothorax, and NIH CXR lung segmentation. These tasks require the model to identify or segment specific anatomical structures, providing a robust test for local feature quality.

Key Results

The uniformity-based approach proved highly effective, outperforming standard global-only methods (CLIP and ConVIRT) on 12 out of 18 tasks.

Performance comparison of uniformity-based methods across 18 localized medical imaging tasks.

Table 4 summarizes the comparative performance. The uniformity method frequently outperformed CLIP and was highly competitive with the full LoVT model. Notably, on tasks where local awareness is paramount, the simple uniformity regularizer captured the majority of the performance gains previously attributed to complex local alignment.

Discussion and Limitations

Our findings suggest that enforcing diversity within a sample is more critical than explicit cross-modal patch matching for many medical tasks. By decoupling the modalities and enforcing uniformity independently on images and text, we can simplify the training objective.

However, the original LoVT still performed best on 9 of the 18 tasks, indicating that for certain highly specific pathologies, the explicit patch-to-sentence mapping provides unique semantic information. Additionally, performance is sensitive to the "local temperature" hyperparameter. Future work will focus on content-aware regularization strategies to automatically adapt these parameters.

Conclusion

This work demonstrates that local uniformity is a critical factor for localized medical AI. By prioritizing representation diversity over complex alignment, researchers can develop simpler, more efficient models that excel at the fine-grained tasks required in clinical settings.

Summary updated

Abstract

Image-text contrastive learning has proven effective for pretraining medical image models. When targeting localized downstream tasks like semantic segmentation or object detection, additional local contrastive losses that align image regions with sentences have shown promising results. We study how local contrastive losses are related to global (per-sample) contrastive losses and which effects they have on localized medical downstream tasks. Based on a theoretical comparison, we propose to remove some components of local losses and replace others by a novel distribution prior which enforces uniformity of representations within each sample. We empirically study this approach on chest X-ray tasks and find it to be very effective, outperforming methods without local losses on 12 of 18 tasks.
BibTeX
@misc{muller2022role,
  author = {Philip Müller and Georgios Kaissis and Daniel Rueckert},
  title = {{The Role of Local Alignment and Uniformity in Image-Text Contrastive Learning on Medical Images}},
  year = {2022},
  eprint = {2211.07254},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.07254},
}
Figure