Abstract
Contrastive learning models like CLIP often lack the local awareness necessary for medical imaging tasks such as lesion detection or organ segmentation. While previous methods like LoVT introduced complex local alignment losses, this work demonstrates that the primary benefit of these losses stems from enforcing local uniformity. By replacing alignment with a simple Gaussian uniformity regularizer, the authors achieve state-of-the-art performance on localized tasks, simplifying the pretraining process while maintaining high performance.
Introduction
General-purpose vision-language models rely on global contrastive losses to align entire images with text descriptions. However, medical applications require fine-grained, local awareness. Previous state-of-the-art methods addressed this by introducing local contrastive losses that match image patches to specific sentences in radiology reports. While effective, these methods are computationally intensive and theoretically opaque. This research explores whether the complexity of local alignment is necessary or if a simpler principle—uniformity—is the true driver of success.
The Uniformity Hypothesis
Contrastive learning forces can be categorized into two components: Alignment, which pulls related representations together, and Uniformity, which ensures representations are spread evenly across the feature space to avoid collapse.
We hypothesize that while global alignment is essential, the "local" version of alignment (patch-to-sentence) is largely redundant. Instead, the critical contribution of local losses is local uniformity—preventing the model from collapsing all local features into a single global representation. To test this, we introduced LoVT-uni-gauss, a model that replaces complex alignment machinery with a Gaussian uniformity loss acting as a regularizer.
As shown in Figure 1, the introduction of Gaussian and Cross-Entropy uniformity regularizers significantly increases the mean per-sample uniformity, ensuring that different regions of a chest X-ray maintain distinct feature representations.
Experimental Setup
The models were pretrained on 30% of the MIMIC-CXRv2 dataset. Evaluation was conducted on a rigorous suite of 18 localized tasks across five public datasets, including RSNA Pneumonia Detection, SIIM Pneumothorax, and NIH CXR lung segmentation. These tasks require the model to identify or segment specific anatomical structures, providing a robust test for local feature quality.
Key Results
The uniformity-based approach proved highly effective, outperforming standard global-only methods (CLIP and ConVIRT) on 12 out of 18 tasks.
Table 4 summarizes the comparative performance. The uniformity method frequently outperformed CLIP and was highly competitive with the full LoVT model. Notably, on tasks where local awareness is paramount, the simple uniformity regularizer captured the majority of the performance gains previously attributed to complex local alignment.
Discussion and Limitations
Our findings suggest that enforcing diversity within a sample is more critical than explicit cross-modal patch matching for many medical tasks. By decoupling the modalities and enforcing uniformity independently on images and text, we can simplify the training objective.
However, the original LoVT still performed best on 9 of the 18 tasks, indicating that for certain highly specific pathologies, the explicit patch-to-sentence mapping provides unique semantic information. Additionally, performance is sensitive to the "local temperature" hyperparameter. Future work will focus on content-aware regularization strategies to automatically adapt these parameters.
Conclusion
This work demonstrates that local uniformity is a critical factor for localized medical AI. By prioritizing representation diversity over complex alignment, researchers can develop simpler, more efficient models that excel at the fine-grained tasks required in clinical settings.
Summary updated