Abstract
Vision-Language Pretraining (VLP) in the medical domain often fails to capture the intricate, structured relationships between radiology reports and clinical images. We present Adaptive Group Alignment (AGA), a framework that utilizes dynamic thresholding and instance-aware loss to model these complex dependencies. By moving beyond simple word-to-patch matching, AGA achieves state-of-the-art performance on medical image-text retrieval and classification tasks without the need for massive negative sampling.
Introduction
In medical AI, training models to understand the relationship between images (e.g., chest X-rays) and their corresponding textual reports is a cornerstone of Vision-Language Pretraining. However, existing VLP methods face two significant hurdles. First, they often oversimplify the data, treating reports as either a single global vector or a collection of isolated words. This ignores the structured nature of medical findings, where a phrase like "hazy opacity in the right lung" describes a regional phenomenon rather than a single pixel-patch. Second, most models rely on contrastive learning, which requires a vast number of "negative samples" to learn effectively—a requirement that is difficult to meet with specialized, smaller medical datasets.
Methodology
The AGA framework introduces three primary innovations to address these structural and data-efficiency challenges:
1. Adaptive Grouping with Threshold Gates
Rather than forcing a rigid 1-to-1 mapping between words and image patches, AGA creates dynamic "groups." A single clinical term may be associated with a cluster of patches, and vice versa. The framework employs Language-grouped and Vision-grouped Threshold Gates, which use a momentum-based mechanism to learn the optimal similarity threshold for group membership. This allows the model to adapt to the varying density of information across different modalities (e.g., X-rays vs. Ultrasound).
2. Instance-aware Group Alignment (IGA) Loss
To solve the negative sample problem, the authors developed the IGA loss. This loss function operates entirely within a single image-text pair (intra-instance). It aligns a specific token with its corresponding group while simultaneously contrasting it against other groups within the same instance. This self-sufficient approach eliminates the dependency on external negative samples from other patients, making the model highly efficient on smaller datasets.
3. Bidirectional Cross-modal Grouped Alignment (BCGA)
This module utilizes a cross-attention mechanism to refine group representations. It ensures that visual and linguistic groups are mathematically aligned within a shared latent space, facilitating more accurate cross-modal retrieval.
Experimental Results
AGA was pretrained on the public MIMIC-CXR dataset and a private dataset of Endoscopic Ultrasound (SMT) images. Evaluation across several benchmarks demonstrated its superiority over existing models like ConVIRT and MGCA.
- Retrieval Performance: On the CheXpert dataset, AGA achieved a Precision@5 of 50.28%, outperforming the previous state-of-the-art by nearly 1%.
- Zero-Shot Classification: Without explicit label training, AGA reached 63.6% accuracy on CheXpert and 51.1% on RSNA Pneumonia, showcasing the robustness of its learned representations.
- Data Efficiency: When fine-tuned on only 1% of labeled data, AGA maintained a significant performance lead, proving that its structured grouping strategy captures more clinically relevant features than traditional methods.
Discussion and Conclusion
The success of AGA underscores that structure matters in medical VLP. Clinical findings are regional and descriptive; by treating them as groups rather than isolated tokens, AGA more accurately reflects the "logic" of a radiologist's report. Furthermore, the dynamic thresholding revealed that different medical modalities require different levels of sparsity—chest X-rays benefit from lower thresholds due to their descriptive nature, while ultrasound requires tighter, more direct correlations.
Future work may involve extending this grouping strategy to the sample level to align similar patient cases across entire datasets or integrating generative pre-training to further enhance the model's understanding of medical narratives.
Summary updated