Abstract
Biomedical Visual Question Answering (VQA) is often hindered by the scarcity of labeled data and the complexity of clinical images. This paper introduces RAMM, a retrieval-augmented model designed to mimic clinical decision-making by consulting historical cases. By leveraging a massive new dataset, PMCPM, and a novel retrieval-attention mechanism, RAMM achieves state-of-the-art performance across multiple medical benchmarks, effectively bridging the knowledge gap in automated medical image interpretation.
Introduction
Visual Question Answering (VQA) in the biomedical domain presents unique challenges compared to general-domain VQA. While general models benefit from millions of training pairs, biomedical datasets are typically limited to a few thousand. This scarcity leads to overfitting and a lack of deep clinical context necessary for interpreting radiology images like CT, MRI, and X-rays.
Traditionally, AI models operate in a "closed-book" fashion, relying solely on patterns learned during training. In contrast, medical professionals often employ case-based reasoning, searching for similar historical examples to inform a diagnosis. RAMM (Retrieval-augmented Multi-modal model) formalizes this process for AI, allowing models to "look up" relevant medical literature dynamically.
Methodology
The RAMM framework operates through a two-stage paradigm: multi-modal pre-training followed by retrieval-augmented fine-tuning.
Multi-modal Pre-training
The model is first pre-trained using three standard tasks: Image-Text Contrastive (ITC) learning to align visual and textual embeddings, Image-Text Matching (ITM), and Masked Language Modeling (MLM). The architecture utilizes Swin-Base as the image encoder and PubMedBERT as the text encoder to ensure high-quality feature extraction from both modalities.
Retrieval-Augmented Fine-tuning
The core innovation of RAMM lies in its retrieval mechanism during the fine-tuning phase.
- The Retriever: Given a query image, the model identifies the top-$r$ most similar image-text pairs from an external database (e.g., PMCPM, ROCO, MIMIC-CXR).
- Retrieval-Attention Module: A specialized module is integrated into the multi-modal encoder. It treats the current image and question features as "queries" to attend to the retrieved images and captions, which serve as "keys" and "values." This enables the model to incorporate external clinical knowledge directly into its reasoning process.
Datasets
A key contribution of this work is the introduction of PMCPM (PMC Patients-Multi-modal). This dataset contains 398,109 high-quality, patient-based image-text pairs filtered from PubMed Central Case Reports. Unlike previous datasets, PMCPM emphasizes diverse clinical conditions and detailed patient narratives, providing a richer context for retrieval.
Evaluation was conducted on four standard benchmarks: Med-VQA 2019, Med-VQA 2021, VQARAD, and SLAKE (English version).
Key Insights
- Case-Based Reasoning: Retrieval effectively simulates clinical experience. For instance, when presented with an ambiguous MRI, the model can retrieve similar cases where the caption provides a definitive diagnosis.
- Image-Centric Retrieval: The authors discovered that using only the image for retrieval outperformed using both the image and question. Biomedical questions are often generic (e.g., "What is the abnormality?"), whereas the image contains the specific diagnostic signals needed to find relevant literature.
- Optimal Retrieval Volume: Performance peaks when retrieving 2 to 4 pairs. Retrieving more (e.g., 8 pairs) introduces noise that can degrade accuracy.
Results
RAMM established new State-of-the-Art (SOTA) benchmarks across all tested datasets:
- Med-VQA 2019: 82.13% accuracy (prev. best 79.87%).
- VQARAD: 78.27% accuracy.
- SLAKE: 86.05% accuracy.
- Med-VQA 2021: 39.20% accuracy, outperforming top ensemble methods from the original challenge.
Ablation studies confirmed that the retrieval mechanism consistently provided a 1.5% to 2% performance boost over non-retrieval baselines.
Conclusion and Future Work
RAMM demonstrates the power of retrieval-augmented learning in specialized domains like medicine. While the model significantly improves VQA performance, challenges remain regarding inference latency and the scope of the retrieval source. Future research may focus on scaling the retriever to the entire PubMed database and improving performance on complex, open-ended clinical questions.
Summary updated