Abstract
As medical diagnostics increasingly rely on multimodal data—integrating high-resolution imaging with complex clinical narratives—the demand for AI systems capable of joint visual and textual reasoning has surged. This survey provides a comprehensive roadmap of Medical Vision-and-Language Models (MVLMs), categorizing architectures, datasets, and evaluation metrics across five core clinical tasks: report generation, visual question answering (VQA), segmentation, retrieval, and diagnosis. We highlight the shift from traditional CNN-based pipelines to modern Large Language Model (LLM) integrations while identifying critical gaps in clinical reliability, interpretability, and evaluation.
Introduction: The Multimodal Challenge in Healthcare
General-purpose vision-language models, such as CLIP or GPT-4V, often struggle in the medical domain due to a lack of specialized terminology and an inability to discern subtle pathological patterns. Medical data presents unique hurdles that necessitate "purpose-built" models:
- Data Scarcity & Privacy: Stringent regulations like HIPAA limit the availability of large-scale, annotated medical datasets.
- High Stakes: Errors in medical AI, such as an incorrect radiology report, have direct implications for patient safety.
- Heterogeneity: Models must process diverse modalities (X-ray, CT, MRI, pathology) and varied clinical documentation styles.
The field of MVLMs has seen explosive growth as researchers attempt to bridge these gaps by integrating medical domain knowledge into multimodal architectures.
The MVLM Architecture: From Input to Application
The survey decomposes MVLM architectures into three functional stages: Input (feature extraction), Network (cross-modal integration), and Output (task-specific generation or classification).
Technical Trends in Model Design
- Visual Encoding: There is a clear transition from traditional Convolutional Neural Networks (CNNs) like ResNet to Vision Transformers (ViT) and hierarchical encoders (e.g., HRNet), which better capture global and local features in medical images.
- Cross-Modal Alignment: Inspired by CLIP, contrastive learning is widely used to map images and text into a shared feature space. Furthermore, the integration of Medical Knowledge Graphs provides essential "common sense" clinical context that raw data alone may lack.
- Decoding and Generation: For Medical Report Generation (MRG), the field has moved from simple LSTMs to hierarchical decoders and LLM-based architectures (e.g., LLaMA, GPT-4). These models leverage prompt engineering to produce fluent, structured clinical reports.
- Fusion Techniques: In Medical VQA, methods like Bilinear Attention Networks (BAN) are employed to capture the complex, fine-grained interactions between image pixels and clinical queries.
Core Clinical Tasks and Datasets
The survey identifies several benchmark datasets that have become the "gold standards" for evaluating MVLMs:
- Report Generation: MIMIC-CXR (~377k images), IU X-ray, and PadChest.
- Medical VQA: VQA-RAD, SLAKE (bilingual), and PathVQA (pathology-specific).
- Segmentation & Retrieval: IDRiD (retinal), the Medical Segmentation Decathlon (MSD), and ROCO for multimodal retrieval.
Key Insights: The LLM Revolution and Domain Knowledge
The integration of LLMs has significantly improved the linguistic fluency of medical AI. However, this progress comes with the risk of "hallucinations"—the generation of medically plausible but factually incorrect statements. The most successful models mitigate this by explicitly incorporating medical ontologies or "prior knowledge" modules to focus the model's attention on clinically relevant regions.
Another emerging frontier is Text-Guided Segmentation, where models use clinical text (e.g., "find the nodule in the left lower lobe") to perform zero-shot or highly specific segmentation, moving beyond static, predefined classes.
Challenges and Future Directions
Despite rapid progress, several bottlenecks remain:
- The Evaluation Gap: Standard NLP metrics (BLEU, ROUGE) are insufficient for medicine. New metrics are needed to prioritize clinical correctness over grammatical fluency.
- Multi-round Dialogue: Current VQA models are largely "one-shot." Real-world clinical workflows require interactive, back-and-forth dialogue between clinicians and AI.
- Interpretability: For clinical adoption, models must be "explainable," allowing doctors to understand the reasoning behind a specific diagnosis or flagged region.
- Global Bias: Most current datasets are English-centric, necessitating more diverse, multilingual data to ensure equitable performance across different clinical environments.
Summary updated