A Survey of Medical Vision-and-Language Applications and Their Techniques

Qi Chen, Ruoshan Zhao, Sinuo Wang, Vu Minh Hieu Phan, Anton van den Hengel, Johan Verjans, Zhibin Liao, Minh-Son To, Yong Xia, Jian Chen, Yutong Xie, Qi WuarXiv 2024 · DOI · OpenReview

Abstract

As medical diagnostics increasingly rely on multimodal data—integrating high-resolution imaging with complex clinical narratives—the demand for AI systems capable of joint visual and textual reasoning has surged. This survey provides a comprehensive roadmap of Medical Vision-and-Language Models (MVLMs), categorizing architectures, datasets, and evaluation metrics across five core clinical tasks: report generation, visual question answering (VQA), segmentation, retrieval, and diagnosis. We highlight the shift from traditional CNN-based pipelines to modern Large Language Model (LLM) integrations while identifying critical gaps in clinical reliability, interpretability, and evaluation.

Introduction: The Multimodal Challenge in Healthcare

General-purpose vision-language models, such as CLIP or GPT-4V, often struggle in the medical domain due to a lack of specialized terminology and an inability to discern subtle pathological patterns. Medical data presents unique hurdles that necessitate "purpose-built" models:

  • Data Scarcity & Privacy: Stringent regulations like HIPAA limit the availability of large-scale, annotated medical datasets.
  • High Stakes: Errors in medical AI, such as an incorrect radiology report, have direct implications for patient safety.
  • Heterogeneity: Models must process diverse modalities (X-ray, CT, MRI, pathology) and varied clinical documentation styles.

The field of MVLMs has seen explosive growth as researchers attempt to bridge these gaps by integrating medical domain knowledge into multimodal architectures.

Trends in MVLM research: (a) total publications per year, (b) growth of specific tasks, and (c) the evolving proportion of research focus.

The MVLM Architecture: From Input to Application

The survey decomposes MVLM architectures into three functional stages: Input (feature extraction), Network (cross-modal integration), and Output (task-specific generation or classification).

A comprehensive overview of the MVLM pipeline, illustrating the flow from raw medical data and preprocessing to network integration and clinical output.
  1. Visual Encoding: There is a clear transition from traditional Convolutional Neural Networks (CNNs) like ResNet to Vision Transformers (ViT) and hierarchical encoders (e.g., HRNet), which better capture global and local features in medical images.
  2. Cross-Modal Alignment: Inspired by CLIP, contrastive learning is widely used to map images and text into a shared feature space. Furthermore, the integration of Medical Knowledge Graphs provides essential "common sense" clinical context that raw data alone may lack.
  3. Decoding and Generation: For Medical Report Generation (MRG), the field has moved from simple LSTMs to hierarchical decoders and LLM-based architectures (e.g., LLaMA, GPT-4). These models leverage prompt engineering to produce fluent, structured clinical reports.
  4. Fusion Techniques: In Medical VQA, methods like Bilinear Attention Networks (BAN) are employed to capture the complex, fine-grained interactions between image pixels and clinical queries.

Core Clinical Tasks and Datasets

The survey identifies several benchmark datasets that have become the "gold standards" for evaluating MVLMs:

  • Report Generation: MIMIC-CXR (~377k images), IU X-ray, and PadChest.
  • Medical VQA: VQA-RAD, SLAKE (bilingual), and PathVQA (pathology-specific).
  • Segmentation & Retrieval: IDRiD (retinal), the Medical Segmentation Decathlon (MSD), and ROCO for multimodal retrieval.

Key Insights: The LLM Revolution and Domain Knowledge

The integration of LLMs has significantly improved the linguistic fluency of medical AI. However, this progress comes with the risk of "hallucinations"—the generation of medically plausible but factually incorrect statements. The most successful models mitigate this by explicitly incorporating medical ontologies or "prior knowledge" modules to focus the model's attention on clinically relevant regions.

Another emerging frontier is Text-Guided Segmentation, where models use clinical text (e.g., "find the nodule in the left lower lobe") to perform zero-shot or highly specific segmentation, moving beyond static, predefined classes.

Challenges and Future Directions

Despite rapid progress, several bottlenecks remain:

  1. The Evaluation Gap: Standard NLP metrics (BLEU, ROUGE) are insufficient for medicine. New metrics are needed to prioritize clinical correctness over grammatical fluency.
  2. Multi-round Dialogue: Current VQA models are largely "one-shot." Real-world clinical workflows require interactive, back-and-forth dialogue between clinicians and AI.
  3. Interpretability: For clinical adoption, models must be "explainable," allowing doctors to understand the reasoning behind a specific diagnosis or flagged region.
  4. Global Bias: Most current datasets are English-centric, necessitating more diverse, multilingual data to ensure equitable performance across different clinical environments.

Summary updated

Abstract

Medical vision-and-language models (MVLMs) have attracted substantial interest due to their capability to offer a natural language interface for interpreting complex medical data. Their applications are versatile and have the potential to improve diagnostic accuracy and decision-making for individual patients while also contributing to enhanced public health monitoring, disease surveillance, and policy-making through more efficient analysis of large data sets. MVLMS integrate natural language processing with medical images to enable a more comprehensive and contextual understanding of medical images alongside their corresponding textual information. Unlike general vision-and-language models trained on diverse, non-specialized datasets, MVLMs are purpose-built for the medical domain, automatically extracting and interpreting critical information from medical images and textual reports to support clinical decision-making. Popular clinical applications of MVLMs include automated medical report generation, medical visual question answering, medical multimodal segmentation, diagnosis and prognosis and medical image-text retrieval. Here, we provide a comprehensive overview of MVLMs and the various medical tasks to which they have been applied. We conduct a detailed analysis of various vision-and-language model architectures, focusing on their distinct strategies for cross-modal integration/exploitation of medical visual and textual features. We also examine the datasets used for these tasks and compare the performance of different models based on standardized evaluation metrics. Furthermore, we highlight potential challenges and summarize future research trends and directions. The full collection of papers and codes is available at: https://github.com/YtongXie/Medical-Vision-and-Language-Tasks-and-Methodologies-A-Survey.
BibTeX
@misc{chen2024survey,
  author = {Qi Chen and Ruoshan Zhao and Sinuo Wang and Vu Minh Hieu Phan and Anton van den Hengel and Johan Verjans and Zhibin Liao and Minh-Son To and Yong Xia and Jian Chen and Yutong Xie and Qi Wu},
  title = {{A Survey of Medical Vision-and-Language Applications and Their Techniques}},
  year = {2024},
  eprint = {2411.12195},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.12195},
}
Figure