A scoping review on multimodal deep learning in biomedical images and texts

Zhaoyi Sun, Mingquan Lin, Qingqing Zhu, Qianqian Xie, Fei Wang, Zhiyong Lu, Yifan Peng

Abstract

Future computer-assisted diagnostic and prognostic systems must be capable of simultaneously processing multimodal data. Multimodal deep learning (MDL), which integrates multiple data sources such as images and text, has the potential to revolutionize biomedical data interpretation. This scoping review provides a comprehensive overview of the current state of the field, identifying key concepts, study types, and research gaps in image-text joint learning. By analyzing 77 core studies, we categorize the field into five clinical tasks and highlight directions for future research to support the next generation of decision-making systems.

Introduction

In clinical practice, diagnosis rarely relies on a single information source. Radiologists interpret images while considering patient history, and pathologists examine slides alongside clinical notes. However, most medical AI has historically been "unimodal," focusing on computer vision or natural language processing (NLP) in isolation. Medical data is inherently heterogeneous and often unstructured, such as free-text notes. While MDL aims to fuse these sources to mimic human clinical reasoning, the field is still maturing. This paper addresses the lack of a systematic overview of image-text joint learning, identifying effective architectures and remaining research gaps.

Methodology

This scoping review followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. The authors screened 361 articles published between 2018 and 2022 across five major databases: PubMed, ACM, IEEE, Google Scholar, and Semantic Scholar. After rigorous screening for relevance and quality, 77 papers were included in the final analysis.

The PRISMA flow diagram illustrating the systematic selection of 77 core studies from an initial pool of 361.

The review categorizes research into five "downstream tasks":

  1. Report Generation: Automated drafting of radiology reports from images.
  2. Visual Question Answering (VQA): Answering natural language questions about medical images.
  3. Cross-modal Retrieval: Finding relevant images via text queries or vice versa.
  4. Computer-aided Diagnosis (CAD): Classifying diseases using combined text and image data.
  5. Semantic Segmentation: Utilizing text descriptions to improve the outlining of organs or lesions.

Data Resources and Benchmarks

A critical component of MDL research is the availability of high-quality, paired datasets. The review provides a comprehensive directory of these resources, noting a significant "radiology bias" where most data focuses on chest X-rays or CT scans.

A comprehensive list of multimodal datasets categorized by image and text types, highlighting the diversity and availability of resources.

Key benchmarks include MIMIC-CXR and IU X-Ray for report generation, alongside VQA-specific datasets like VQA-RAD and PathVQA. Specialized datasets like ROCO (Radiology Objects in COntext) facilitate cross-modal retrieval, while MS-CXR supports semantic segmentation.

The Transformer Shift

There has been a clear methodological shift in recent years. Early models (pre-2020) typically relied on CNNs for images and LSTMs for text. Modern approaches almost exclusively utilize Transformers and Self-Supervised Learning (such as CLIP-based architectures) to align image and text features within a shared latent space.

Addressing Data Scarcity

Because high-quality, paired image-text data is difficult to annotate, researchers are increasingly adopting Meta-Learning (e.g., MAML) and Contrastive Learning. These techniques allow models to achieve high performance with fewer labeled examples. Additionally, successful models are beginning to incorporate external medical knowledge graphs to ensure factual correctness in generated text.

Clinical Applications and Results

  • Report Generation: Models using reinforcement learning and reward mechanisms (like factENT) have improved factual correctness by up to 14.2% and completeness by 25.3%.
  • Medical VQA: State-of-the-art models like UnICLAM achieve accuracies around 0.83 on benchmarks like SLAKE, though performance varies significantly between closed-ended and open-ended questions.
  • Diagnosis (CAD): Fusing text and images consistently outperforms unimodal models. Architectures like TieNet have achieved AUCs over 0.90 for thoracic disease classification.

Challenges and Future Directions

Despite rapid progress, several bottlenecks remain:

  1. Validation Gap: Only 10 of 25 report generation papers and zero VQA papers involved external validation or clinical trials. Most research remains confined to benchmark "sandboxes."
  2. Evaluation Metrics: Automated metrics like BLEU often fail to capture clinical nuances. There is an urgent need for standardized protocols for clinician-led evaluation.
  3. Interpretability: MDL presents unique "black box" challenges. Integrating image-based Grad-CAM with text-based attention maps to explain multimodal decisions is a key area for future work.
  4. Privacy and Fairness: To enable multi-institutional collaboration without compromising patient privacy, Federated Learning is a necessary future direction. Furthermore, the potential for algorithmic bias in multimodal settings requires more dedicated discussion in the literature.

Summary updated

Abstract

ObjectiveComputer-assisted diagnostic and prognostic systems of the future should be capable of simultaneously processing multimodal data. Multimodal deep learning (MDL), which involves the integration of multiple sources of data, such as images and text, has the potential to revolutionize the analysis and interpretation of biomedical data. However, it only caught researchers’ attention recently. To this end, there is a critical need to conduct a systematic review on this topic, identify the limitations of current work, and explore future directions.MethodsIn this scoping review, we aim to provide a comprehensive overview of the current state of the field and identify key concepts, types of studies, and research gaps with a focus on biomedical images and texts joint learning, mainly because these two were the most commonly available data types in MDL research.ResultThis study reviewed the current uses of multimodal deep learning on five tasks: (1) Report generation, (2) Visual question answering, (3) Cross-modal retrieval, (4) Computer-aided diagnosis, and (5) Semantic segmentation.ConclusionOur results highlight the diverse applications and potential of MDL and suggest directions for future research in the field. We hope our review will facilitate the collaboration of natural language processing (NLP) and medical imaging communities and support the next generation of decision-making and computer-assisted diagnostic system development.
BibTeX
@article{sun2023scoping,
  author = {Zhaoyi Sun and Mingquan Lin and Qingqing Zhu and Qianqian Xie and Fei Wang and Zhiyong Lu and Yifan Peng},
  title = {{A scoping review on multimodal deep learning in biomedical images and texts}},
  journal = {J. Biomed. Informatics},
  year = {2023},
  url = {https://openreview.net/forum?id=FjkBuB1K6U},
}
Figure