Abstract
Future computer-assisted diagnostic and prognostic systems must be capable of simultaneously processing multimodal data. Multimodal deep learning (MDL), which integrates multiple data sources such as images and text, has the potential to revolutionize biomedical data interpretation. This scoping review provides a comprehensive overview of the current state of the field, identifying key concepts, study types, and research gaps in image-text joint learning. By analyzing 77 core studies, we categorize the field into five clinical tasks and highlight directions for future research to support the next generation of decision-making systems.
Introduction
In clinical practice, diagnosis rarely relies on a single information source. Radiologists interpret images while considering patient history, and pathologists examine slides alongside clinical notes. However, most medical AI has historically been "unimodal," focusing on computer vision or natural language processing (NLP) in isolation. Medical data is inherently heterogeneous and often unstructured, such as free-text notes. While MDL aims to fuse these sources to mimic human clinical reasoning, the field is still maturing. This paper addresses the lack of a systematic overview of image-text joint learning, identifying effective architectures and remaining research gaps.
Methodology
This scoping review followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. The authors screened 361 articles published between 2018 and 2022 across five major databases: PubMed, ACM, IEEE, Google Scholar, and Semantic Scholar. After rigorous screening for relevance and quality, 77 papers were included in the final analysis.
The review categorizes research into five "downstream tasks":
- Report Generation: Automated drafting of radiology reports from images.
- Visual Question Answering (VQA): Answering natural language questions about medical images.
- Cross-modal Retrieval: Finding relevant images via text queries or vice versa.
- Computer-aided Diagnosis (CAD): Classifying diseases using combined text and image data.
- Semantic Segmentation: Utilizing text descriptions to improve the outlining of organs or lesions.
Data Resources and Benchmarks
A critical component of MDL research is the availability of high-quality, paired datasets. The review provides a comprehensive directory of these resources, noting a significant "radiology bias" where most data focuses on chest X-rays or CT scans.
Key benchmarks include MIMIC-CXR and IU X-Ray for report generation, alongside VQA-specific datasets like VQA-RAD and PathVQA. Specialized datasets like ROCO (Radiology Objects in COntext) facilitate cross-modal retrieval, while MS-CXR supports semantic segmentation.
Key Technological Trends
The Transformer Shift
There has been a clear methodological shift in recent years. Early models (pre-2020) typically relied on CNNs for images and LSTMs for text. Modern approaches almost exclusively utilize Transformers and Self-Supervised Learning (such as CLIP-based architectures) to align image and text features within a shared latent space.
Addressing Data Scarcity
Because high-quality, paired image-text data is difficult to annotate, researchers are increasingly adopting Meta-Learning (e.g., MAML) and Contrastive Learning. These techniques allow models to achieve high performance with fewer labeled examples. Additionally, successful models are beginning to incorporate external medical knowledge graphs to ensure factual correctness in generated text.
Clinical Applications and Results
- Report Generation: Models using reinforcement learning and reward mechanisms (like factENT) have improved factual correctness by up to 14.2% and completeness by 25.3%.
- Medical VQA: State-of-the-art models like UnICLAM achieve accuracies around 0.83 on benchmarks like SLAKE, though performance varies significantly between closed-ended and open-ended questions.
- Diagnosis (CAD): Fusing text and images consistently outperforms unimodal models. Architectures like TieNet have achieved AUCs over 0.90 for thoracic disease classification.
Challenges and Future Directions
Despite rapid progress, several bottlenecks remain:
- Validation Gap: Only 10 of 25 report generation papers and zero VQA papers involved external validation or clinical trials. Most research remains confined to benchmark "sandboxes."
- Evaluation Metrics: Automated metrics like BLEU often fail to capture clinical nuances. There is an urgent need for standardized protocols for clinician-led evaluation.
- Interpretability: MDL presents unique "black box" challenges. Integrating image-based Grad-CAM with text-based attention maps to explain multimodal decisions is a key area for future work.
- Privacy and Fairness: To enable multi-institutional collaboration without compromising patient privacy, Federated Learning is a necessary future direction. Furthermore, the potential for algorithmic bias in multimodal settings requires more dedicated discussion in the literature.
Summary updated