Abstract
Medical images are difficult to comprehend for a person without expertise. The scarcity of medical practitioners across the globe often face the issue of physical and mental fatigue due to the high number of cases, inducing human errors during the diagnosis. In such scenarios, having an additional opinion can be helpful in boosting the confidence of the decision maker. Thus, it becomes crucial to have a reliable visual question answering (VQA) system to provide a 'second opinion' on medical cases. However, most of the VQA systems that work today cater to real-world problems and are not specifically tailored for handling medical images. Moreover, the VQA system for medical images needs to consider a limited amount of training data available in this domain. In this paper, we develop MedFuseNet, an attention-based multimodal deep learning model, for VQA on medical images taking the associated challenges into account. Our MedFuseNet aims at maximizing the learning with minimal complexity by breaking the problem statement into simpler tasks and predicting the answer. We tackle two types of answer prediction—categorization and generation. We conducted an extensive set of quantitative and qualitative analyses to evaluate the performance of MedFuseNet. Our experiments demonstrate that MedFuseNet outperforms the state-of-the-art VQA methods, and that visualization of the captured attentions showcases the interpretability of our model's predicted results.
Introduction
Interpreting medical images, such as CT scans, pathology slides, or MRIs, requires years of specialized training. With a global shortage of physicians and increasing diagnostic workloads, human fatigue is a significant factor in diagnostic errors. A Medical Visual Question Answering (VQA) system—where an AI provides answers to natural language questions about a medical image—could serve as a vital "second opinion" to support clinical decision-making.
However, standard VQA models trained on general datasets (e.g., COCO or VQA v2.0) struggle in the medical domain due to two primary challenges: data scarcity and image complexity. While general datasets contain millions of samples, medical datasets are often limited to a few thousand. Furthermore, medical images are highly heterogeneous and contain subtle, critical details that general-purpose models often fail to capture.
Methodology
MedFuseNet is designed to maximize learning efficiency with minimal complexity, specifically addressing the constraints of medical data. The architecture consists of four primary modules:
- Feature Extraction: The model utilizes a ResNet-152 backbone (pre-trained on ImageNet) for visual feature extraction and BERT for text processing. BERT was selected for its superior ability to capture positional semantics compared to other NLP models like XLNet.
- Multimodal Factorized Bilinear (MFB) Pooling: To fuse image and text features, MedFuseNet employs MFB pooling. Unlike simple concatenation, MFB captures complex interactions between modalities while remaining computationally efficient—a critical requirement for small-scale datasets where full bilinear pooling would lead to overfitting.
- Co-Attention Mechanism: The model implements dual attention: "Question Attention" identifies key linguistic cues, while "Image Attention" focuses the model on relevant anatomical regions or pathologies.
- Dual-Mode Prediction: The system supports both Categorization (classification for fixed-set answers like modality or plane) and Generation (an LSTM-based decoder using Teacher Forcing and Beam Search for open-ended clinical descriptions).
Datasets and Evaluation
The model was benchmarked on two key datasets:
- MED-VQA 2019 (ImageCLEF): Contains 4,200 radiology images with questions regarding modality, plane, organ, and abnormality.
- PathVQA: A larger dataset of over 30,000 questions based on pathology images from medical textbooks, used for both "yes/no" categorization and open-ended generation tasks.
Results and Discussion
MedFuseNet consistently outperformed established baselines, including SAN, HiCAt, and BAN. On the MED-VQA dataset, it achieved 84% accuracy for modality identification and 78% for anatomical plane classification. In the PathVQA "yes/no" task, it reached 63.6% accuracy, surpassing the state-of-the-art BAN model (60.4%).
For open-ended generation, MedFuseNet achieved a BLEU-1 score of 0.605 on PathVQA, demonstrating a strong capability for generating clinically relevant sequences. Beyond quantitative metrics, qualitative analysis of attention maps confirmed that the model focuses on the correct pathological features (e.g., tumors or fractures), enhancing the interpretability and trustworthiness of the system.
Conclusion
MedFuseNet represents a significant step toward reliable AI-assisted medical diagnosis. By combining efficient bilinear fusion with co-attention, it overcomes the limitations of small medical datasets. Future work will focus on expanding the model's scope to rare diseases and more tightly integrating the generative decoder to improve the flow of information during complex answer synthesis.
Summary updated