Falcon Medical Visual Question Answering

Ameera Ali Bawazir, Hassan Alshanqiti, Kebin Wu, Fatima AlbreikiAAAI 2025 · DOI

TL;DR

This paper introduces Falcon Med-VQA, a novel generative system for medical visual question answering (VQA) that leverages a powerful vision-language model (Falcon-11B) and a dynamic model selection mechanism. It aims to provide more accurate, explainable, and user-friendly answers to complex medical image queries, outperforming existing generative approaches on key benchmarks.

What Problem Does This Paper Address?

The paper tackles the challenge of accurately answering natural language questions about medical images. While general-purpose Vision-Language Models (VLMs) have made strides, they often struggle with the nuances of medical imaging due to being trained primarily on natural images. Existing medical VQA models have limitations: traditional classification-based approaches are constrained by predefined answer sets and suffer from class imbalance, while newer generative models, though more flexible, can still be improved in terms of accuracy and interpretability for complex medical queries. The goal is to bridge this gap by creating a system that can interpret medical images and provide free-form, insightful answers, aiding clinical decision-making.

Methods

Falcon Med-VQA is built upon a three-component architecture:

  • Medical Image Encoder (PMC-CLIP): This component extracts visual features from medical images using PMC-CLIP, an encoder specifically optimized for medical image analysis.
  • Vision-Language Connection (Multi-layer Transformer Decoder): A multi-layer transformer decoder with learnable query embeddings bridges the gap between visual features and the language latent space.
  • Language Processing and Response Generation (Falcon-11B): The powerful Falcon-11B large language model is employed for generating free-form, context-aware answers to medical questions.

The system also incorporates several novel features:

  • Dynamic Model Routing: A classifier (ClipMD) determines the image modality (pathology or radiology), allowing the system to dynamically select specialized model checkpoints optimized for that domain, thereby improving accuracy.
  • Intuitive User Interface: The system presents the top three answers with confidence scores (CF), aiding users in assessing the reliability of the predictions.
  • Enhanced Explainability: Key medical terms from the generated answer are identified using a Named Entity Recognition (NER) model (DeBERTa-Med-NER-2), and these terms are then explained by GPT-4, providing deeper context.
  • Attention Map Visualization: Grad-CAM is used to generate attention maps, visually highlighting the specific regions of the medical image that the model focused on, offering insights into its decision-making process.

The training process involves three stages: pre-training on image-caption pairs for visual-linguistic alignment, fine-tuning on medical VQA pairs, and further fine-tuning on specialized datasets like SLAKE and VQA-RAD for radiology and pathology expertise. Low-Rank Adaptation (LoRA) is used for efficient fine-tuning of the LLM component.

Datasets

  • PMC-VQA (version 2): Used for pre-training and initial fine-tuning, containing approximately 152K image-caption and image-question pairs.
  • SLAKE: Used for fine-tuning, particularly for radiology-specific tasks.
  • VQA-RAD: Used for fine-tuning, covering radiology images.
  • PathVQA: Used for fine-tuning, covering pathology images.

The paper does not introduce a new benchmark dataset but evaluates its performance on existing ones.

Key Insights and Ideas

  • Generative VQA for Medical Domain: Treating medical VQA as a generative task, rather than classification, is crucial for handling the complexity and open-ended nature of medical questions.
  • Domain-Specific Model Specialization: Dynamically routing to specialized models based on image modality (radiology vs. pathology) significantly enhances performance.
  • Explainability is Key: Providing confidence scores, attention maps, and term explanations is vital for building trust and enabling clinical adoption of AI in medicine.
  • Leveraging Powerful LLMs: The integration of advanced LLMs like Falcon-11B is essential for generating coherent and contextually relevant free-form answers.

Key Results

Falcon Med-VQA demonstrates competitive performance against highly specialized medical VQA models and outperforms recent generative approaches on key benchmarks. Specifically, it achieves:

  • 86.6% accuracy on open-set questions and 89.5% on closed-set questions for the SLAKE VQA task.
  • Comparable performance to MedVint-TD and PefoMed on radiology and pathology benchmarks.
  • User requests are processed within an average of 4.9 seconds, ensuring fast and efficient responses.

Limitations and Future Work

  • Limitations: The paper does not explicitly detail limitations beyond the general challenges of medical VQA.
  • Future Work: The authors plan to expand the system's capabilities to include comprehensive medical report generation, aiming to further assist healthcare professionals in diagnosis and treatment planning.

Summary updated

BibTeX
@inproceedings{bawazir2025falcon,
  author = {Ameera Ali Bawazir and Hassan Alshanqiti and Kebin Wu and Fatima Albreiki},
  title = {{Falcon Medical Visual Question Answering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35346},
}
Figure