Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering

Cuong Ha, Shima Asaadi, Sanjeev Kumar Karn, Oladimeji Farri, Tobias Heimann, Thomas RunklerNAACL 2024

TL;DR

This paper introduces a novel vision-language model (VLM) specifically designed for medical visual question answering (MedVQA). By fusing a radiology-domain-adapted language model with a biomedical vision encoder and employing a parameter-efficient, multi-stage training strategy, the model achieves state-of-the-art performance on the SLAKE 1.0 MedVQA benchmark.

What Problem Does This Paper Address?

Vision-language models (VLMs) have shown great promise in general domains, but they often struggle to perform at the same level when applied to specialized fields like medicine. This is due to the complex medical terminology, the nuanced interpretation of medical images, and the limited availability of large, high-quality medical datasets. Existing VLMs, when adapted to the medical domain, often treat MedVQA as a classification problem, limiting their ability to generate free-form answers and potentially leading to inaccurate evaluations. This paper aims to bridge this gap by developing a VLM that can effectively understand and answer questions about medical images, treating MedVQA as a free-text generation task.

Methods

The core of the proposed approach is a VLM architecture that combines a domain-specific Large Language Model (LLM) tailored for radiology with a vision encoder designed for biomedical tasks. Crucially, the parameters of both the pre-trained vision and language models are kept frozen during training. To efficiently adapt these models to the MedVQA task, the authors employ a parameter-efficient training strategy using Low-Rank Adaptation (LoRA).

The training process is structured into three stages:

  1. Medical Concept Alignment: This stage uses an image-captioning task on the PMC-OA dataset to align medical concepts with visual content and pre-train the fusion module.
  2. General Medical VQA Adaptation: The model is then adapted to general medical VQA using the PMC-VQA dataset, exposing it to a wide variety of medical scenarios and modalities.
  3. Downstream Task Fine-tuning: Finally, the model is fine-tuned on specific MedVQA benchmarks like VQA-RAD and SLAKE 1.0-English.

This multi-stage, parameter-efficient approach aims to leverage the strengths of domain-specific models while minimizing computational costs and training time.

Datasets

The paper utilizes the following datasets:

  • PMC-OA: Used for the first pre-training stage (medical concept alignment) via image-captioning.
  • PMC-VQA (version 2): Used for the second pre-training stage to adapt the model to general medical VQA.
  • VQA-RAD: Used for downstream fine-tuning and evaluation.
  • SLAKE 1.0-English: Used for downstream fine-tuning and evaluation.

Key Insights and Ideas

  • Domain Adaptation is Crucial: General-domain VLMs perform poorly on specialized medical tasks. Adapting both the vision and language components to the medical domain is essential for achieving high performance.
  • Radiology-Specific LLM Matters: Using a radiology-domain-adapted LLM (RadBloomz-7b) significantly boosts performance compared to a general-domain LLM (Bloomz-7b1), especially for spatially-oriented queries.
  • Parameter-Efficient Training is Effective: LoRA allows for efficient adaptation of LLMs without retraining all parameters, leading to faster training and reduced computational resources while maintaining high performance.
  • Free-Text Generation Outperforms Classification: Framing MedVQA as a free-text generation task, rather than a classification problem, offers greater flexibility and potentially more accurate evaluation, although it can be more challenging.
  • Vision Encoder Choice Impacts Performance: The choice of vision encoder (e.g., BiomedCLIP-ViT vs. PMC-CLIP ResNet) can influence the overall performance, with some encoders demonstrating better capabilities for domain-specific knowledge integration.

Key Results

The proposed model, BiomedCLIP-RadBloomz-7b, achieved state-of-the-art performance on the SLAKE 1.0 MedVQA dataset with an overall accuracy of 87.5%. It also demonstrated strong performance on the VQA-RAD dataset with an overall accuracy of 73.2%.

  • On SLAKE 1.0, the model achieved 92.1% accuracy on closed-ended questions and 84.5% on open-ended questions.
  • On VQA-RAD, the model achieved 83.5% accuracy on closed-ended questions and 57.5% on open-ended questions.
  • The multi-stage training approach led to a significant 25% improvement in accuracy compared to direct fine-tuning.
  • Using the domain-adapted RadBloomz-7b LLM resulted in a 7.5% improvement in overall accuracy on SLAKE 1.0 and a 4.9% improvement on VQA-RAD compared to a general-domain LLM.

Limitations and Future Work

  • Generalizability: The model is tailored for radiology and may not be easily generalizable to other diverse medical domains and tasks.
  • Wh-Questions: In some wh-question cases, the model generated yes/no answers, indicating a need for further investigation into optimizing training to better capture question types.
  • Model Architecture: The study focused on decoder-only LLM architectures, and performance might differ with other architectures.
  • Future Work: The authors suggest further research into exploring the fusion of different domain-adapted models and investigating more efficient domain adaptation techniques for VLMs in challenging medical domains. They also highlight the need for careful assessment of MedVQA models before clinical application.

Summary updated

BibTeX
@inproceedings{ha2024fusion,
  author = {Cuong Ha and Shima Asaadi and Sanjeev Kumar Karn and Oladimeji Farri and Tobias Heimann and Thomas Runkler},
  title = {{Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure