LTRC-IIITH at MEDIQA-M3G 2024: Medical Visual Question Answering with Vision-Language Models

Jerrin Thomas, Sushvin Marimuthu, Parameswari KrishnamurthyNAACL 2024

TL;DR

This paper tackles the challenge of medical visual question answering (VQA) in dermatology, specifically for multilingual and multimodal inputs. The authors fine-tuned a lightweight Vision-and-Language Transformer (ViLT) model for the task, achieving a respectable 6th place in the MEDIQA-M3G 2024 challenge. They also explored the potential of larger, more advanced Vision-Language Models (VLMs) like Gemini and LLaVA for this domain.

What Problem Does This Paper Address?

The paper addresses the growing need for effective remote clinical diagnosis and treatment, accelerated by recent global health challenges. Specifically, it focuses on the MEDIQA-M3G 2024 shared task, which aims to generate textual answers to dermatology-related questions using multimodal inputs: clinical history, user queries, and accompanying images. This task is particularly challenging due to the need to understand and synthesize information from both visual and textual data, often in multiple languages. Existing research has often focused on single modalities or specific medical imaging types (like radiology), leaving a gap in comprehensive, multilingual VQA for dermatology.

Methods

The core of the proposed solution is a fine-tuned Vision-and-Language Transformer (ViLT) model. ViLT is chosen for its lightweight nature and its ability to efficiently process both visual and textual data without relying on complex region-based processing. The model was pre-trained on three objectives: image-text matching, masked language modeling, and word-patch alignment. For the VQA task, a classifier head was added to the ViLT model, consisting of a linear layer applied to the final hidden state of the [CLS] token.

After the initial challenge submission, the authors explored the performance of larger VLMs: Gemini 1.0 Pro Vision and LLaVA-1.6 34B. These models are known for their advanced multimodal understanding capabilities. A separate translation module was also developed using Gemini 1.0 Pro to handle the multilingual aspect of the task.

Datasets

The primary dataset used is DermaVQA: A Multilingual Visual Question Answering Dataset for Dermatology. This dataset is derived from Chinese telemedicine data and translated into English, Chinese, and Spanish. The dataset splits are:

  • Train: 842 samples
  • Validation: 56 samples
  • Test: 100 samples

Each sample includes a query, clinical history, and one or more associated images. The English and Spanish training sets were machine-translated from Chinese, while validation and test sets were manually translated.

Key Insights and Ideas

  • ViLT's Efficiency: The paper highlights ViLT's effectiveness as a lightweight model for medical VQA, demonstrating that even simpler architectures can achieve competitive results when fine-tuned on relevant data.
  • Potential of Larger VLMs: The exploration of Gemini and LLaVA suggests that more powerful, general-purpose VLMs hold significant promise for specialized medical VQA tasks, potentially outperforming fine-tuned smaller models.
  • Multilingual Challenges: The task explicitly addresses multilingual VQA, underscoring the importance of handling different languages for broader clinical applicability. The approach of using machine translation for training data and a separate translation module for inference is a practical strategy.
  • Domain-Specific Fine-tuning is Crucial: While large VLMs are powerful, the paper shows that fine-tuning a model like ViLT on the specific medical VQA task yields good results, indicating the value of domain adaptation.

Key Results

  • The initial submission using ViLT (trained on 200 samples) ranked 8th in the MEDIQA-M3G 2024 challenge.
  • After the challenge, ViLT trained on the full 842 samples achieved a deltaBLEU of 0.52 and BERTScore of 0.82 in English.
  • LLaVA-1.6 34B showed strong performance, achieving deltaBLEU of 0.53 and BERTScore of 0.82 in English, and even better results in Spanish.
  • Gemini 1.0 Pro Vision demonstrated the highest performance among the tested models, with deltaBLEU of 0.86 and BERTScore of 0.70 in English, and deltaBLEU of 1.39 and BERTScore of 0.66 in Spanish.

Limitations and Future Work

  • Limited Training Data: The dataset size, especially for validation and testing, is relatively small, which can impact the robustness of model evaluation.
  • Focus on English: While the dataset is multilingual, the primary focus for the VQA module was on English predictions, with Chinese and Spanish outputs left empty in the initial submission.
  • Future Directions: The authors suggest exploring larger and more diverse datasets, incorporating stronger multi-modal reasoning capabilities, and conducting more rigorous evaluations with medical experts to ensure clinical utility and safety before real-world deployment. They also emphasize the need for further advancements to make such multimodal approaches reliable for remote diagnosis and treatment.

Summary updated

BibTeX
@inproceedings{thomas2024ltrc,
  author = {Jerrin Thomas and Sushvin Marimuthu and Parameswari Krishnamurthy},
  title = {{LTRC-IIITH at MEDIQA-M3G 2024: Medical Visual Question Answering with Vision-Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure