NEUI at MEDIQA-M3G 2024: Medical VQA through consensus

Ricardo García, Oscar Lithgow-SerranoNAACL 2024

TL;DR

This paper presents a novel two-stage approach for medical visual question answering (VQA) that combines a visual language model (VLM) with a large language model (LLM). By leveraging the strengths of both, the system achieves impressive results on the MEDIQA-M3G task, demonstrating the power of multimodal reasoning for clinical applications.

What Problem Does This Paper Address?

The medical field is increasingly looking towards AI to assist with complex tasks like diagnosis and patient care. Medical Visual Question Answering (VQA) is a particularly promising area, aiming to enable AI systems to understand medical images and answer free-form questions about them. This can help alleviate the burden on healthcare professionals and improve patient engagement. However, medical VQA faces unique challenges: it requires not only understanding visual features but also integrating them with complex textual clinical history and queries. Furthermore, the data itself can be highly heterogeneous, with varying image quality and textual descriptions. The MEDIQA-M3G task specifically targets this by requiring the generation of clinical responses from textual history, user-generated images, and queries, with a focus on dermatology.

Methods

The authors propose a two-stage pipeline:

  1. Visual Language Model (VLM) Analysis: A compact VLM, Moondream (built on SigLIP and Phi-1.5), is fine-tuned on the MEDIQA-M3G and MEDIQA-CORR datasets. For each case, the VLM processes each image individually, generating an initial response. This is akin to "image analysis."
  2. Large Language Model (LLM) Consolidation: A fine-tuned BioMistral-7B-DARE LLM then takes the VLM's individual responses, along with the original textual context (query title and content), and synthesizes them into a single, consolidated response. This "consensus generation" step aims to overcome redundancies and short answers from the VLM.

The system was adapted for both English and Spanish by translating the original English case content to Spanish for the LLM's context.

Datasets

  • MEDIQA-M3G: Used for fine-tuning the VLM and as the primary testbed for the multimodal VQA task. This dataset focuses on clinical dermatology and includes textual history, user-generated images, and queries.
  • MEDIQA-CORR: Used for fine-tuning the BioMistral LLM. This dataset is focused on error detection and correction in clinical notes.

Key Insights and Ideas

  • Two-stage approach is key: Simply relying on a VLM for individual image analysis isn't enough. A subsequent LLM stage is crucial for consolidating information, reducing redundancy, and generating a more coherent and contextually relevant final answer.
  • Compact models can perform well: The use of a relatively small VLM (Moondream, 1.83B parameters) demonstrates that effective multimodal VQA can be achieved without resorting to massive, computationally expensive models.
  • Multilingual adaptation is feasible: By translating the textual context, the system could effectively handle Spanish submissions, leveraging the LLM's multilingual capabilities.
  • LLM reasoning is vital: Ablation studies showed that removing the LLM's reasoning capabilities (Strategy-2 without context) led to a significant performance drop, highlighting its importance in synthesizing information.
  • Specialized LLM training pays off: Fine-tuning the LLM on the MEDIQA-CORR dataset for error detection and correction significantly boosted performance, suggesting that task-specific fine-tuning is highly beneficial.

Key Results

  • The proposed two-stage strategy (Strategy-2) achieved the best results, ranking 4th in English and 2nd in Spanish during the competition.
  • Fine-tuning the VLM alone (Strategy-1) improved performance significantly over the baseline non-fine-tuned VLM.
  • Strategy-2, combining the fine-tuned VLM and LLM, showed a substantial leap in performance, achieving a deltableu score of 2.133 for English.
  • Ablation studies indicated that removing visual input led to a 33% degradation in scores, while neglecting LLM reasoning caused a 44% drop.
  • The LLM fine-tuned on the CORR dataset achieved the highest scores for Spanish submissions, even surpassing other runs.

Limitations and Future Work

  • Computational Cost: The current pipeline requires posing multiple queries to the VLM for each image in a case, which can increase computational costs and may be impractical for real-world deployment.
  • VLM Alignment: The VLM lacks specific alignment for visual-medical texts, making it more demanding for instruction tuning data.
  • Image Resolution: Handling cases where critical visual cues are in tiny regions of large images remains a challenge. Techniques like visual search or cropping are suggested.
  • LLM Scalability and Hallucinations: The computational demands of LLMs can be substantial, and their propensity to generate "hallucinations" (incorrect or misleading information) is a serious concern in medical applications.
  • Data Bias and Privacy: Biased training data can perpetuate unfair medical advice, and ensuring patient privacy is paramount.
  • Real-world Context: The current metrics and benchmarks don't fully capture performance in real-world clinical settings, where trustworthiness, explainability, and faithfulness are critical.

Future work includes incorporating a broader range of medical datasets (like the SCIN dataset for dermatology), exploring retrieval-augmented generation (RAG) for knowledge updating and hallucination mitigation, and fostering interdisciplinary collaboration with medical professionals.

Summary updated

BibTeX
@inproceedings{garcia2024neui,
  author = {Ricardo García and Oscar Lithgow-Serrano},
  title = {{NEUI at MEDIQA-M3G 2024: Medical VQA through consensus}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure