Gemini Goes to Med School: Exploring the Capabilities of Multimodal Large Language Models on Medical Challenge Problems & Hallucinations

Ankit Pal, Malaikannan SankarasubbuNAACL 2024

TL;DR

This paper rigorously evaluates Google's Gemini multimodal LLM on various medical tasks, finding it underperforms against leading models like Med-PaLM 2 and GPT-4, particularly in visual question answering. While Gemini shows promise, it exhibits significant susceptibility to hallucinations, overconfidence, and knowledge gaps, highlighting risks for critical medical applications.

What Problem Does This Paper Address?

The healthcare industry is increasingly exploring the potential of Large Language Models (LLMs) to revolutionize medical practices, from diagnostics to research. However, the safety and effectiveness of these powerful AI tools in such a high-stakes domain remain critical concerns. Specifically, this paper addresses the need for a comprehensive evaluation of multimodal LLMs, like Google's Gemini, on complex medical challenges. The authors aim to understand Gemini's capabilities in medical reasoning, its propensity for generating false information (hallucinations), and its comparative performance against other state-of-the-art models. This is crucial for identifying its strengths, limitations, and potential risks before widespread deployment in healthcare.

Methods

The study employs a multi-pronged evaluation strategy for Gemini Pro. Key methods include:

  • Benchmarking: Gemini was tested across three established medical benchmarks:
  • MultiMedQA: A comprehensive set of medical QA datasets designed to assess complex reasoning across various medical subjects, including licensing exams like USMLE and NEET-PG, and broader scientific domains like genetics and anatomy.
  • Med-HALT: A benchmark specifically designed to probe LLMs for unsafe reasoning tendencies and hallucinations, using tests like the Reasoning Hallucination Test (RHT) and Memory Hallucination Test (MHT).
  • Medical Visual Question Answering (VQA): A task using questions from the New England Journal of Medicine (NEJM) Image Challenge to evaluate Gemini's ability to interpret medical imagery.
  • Prompting Techniques: To thoroughly assess Gemini's reasoning and answer-generation capabilities, various advanced prompting methods were utilized, including:
  • Zero-Shot Prompting
    • Few-Shot Prompting
    • Chain-of-Thought (CoT) Prompting
    • Self-Consistency (SC)
    • Ensemble Refinement (ER)
  • Comparative Analysis: Gemini's performance was benchmarked against a range of open-source LLMs (e.g., Llama-2, Mistral, Qwen) and commercial models (Med-PaLM 2, GPT-4).
  • Evaluation Metrics: Accuracy and a Pointwise Score (awarding points for correct answers and penalizing incorrect ones) were used to quantify performance.

Datasets

The evaluation utilized the following datasets:

  • MultiMedQA: This benchmark encompasses several datasets, including:
  • MedQA (USMLE)
    • MedMCQA
    • PubMedQA
    • MMLU subsets (Clinical Knowledge, Medical Genetics, Anatomy, Professional Medicine, College Biology, College Medicine)
  • Med-HALT: This benchmark includes specific tests for hallucination detection.
  • Medical Visual Question Answering (VQA): Questions were sourced from the New England Journal of Medicine (NEJM) Image Challenge.

Key Insights and Ideas

  • Gemini's Hallucination Susceptibility: A major finding is Gemini's significant tendency to hallucinate, overconfidently provide incorrect information, and exhibit knowledge gaps. This is particularly concerning for medical applications where accuracy is paramount.
  • Visual Reasoning Gap: Gemini significantly underperformed in Medical VQA compared to GPT-4V, indicating limitations in its ability to integrate visual and textual information for medical image analysis.
  • Prompting Strategy Matters: The effectiveness of different prompting strategies (e.g., CoT, few-shot) varied significantly across medical domains, highlighting the need for domain-specific prompt engineering.
  • Subject-Specific Strengths and Weaknesses: Gemini demonstrated 100% accuracy in certain medical domains like Biostatistics and Epidemiology but struggled in areas requiring intricate visual analysis or complex diagnoses, such as Cardiology and Dermatology.
  • Open-Source Models Show Promise: Some open-source models, like Yi-34B and Qwen-72B, showed strong performance in specific medical areas, indicating their growing potential.

Key Results

  • MultiMedQA Performance: Gemini Pro scored 67.0% on MedQA (USMLE), significantly lower than Med-PaLM 2 (86.5%) and GPT-4 (86.1%). Similar gaps were observed in MedMCQA (Gemini 62.2% vs. Med-PaLM 2 72.3%) and PubMedQA (Gemini 70.7% vs. Med-PaLM 2 81.8%).
  • Medical VQA Accuracy: Gemini achieved an accuracy of 61.45% on the medical VQA dataset, falling considerably short of GPT-4V's 88% accuracy.
  • Hallucination Detection: Gemini showed high accuracy (82.59%) in detecting false medical questions (RHT) but exhibited a tendency towards overconfidence in diagnostics (False Confidence Test - FCT) with a low pointwise score of 2.
  • Subject-wise Accuracy: Gemini achieved 100% accuracy in Biostatistics, Cell Biology, Epidemiology, Gastroenterology, and Obstetrics & Gynecology. However, it showed lower scores in Cardiology (26.67%), Forensic Medicine (44.19%), and Dermatology (58.82%).

Limitations and Future Work

  • API Constraints: The evaluation was limited to Gemini Pro's available APIs, potentially not leveraging the full capabilities of models like Gemini Ultra.
  • Long-Form Question Answering: The study did not encompass the evaluation of long-form question answering, a critical aspect for comprehensive medical understanding.
  • VQA Dataset Size: The VQA task used a relatively small sample of 100 questions due to the extensive human examination required for each output. Future work could explore larger VQA datasets.
  • Future Research Directions: The authors suggest exploring Gemini Ultra APIs, evaluating long-form question answering, incorporating retrieval-augmented generation (RAG) for enhanced performance, and expanding VQA datasets. They also emphasize the need for further research to address Gemini's limitations and develop more sophisticated AI tools for medicine.

Summary updated

BibTeX
@inproceedings{pal2024gemini,
  author = {Ankit Pal and Malaikannan Sankarasubbu},
  title = {{Gemini Goes to Med School: Exploring the Capabilities of Multimodal Large Language Models on Medical Challenge Problems \& Hallucinations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure