Hidden Flaws Behind Expert-Level Accuracy of GPT-4 Vision in Medicine

Qiao Jin, Fangyuan Chen, Yiliang Zhou, Ziyang Xu, Justin M. Cheung, Robert Chen, Ronald M. Summers, Justin F. Rousseau, Peiyun Ni, Marc J. Landsman, Sally L. Baxter, Subhi J. Al'Aref, Yijia Li, Michael F. Chiang, Yifan Peng, Zhiyong Lu

TL;DR

This paper dives deeper than just accuracy, evaluating GPT-4V's reasoning process on medical image challenges. While GPT-4V matches physician accuracy, it often provides flawed explanations, especially in image comprehension, highlighting the need for caution before clinical deployment.

What Problem Does This Paper Address?

The rapid advancement of large language models (LLMs) with vision capabilities, like GPT-4V, has shown impressive performance on various tasks, including medical challenges. However, most evaluations have focused solely on the final answer's accuracy. This overlooks a critical aspect: why the model arrived at that answer. In medicine, understanding the reasoning behind a diagnosis or decision is as crucial as the decision itself. Flawed reasoning, even if it leads to a correct answer, can be dangerous in a clinical setting. This paper addresses this gap by comprehensively analyzing GPT-4V's rationales (its step-by-step explanations) on medical image interpretation tasks, going beyond simple accuracy metrics.

Methods

The researchers evaluated GPT-4V on 207 multiple-choice questions from the New England Journal of Medicine (NEJM) Image Challenge. Their methodology involved:

  1. Prompting GPT-4V: They used a structured prompt to elicit responses from GPT-4V, specifically asking it to provide:
  2. Image Comprehension: A description of the provided medical image(s).
  3. Recall of Medical Knowledge: Relevant medical knowledge needed to solve the question.
  4. Step-by-step Reasoning: A detailed multimodal reasoning process to arrive at the answer.
  5. Human Evaluation: A multidisciplinary cohort of nine physicians evaluated GPT-4V's responses. They assessed:
  6. Accuracy: The correctness of GPT-4V's final answer.
  7. Rationale Quality: The accuracy and completeness of each of the three components (image comprehension, medical knowledge recall, and step-by-step reasoning), categorizing them as "Correct," "Partially Correct," or "Incorrect."
  8. Baseline Comparison: They established baselines by having a senior medical student and human physicians answer the same questions under both closed-book and open-book settings.
  9. BiomedCLIP Evaluation: As a point of comparison for vision-language foundation models, they also tested BiomedCLIP on the same dataset.

Datasets

  • NEJM Image Challenge: The primary dataset used for evaluation consisted of 207 recent multiple-choice questions from the NEJM Image Challenge. These questions are designed to test medical professionals' diagnostic capabilities and knowledge. The difficulty of these questions was categorized into "easy," "medium," and "hard" based on the proportion of correct answers by NEJM users.

Key Insights and Ideas

  • Accuracy isn't Everything: GPT-4V can achieve impressive accuracy, even matching or exceeding human physicians in some scenarios, but this doesn't guarantee sound reasoning.
  • Image Comprehension is a Bottleneck: The model struggles most with accurately interpreting visual information within medical images, leading to flawed rationales even when the final answer is correct.
  • The "Why" Matters for Trust: In high-stakes fields like medicine, understanding the AI's reasoning process is crucial for building trust and ensuring patient safety. A correct answer with a flawed explanation is a red flag.
  • Synergy Potential: Despite its flaws, GPT-4V's ability to answer questions that human physicians missed suggests potential for decision support.

Key Results

  • Comparable Accuracy: GPT-4V achieved an overall multi-choice accuracy of 81.6%, comparable to human physicians at 77.8% (though not statistically significant).
  • Outperforming Humans in Some Cases: GPT-4V correctly answered 78.3% of questions that physicians failed in a closed-book setting.
  • Flawed Rationales are Common: In 35.5% of cases where GPT-4V provided the correct final answer, its rationales were flawed.
  • Image Comprehension Errors: Flawed rationales were most prominent in image comprehension (27.2% of cases where the final answer was correct).
  • Medical Knowledge Recall is Stronger: Medical knowledge recall was found to be more reliable, with error rates ranging from 11.6% to 13.0%.
  • BiomedCLIP Performance: BiomedCLIP, a vision-language foundation model, achieved a low accuracy of 25.1%, highlighting the difficulty of the NEJM Image Challenge for such models.

Limitations and Future Work

  • Limited Dataset Size: The study used 207 questions, and each was annotated by only one physician per specialty, which is noted as a limitation due to cost and time constraints.
  • Potential Bias in Dataset: The distribution of NEJM Image Challenge cases might be biased towards certain specialties (e.g., dermatology).
  • Simplified Question Format: The study used questions with single correct answers, whereas real-world clinical scenarios often involve differential diagnoses and require listing rationales for multiple possibilities.
  • Future Work: The authors plan to evaluate physicians' rationales for answering medical questions to enable a more direct comparison with AI models. They also suggest exploring more nuanced prompt designs and alternative strategies for generating structured rationales.

Summary updated

Abstract

Recent studies indicate that Generative Pre-trained Transformer 4 with Vision (GPT-4V) outperforms human physicians in medical challenge tasks. However, these evaluations primarily focused on the accuracy of multi-choice questions alone. Our study extends the current scope by conducting a comprehensive analysis of GPT-4V's rationales of image comprehension, recall of medical knowledge, and step-by-step multimodal reasoning when solving New England Journal of Medicine (NEJM) Image Challenges - an imaging quiz designed to test the knowledge and diagnostic capabilities of medical professionals. Evaluation results confirmed that GPT-4V performs comparatively to human physicians regarding multi-choice accuracy (81.6% vs. 77.8%). GPT-4V also performs well in cases where physicians incorrectly answer, with over 78% accuracy. However, we discovered that GPT-4V frequently presents flawed rationales in cases where it makes the correct final choices (35.5%), most prominent in image comprehension (27.2%). Regardless of GPT-4V's high accuracy in multi-choice questions, our findings emphasize the necessity for further in-depth evaluations of its rationales before integrating such multimodal AI models into clinical workflows.
BibTeX
@inproceedings{jin2024hidden,
  author = {Qiao Jin and Fangyuan Chen and Yiliang Zhou and Ziyang Xu and Justin M. Cheung and Robert Chen and Ronald M. Summers and Justin F. Rousseau and Peiyun Ni and Marc J. Landsman and Sally L. Baxter and Subhi J. Al'Aref and Yijia Li and Michael F. Chiang and Yifan Peng and Zhiyong Lu},
  title = {{Hidden Flaws Behind Expert-Level Accuracy of GPT-4 Vision in Medicine}},
  booktitle = {Npj Digit. Medicine},
  year = {2024},
  url = {https://openreview.net/forum?id=UxuqHnMSxP},
}
Figure