TL;DR
This paper investigates the reliability and error patterns of large language models (LLMs) in medical assessment tasks, moving beyond simple accuracy metrics. They found that while LLMs like GPT-4 show strong performance and consistency across various question formats, their ability to reason and explain errors is still limited, highlighting the need for careful evaluation and potential human oversight in clinical applications.
What Problem Does This Paper Address?
The medical field, like many others, is increasingly exploring the use of AI, particularly large language models (LLMs), for tasks like patient assessment. While LLMs have demonstrated impressive capabilities on general tasks, their performance in high-stakes domains like medicine is less understood, especially concerning their reliability and how they handle errors. Human medical professionals make errors, but systems are in place to mitigate them. For LLMs, however, their error characteristics, consistency across different inputs, and ability to explain their reasoning are not well-studied. This paper aims to bridge this gap by systematically evaluating state-of-the-art LLMs on medical assessment questions, focusing on their error generation, consistency, and ability to recover from or explain mistakes, drawing parallels to human error analysis in medicine.
Methods
The researchers employed a multi-faceted approach to evaluate LLMs on medical assessment tasks. They started with a standard medical question-answering dataset (MedQA) and created several variations:
- Multiple Choice (MC) Variants: They tested 4-option and 5-option MC questions, and also explored the impact of shuffling the order of options.
- Open-ended Questions: MC questions were converted into free-text, open-ended formats to assess generative capabilities.
- Binarized Questions: Each option was presented as a True/False question to test a different reasoning style.
To assess consistency, they performed:
- Repeated Runs: Running the same prompt multiple times.
- Temperature Variation: Adjusting the model's temperature parameter to control randomness.
- N-shot Learning: Providing varying numbers of examples in the prompt.
- MC Reordering: Shuffling the order of answer choices.
Error analysis involved:
- Error Overlap: Quantifying how often different models (or models and humans) make the same errors.
- Error Recovery and Explanation: Presenting models with incorrect answers and asking them to explain why they are wrong, comparing their explanations to those provided by medical experts.
Datasets
The primary dataset used was the MedQA dataset (specifically the US portion), which contains medical questions, often with clinical context and multiple-choice answers, designed for medical licensing exams.
The authors also created new variants of the MedQA dataset:
- Open-ended Question Variants: Generated by rule-based transformations (e.g., changing "which" to "what") and human review.
- Binarized Question Variants: Created by framing each option as a True/False question.
- Annotated Errors: A subset of 200 questions were annotated by medical students and doctors to identify "next best" incorrect answers, serving as a proxy for human error.
Key Insights and Ideas
- Consistency is Key (and Correlated with Accuracy): The study strongly suggests that for LLMs in medical assessment, higher consistency across different experimental settings (like temperature or prompt variations) generally correlates with higher accuracy. This implies that if an LLM consistently provides the same answer across minor perturbations, it's more likely to be correct.
- LLMs Struggle with Ambiguity and Open-endedness: While LLMs perform well on structured multiple-choice questions, their performance degrades significantly when faced with open-ended questions or scenarios with multiple plausible answers. This highlights a limitation in their ability to handle the nuanced, less deterministic nature of some medical problems.
- "Explainability" is a Double-Edged Sword: LLMs can often provide explanations for why an answer is wrong, and these explanations can sometimes align with human reasoning. However, they also frequently "hallucinate" explanations for correct answers, claiming there's an error when there isn't, indicating a superficial understanding rather than deep reasoning.
- Human-like Errors: The study found that LLMs often make errors that are similar to those made by human annotators, suggesting that the confusing aspects of medical questions can challenge both humans and AI.
Key Results
- GPT-4 Leads: GPT-4 consistently demonstrated the strongest performance across various metrics, including accuracy and consistency, even when faced with different question formats and perturbations.
- Performance Drops with Complexity: Accuracy significantly decreased when moving from 5-option MC questions to 4-option, then to open-ended and binarized formats. For example, accuracy dropped by 3.1 to 7.4 across models when going from 5-option to 4-option MC.
- Temperature and N-shot Impact: While most LLMs were relatively stable across temperature variations, N-shot experiments showed higher deviations, with performance improvements diminishing after the first few examples.
- Error Overlap: LLMs showed substantial overlap in incorrect answers with each other and with human annotators, suggesting shared points of confusion.
- Explanation Quality Varies: GPT-4 and PALM were generally better at providing explanations for errors that aligned with physician explanations, while GPT-3.5 and Llama 2 sometimes contradicted expert reasoning or failed to follow instructions.
Limitations and Future Work
- Dataset Specificity: The analysis is based on a specific medical assessment dataset (MedQA). Trends observed might not hold for other medical domains or question types.
- Limited Scope of "Error Recovery": The study primarily focused on explaining why an answer is wrong. The ability to correct an error or recover from a false presupposition was not fully explored.
- Future Directions: The authors suggest developing new consistency metrics, creating benchmarks that query the same knowledge from different angles, and exploring methods to generate a small, closed set of optimal choices to reduce errors in ambiguous scenarios. They also emphasize the need for further research into better automated metrics for evaluating LLM explanations and the importance of human oversight in clinical AI applications.
Summary updated