Abstract
While Large Language Models (LLMs) have demonstrated remarkable performance on standardized medical examinations, a significant "performance-practice gap" remains when these models are deployed in clinical settings. This paper introduces MultifacetEval, a novel evaluation framework designed to probe the depth of medical knowledge in LLMs across four distinct dimensions: comparison, rectification, discrimination, and verification. By re-evaluating models on transformed versions of existing benchmarks and new disease-centric datasets, the authors reveal that current LLMs often rely on pattern matching and test-taking strategies rather than a robust, stable knowledge base. The findings show a sharp performance decline—up to 50%—when models are required to demonstrate multifaceted mastery, highlighting the need for more rigorous evaluation before clinical integration.
Introduction: The Performance-Practice Gap
The rapid advancement of LLMs like GPT-4 and Med-PaLM 2 has led to human-level scores on benchmarks such as the USMLE-style MedQA. However, high accuracy on Multiple-Choice Questions (MCQs) does not necessarily translate to clinical competence. In real-world healthcare, practitioners must do more than pick the best option from a list; they must verify facts, correct erroneous information, and distinguish between subtle clinical presentations.
The authors argue that standard benchmarks are too narrow, allowing models to exploit "test-taking strategies" like the process of elimination. This creates an "illusion of competence" where a model appears to understand a concept in one format but fails when the same knowledge is probed differently.
The MultifacetEval Framework
To address these limitations, the researchers developed MultifacetEval. This framework defines "mastery" of a medical knowledge point as the ability to answer correctly across four distinct "facets":
- Comparison ($f^1$): The traditional MCQ format, testing the ability to select the most appropriate option.
- Rectification ($f^2$): Probing the model's ability to identify errors in a clinical scenario and provide the correct information.
- Discrimination ($f^3$): Using multiple-answer questions to ensure the model recognizes all relevant attributes of a disease, preventing reliance on single "giveaway" clues.
- Verification ($f^4$): True/False questions that test the veracity of a statement in isolation, without the contextual aid of distractors.
True mastery is only achieved if a model succeeds across all four facets for a specific knowledge point.
Datasets: MultiDiseK and MultiMedQA
The framework was tested using two primary datasets:
- MultiDiseK: A new dataset derived from the Disease-centric Knowledge base (DiseK), which includes 1,000 common diseases. Each disease is evaluated across four aspects: affected site, symptoms, medicine, and surgeries.
- MultiMedQA: A transformed version of the MedQA benchmark. The authors rephrased 800 original questions into the four facets, including negated versions to test for consistency and robustness against linguistic variations.
Key Results and Insights
The evaluation revealed several critical weaknesses in current LLMs:
The Mastery Cliff
As the requirement for mastery increases from a single facet to all four (quad-facet), performance drops precipitously across all models. For instance, GPT-3.5-turbo’s performance on MultiMedQA dropped by approximately 50% when moving from the standard MCQ format to the multifaceted requirement.
The 70B Parameter "Floor"
A significant performance gap was observed based on model scale. Models with fewer than 70 billion parameters generally struggled, often performing near the level of random guessing on complex facets like rectification and discrimination. This suggests that a certain level of model capacity is necessary to move beyond simple pattern matching.
Verification and Format Sensitivity
Verification (True/False) proved to be the most challenging facet. Without the context provided by multiple choices, models frequently failed to rely on their internal knowledge. This high sensitivity to question framing suggests that LLMs are not yet reasoning from a stable, format-invariant knowledge base.
The Role of Specialization
While general-purpose models like Gemini-pro and GPT-3.5-turbo were top performers, medical-specific models (e.g., Med42-70B) showed significant improvements over their general-purpose base models (Llama2-70B). This underscores the importance of domain-specific fine-tuning, though even these specialized models failed to achieve comprehensive mastery.
Conclusion
MultifacetEval exposes the fragility of medical knowledge in current LLMs. While these models are impressive "encyclopedias," their lack of consistency across different task formats makes them unreliable for autonomous clinical use. Future work must focus on improving reasoning depth and robustness against negation and format changes to bridge the gap between exam performance and clinical practice.
Summary updated