Abstract
Large Language Models (LLMs) have demonstrated remarkable performance on standardized medical examinations, yet their reliability in clinical practice remains a significant concern. This paper introduces MultifacetEval, a novel framework designed to evaluate the depth of medical knowledge in LLMs across four distinct facets: comparison, rectification, discrimination, and verification. By rephrasing standard multiple-choice questions into diverse formats, we reveal a significant "mastery gap"—a sharp decline in performance when models are required to demonstrate consistent knowledge across all facets. Our evaluation of 13 prominent LLMs on two new benchmarks, MultiDiseK and MultiMedQA, shows that even top-tier models like GPT-3.5 and Gemini-pro struggle with non-comparative tasks, particularly verification. These findings suggest that current benchmarks inflate perceived medical competence and highlight the need for more rigorous, multifaceted evaluation strategies to ensure safe clinical deployment.
Introduction
There is a frustrating paradox in medical AI: models are passing the USMLE with flying colors, yet they remain unreliable in actual hospitals. This discrepancy suggests that current benchmarks, such as MedQA, suffer from a "single-facet" bias. Most rely on Multiple-Choice Questions (MCQs), which primarily test a model’s ability to compare provided options. In these formats, the presence of distractors (wrong options) often provides inherent "hints" that inflate perceived intelligence.
In a real clinical setting, a doctor must do much more: they must verify if a lab result is consistent with a diagnosis, discriminate between similar pathologies, and rectify errors in a treatment plan. If an LLM passes an MCQ but fails to spot a blatant error regarding the same disease, it hasn't truly "mastered" the knowledge—it has simply learned to play the multiple-choice game. MultifacetEval is proposed to move beyond simple question-answering and probe the robustness of medical understanding.
The MultifacetEval Framework
The core innovation of this work is the "Multifaceted Evaluation Schema," which probes a single knowledge point ($k$) through four distinct lenses:
- Comparison ($f^1$): The standard MCQ format where the model picks the best option.
- Rectification ($f^2$): The model is given a statement with a potential error and must identify and fix it.
- Discrimination ($f^3$): Multiple-answer questions that require distinguishing nuanced differences between concepts.
- Verification ($f^4$): A True/False format to determine if a statement is factually sound without the help of distractors.
Crucially, we define "Mastery" strictly: a model only "masters" a knowledge point if it answers the questions for all facets correctly. This prevents models from appearing competent through lucky guesses or format-specific heuristics.
Benchmarks: MultiDiseK and MultiMedQA
We introduced two new multifaceted benchmarks to implement this framework:
- MultiDiseK: Built from a curated clinical knowledge base (DiseK) containing 1,000 common diseases. It includes over 31,000 questions across the four facets, covering symptoms, medicines, and surgeries.
- MultiMedQA: A transformed version of the popular MedQA (USMLE-style) dataset. We rephrased 800 original questions into approximately 7,200 multifaceted questions using heuristic rules and expert verification.
Results and Analysis
We evaluated 13 prominent LLMs, including general-purpose giants (GPT-3.5-turbo, Gemini-pro) and medical-specific models (Med42, Meditron). Several key insights emerged:
- The MCQ Illusion: Models perform significantly better on Comparison (MCQs) than any other facet. This suggests that distractors act as a crutch that inflates perceived intelligence.
- Verification is the Hardest: Models struggled most with the "Verification" (True/False) facet. Without other options to compare against, LLMs often fail to recognize when a standalone medical statement is false.
- The "Sharp Decline" of Mastery: When requiring a model to get all facets right for a single disease, performance drops by 40-50% compared to standard MCQ scores. This proves that LLM medical knowledge is often "shallow" and fragmented.
- Size Matters: Models smaller than 13B parameters were essentially useless on complex facets like discrimination, performing near the level of random guessing. Mastery only begins to emerge consistently in models with 70B+ parameters.
Conclusion
MultifacetEval highlights the fragmentation of medical knowledge in current LLMs. While specialized medical tuning (e.g., Med42-70B) improves performance, it does not eliminate the consistency gap. Future work aims to expand the scale of knowledge points and incorporate more complex dimensions like multi-step reasoning and long-term patient history to move closer to a truly expert-level medical assistant.
Summary updated