Abstract
Background: State-of-the-art (SOTA) large language models (LLMs) are poised to revolutionize clinical medicine by transforming diagnostic, therapeutic, and interdisciplinary reasoning. Despite their promising capabilities, rigorous benchmarking of these models is essential to address concerns about their clinical proficiency and safety, particularly in high-risk environments.
Methods: This study implemented a multi-disciplinary, randomized, single-blind evaluation framework involving 27 experienced specialty clinicians with an average of 25.9 years of practice. The assessment covered 685 simulated and real clinical cases across 13 subspecialties, including both common and rare conditions. Evaluators rated LLM responses on medical strength (0–10 scale, where > 9.5 signified leading expert proficiency) and hallucination severity (0–5 scale for fabricated or misleading medical elements). Seven SOTA LLMs were tested, including top-ranked models from the ARENA leaderboard, with statistical analyses applied to adjust for confounders such as response length.
Findings: The evaluation revealed clinical plausibility in general-purpose LLMs, with Gemini 2.0 Flash leading raw scores and DeepSeek R1 excelling in adjusted analyses. Top models demonstrated proficiency comparable to a physician of 6 years post qualification experience (score ∼6.0), yet significant risks were noted. Instances of incompetence (scores ≤4) were detected across specialties, and 40 hallucination instances involving fabricated conditions, medications, and classification errors. These findings underscore the importance of implementing stringent safeguards to mitigate potential adverse outcomes in clinical applications.
Interpretation: While SOTA LLMs show substantial promise in enhancing clinical reasoning and decision-making, their unguarded application in medicine could present serious risks, such as misinformation and diagnostic errors. Human expert oversight remains crucial, particularly given reported incompetence and hallucination risks. Larger, multi-center studies are warranted to evaluate their real-world performance and track their evolution before broader clinical adoption.
Introduction
Most existing AI medical benchmarks, such as MedQA or MMLU, rely on multiple-choice formats. These are prone to "overfitting" and fail to capture the nuances of clinical reasoning required in a hospital setting. Furthermore, many studies do not account for "response length bias"—the tendency for human evaluators to give higher scores to longer, more authoritative-sounding answers, regardless of accuracy. This study aims to bridge this gap by using open-ended clinical scenarios and a single-blind methodology where expert clinicians rated model responses without knowing which AI produced them.
Methods
The researchers implemented a three-round evaluation framework involving 27 veteran clinicians. The study utilized a custom dataset of 685 clinical cases, split between OSCE (structured board-exam style) and real-world cases drawn from hospital records across 13 subspecialties.
Seven models—including Gemini 2.0 Flash, ChatGPT-4o, and DeepSeek-R1 (671B)—generated responses. Clinicians rated these on two scales: Medical Strength (0–10) and Hallucination Severity (0–5). Because Gemini 2.0 Flash produced responses significantly longer than its peers, the team used multiple linear regression and mixed-effects modeling to adjust scores for response length, ensuring a fair comparison.
Results
The evaluation demonstrated that top-tier LLMs are approaching the proficiency of mid-career physicians. The top models achieved a mean score of ~6.0, equivalent to a physician with 3–6 years of post-qualification experience.
Key Findings
- The "Length Confounder": Longer AI responses tend to receive higher raw scores. When unadjusted, Gemini 2.0 Flash led the pack. However, after adjusting for length, DeepSeek R1 emerged as the most robust model for clinical reasoning efficiency.
- The Incompetence Floor: Despite high average scores, all models showed "instances of incompetence" (scores ≤ 4) in approximately 7.2% of cases. Gemini 2.0 had the lowest rate of incompetence (2.7%).
- Dangerous Hallucinations: Researchers documented 40 specific instances of hallucinations, including fabricated drug names and incorrect dosages (e.g., recommending unsafe pediatric doses).
- Rare Diseases: LLMs showed high value in rare disease diagnosis, often synthesizing vast amounts of literature more effectively than generalist physicians.
Discussion and Conclusion
While LLMs are powerful reasoning engines, they are not infallible knowledge databases. The study highlights a persistent risk of clinical incompetence and hallucinations that could prove dangerous without human oversight. The authors conclude that while LLMs show substantial promise in enhancing clinical reasoning, they must be used with stringent human-in-the-loop safeguards. Future work should involve larger, multi-center studies to track the evolution of these models before broader clinical adoption.
Summary updated