Abstract
Recent advancements in Large Language Models (LLMs) have significantly improved their ability to recall medical facts. However, complex clinical reasoning—the ability to synthesize nuanced signs and histories into accurate diagnoses and management plans—remains a challenge. This study evaluates the performance of four state-of-the-art models (DeepSeek-R1, Gemini 2.0 Pro, OpenAI o1, and OpenAI o3-mini) on a bilingual dataset of professional-level ophthalmology questions. DeepSeek-R1 emerged as the top performer, achieving 86.2% accuracy in Chinese and 80.8% in English, demonstrating the efficacy of specialized reasoning-oriented training in closing the performance gap between open-source and proprietary models.
Introduction
While LLMs have become proficient at answering fact-based medical questions, clinical practice requires sophisticated reasoning. Ophthalmology is particularly demanding, as it relies on interpreting subtle clinical signs and complex patient histories. Previous benchmarks, such as GPT-4's performance on "Clinical Challenge" datasets, have shown room for improvement. This paper investigates whether the latest generation of "reasoning" models, utilizing Chain-of-Thought (CoT) processing, can meet the high-stakes logical requirements of senior professional ophthalmology examinations in both Chinese and English.
Methodology
The researchers conducted a head-to-head evaluation of four leading models: DeepSeek-R1, Gemini 2.0 Pro, OpenAI o1, and OpenAI o3-mini. The evaluation framework focused on two primary dimensions:
- Accuracy: Measured by the ratio of correct answers across 130 multiple-choice questions (MCQs).
- Reasoning Ability Analysis: A manual classification of errors into clinical categories, such as "ignoring key positive history" or "inadequate differential consideration."
To test linguistic consistency, questions originally in Chinese were translated into English using DeepSeek-R1. The study also mapped model reasoning steps against standard human diagnostic logic (history summary -> sign highlighting -> preliminary diagnosis -> differential diagnosis -> final answer).
Dataset
The evaluation utilized 130 MCQs from the Chinese ophthalmology senior professional title examination, comprising 39 diagnostic and 91 management questions. These questions covered six subspecialties: anterior segment diseases, external eye/orbital diseases, glaucoma, ocular trauma, refractive disorders/strabismus, and retinal diseases. To prevent data leakage, the authors used "VIP documents" from Baidu Wenku, which are protected by anti-crawling measures, rather than standard public datasets like USMLE.
Results
DeepSeek-R1 demonstrated superior overall accuracy, particularly in the native Chinese portion of the test.
- Chinese Performance: DeepSeek-R1 achieved an accuracy of 0.862, significantly outperforming Gemini 2.0 Pro (0.715), OpenAI o3-mini (0.692), and OpenAI o1 (0.685).
- English Performance: DeepSeek-R1 maintained its lead with 0.808 accuracy, followed by Gemini 2.0 Pro at 0.746.
- Subspecialty Performance: DeepSeek-R1 led in five of the six subtopics in Chinese, showing particular strength in glaucoma and refractive disorders.
Discussion and Insights
The success of DeepSeek-R1 is attributed to its "cold-start" training data—thousands of high-quality CoT examples—which helps maintain complex logical chains during reinforcement learning. Despite high accuracy, common failure modes were observed across all models, most notably an "aggressive bias" in management, such as recommending advanced imaging (e.g., MRI) when simpler first-line tests (e.g., ultrasound) were clinically appropriate.
Language asymmetry was also noted: DeepSeek-R1 performed better in Chinese, likely reflecting its training data distribution, while Gemini and OpenAI o1 showed slightly better performance in English relative to their Chinese results.
Limitations and Future Work
Real-world clinical adoption faces several hurdles. Models must be validated in live clinical workflows to ensure safety and reliability. Furthermore, current evaluations lack standardized metrics for the quality of reasoning, relying instead on manual error classification. Future research should focus on "abductive reasoning"—finding the simplest explanation for symptoms—and improving performance in specialized areas like retinal diseases, where all models showed relative weakness.
Summary updated