TL;DR
Researchers have developed a new, high-quality benchmark for ophthalmic diagnosis consisting of 295 real-world, pathologically confirmed clinical cases. Their evaluation of nine leading multi-modal large language models (MLLMs) reveals that top-tier models—specifically HAIBU-REMUD, ChatGPT-4o, and Gemini 2.5—are now approaching the diagnostic accuracy and consistency of human experts with decades of experience.
What Problem Does This Paper Address?
While general-purpose MLLMs (like GPT-4) show incredible promise, the medical field requires extreme precision. Ophthalmology is particularly challenging because it is "image-intensive," requiring a model to synthesize complex visual data (fundus photos, scans) with nuanced clinical narratives.
Most existing evaluations of AI in medicine rely on synthetic datasets or simplified benchmarks that don't reflect the messy reality of clinical practice. There is a critical "evaluation gap": we don't truly know how these models perform on complex, real-world cases that have been verified by pathology. This paper addresses that gap by creating a rigorous testing ground for "zero-shot" medical diagnosis.
Methods
The researchers conducted a systematic evaluation of nine MLLMs, including both proprietary giants and specialized open-source models:
- Proprietary: ChatGPT-4o, Gemini 2.5.
- Open-Source/Specialized: Qwen-VL 2.5-7B, Gemma 3 (4B & 12B), InternVL 3-8B, MedGemma, HealthGPT, and HAIBU-REMUD.
The Protocol: 1. Prompting: Each model was given a clinical image and its associated text description. They were prompted to provide a diagnosis only. 2. Expert Scoring: Two board-certified ophthalmologists (each with 20+ years of experience) independently scored the model outputs on a scale of 1.0 to 5.0. 3. Statistical Rigor: The team used Bland-Altman plots to measure "inter-rater agreement" (how much the two doctors agreed on the AI's performance) and density plots to visualize the stability of each model's outputs.
Datasets
The study introduces a curated benchmark dataset of 295 ophthalmic cases sourced from internationally authoritative journals (Ophthalmology and JAMA Ophthalmology) between 2023 and 2024.
- Authenticity: Every case is pathologically confirmed, meaning the "ground truth" is based on biological evidence, not just opinion.
- Diversity: Cases cover corneal disorders, retinal diseases, glaucoma, uveitis, tumors, and more.
- Exclusion Criteria: Cases were excluded if image quality was poor or if the diagnosis relied solely on lab tests rather than the provided visual/textual clinical data.
Key Insights and Ideas
- The Rise of Specialized Models: Interestingly, HAIBU-REMUD (a specialized medical MLLM) performed on par with or even slightly exceeded the median scores of general-purpose giants like ChatGPT-4o. This suggests that domain-specific fine-tuning is highly effective for specialized fields like eye care.
- Performance Stratification: There is a clear "tier" system emerging. The top three models (HAIBU-REMUD, ChatGPT-4o, Gemini 2.5) showed high density in the 4.5–5.0 score range, while others (like InternVL 3 8B) struggled significantly, often failing to reach high-score regions at all.
- Consistency is King: The researchers emphasized not just how high a model scored, but how consistent it was. Gemini 2.5 demonstrated the best overall agreement between the two human evaluators (mean difference of only -0.09), indicating its answers were stable and unambiguous.
Key Results
- Top Performers: HAIBU-REMUD, ChatGPT-4o, and Gemini 2.5 achieved "benchmark-level" performance, with HAIBU-REMUD maintaining near-perfect median scores and very compact interquartile ranges (meaning it rarely gave a "bad" answer).
- The "Middle Class": Gemma 3 12B and MedGemma showed acceptable performance but had much wider dispersions and more outliers, suggesting they are less reliable for critical clinical decision-making.
- Statistical Significance: Using the Friedman test, the authors found no statistically significant difference between HAIBU-REMUD and ChatGPT-4o, but both were significantly superior to the other seven models (p < 0.00113).
Limitations and Future Work
- Dataset Size: While 295 cases are high-quality, it is a relatively small sample size compared to the vast variety of ophthalmic conditions.
- Selection Bias: Sourcing cases from top-tier journals might mean the dataset contains "classic" or "textbook" presentations rather than the ambiguous, low-quality images often encountered in a busy, under-resourced clinic.
- Future Directions: The authors suggest that future work should focus on improving the "scoring stability" of open-source models and integrating these tools into actual clinical workflows to see how they assist (rather than just replace) human practitioners.
Summary updated