Benchmark Evaluation of Multi-Modal Large Language Models for Ophthalmic Diagnosis

Weihua Yang, Shoujun Huang, Junhong Chen, J T Wang, Ping Zhang, Wending Du, Hong Yuan, Dexing Kong, Wei Lou, Wei ChiResearch Square 2025 · DOI

TL;DR

Researchers have developed a new, high-quality benchmark for ophthalmic diagnosis consisting of 295 real-world, pathologically confirmed clinical cases. Their evaluation of nine leading multi-modal large language models (MLLMs) reveals that top-tier models—specifically HAIBU-REMUD, ChatGPT-4o, and Gemini 2.5—are now approaching the diagnostic accuracy and consistency of human experts with decades of experience.

What Problem Does This Paper Address?

While general-purpose MLLMs (like GPT-4) show incredible promise, the medical field requires extreme precision. Ophthalmology is particularly challenging because it is "image-intensive," requiring a model to synthesize complex visual data (fundus photos, scans) with nuanced clinical narratives.

Most existing evaluations of AI in medicine rely on synthetic datasets or simplified benchmarks that don't reflect the messy reality of clinical practice. There is a critical "evaluation gap": we don't truly know how these models perform on complex, real-world cases that have been verified by pathology. This paper addresses that gap by creating a rigorous testing ground for "zero-shot" medical diagnosis.

Methods

The researchers conducted a systematic evaluation of nine MLLMs, including both proprietary giants and specialized open-source models:

  • Proprietary: ChatGPT-4o, Gemini 2.5.
  • Open-Source/Specialized: Qwen-VL 2.5-7B, Gemma 3 (4B & 12B), InternVL 3-8B, MedGemma, HealthGPT, and HAIBU-REMUD.

The Protocol: 1. Prompting: Each model was given a clinical image and its associated text description. They were prompted to provide a diagnosis only. 2. Expert Scoring: Two board-certified ophthalmologists (each with 20+ years of experience) independently scored the model outputs on a scale of 1.0 to 5.0. 3. Statistical Rigor: The team used Bland-Altman plots to measure "inter-rater agreement" (how much the two doctors agreed on the AI's performance) and density plots to visualize the stability of each model's outputs.

Datasets

The study introduces a curated benchmark dataset of 295 ophthalmic cases sourced from internationally authoritative journals (Ophthalmology and JAMA Ophthalmology) between 2023 and 2024.

  • Authenticity: Every case is pathologically confirmed, meaning the "ground truth" is based on biological evidence, not just opinion.
  • Diversity: Cases cover corneal disorders, retinal diseases, glaucoma, uveitis, tumors, and more.
  • Exclusion Criteria: Cases were excluded if image quality was poor or if the diagnosis relied solely on lab tests rather than the provided visual/textual clinical data.

Key Insights and Ideas

  • The Rise of Specialized Models: Interestingly, HAIBU-REMUD (a specialized medical MLLM) performed on par with or even slightly exceeded the median scores of general-purpose giants like ChatGPT-4o. This suggests that domain-specific fine-tuning is highly effective for specialized fields like eye care.
  • Performance Stratification: There is a clear "tier" system emerging. The top three models (HAIBU-REMUD, ChatGPT-4o, Gemini 2.5) showed high density in the 4.5–5.0 score range, while others (like InternVL 3 8B) struggled significantly, often failing to reach high-score regions at all.
  • Consistency is King: The researchers emphasized not just how high a model scored, but how consistent it was. Gemini 2.5 demonstrated the best overall agreement between the two human evaluators (mean difference of only -0.09), indicating its answers were stable and unambiguous.

Key Results

  • Top Performers: HAIBU-REMUD, ChatGPT-4o, and Gemini 2.5 achieved "benchmark-level" performance, with HAIBU-REMUD maintaining near-perfect median scores and very compact interquartile ranges (meaning it rarely gave a "bad" answer).
  • The "Middle Class": Gemma 3 12B and MedGemma showed acceptable performance but had much wider dispersions and more outliers, suggesting they are less reliable for critical clinical decision-making.
  • Statistical Significance: Using the Friedman test, the authors found no statistically significant difference between HAIBU-REMUD and ChatGPT-4o, but both were significantly superior to the other seven models (p < 0.00113).

Limitations and Future Work

  • Dataset Size: While 295 cases are high-quality, it is a relatively small sample size compared to the vast variety of ophthalmic conditions.
  • Selection Bias: Sourcing cases from top-tier journals might mean the dataset contains "classic" or "textbook" presentations rather than the ambiguous, low-quality images often encountered in a busy, under-resourced clinic.
  • Future Directions: The authors suggest that future work should focus on improving the "scoring stability" of open-source models and integrating these tools into actual clinical workflows to see how they assist (rather than just replace) human practitioners.

Summary updated

Abstract

<title>Abstract</title> Multi-modal large language models (MLLMs) are increasingly demonstrating significant potential in medical applications, particularly in image-intensive fields such as ophthalmology. While state-of-the-art models like ChatGPT-4o and Qwen-VL 2.5 exhibit impressive performance in general-domain tasks, there remains a lack of real-world clinical benchmark datasets to rigorously evaluate their diagnostic capabilities in specialized medical contexts. To address this gap, we constructed a curated benchmark dataset comprising 295 pathologically confirmed ophthalmic cases with representative clinical presentations. Using this dataset, we conducted a systematic evaluation of nine leading MLLMs, both open-source and proprietary. Our results reveal that models such as HAIBU-REMUD, ChatGPT-4o and Gemini 2.5 achieve high diagnostic accuracy and strong consistency, with performance approaching that of human experts. These findings suggest that current MLLMs have reached a promising stage in terms of applicability to real-world clinical settings, laying the groundwork for their integration into ophthalmology practice.
BibTeX
@inproceedings{yang2025benchmark,
  author = {Weihua Yang and Shoujun Huang and Junhong Chen and J T Wang and Ping Zhang and Wending Du and Hong Yuan and Dexing Kong and Wei Lou and Wei Chi},
  title = {{Benchmark Evaluation of Multi-Modal Large Language Models for Ophthalmic Diagnosis}},
  booktitle = {Research Square},
  year = {2025},
  doi = {10.21203/rs.3.rs-7186903/v1},
}
Figure