Abstract
The need for improved diagnostic methods in ophthalmology is acute, especially in less developed regions with limited access to specialists and advanced equipment. Therefore, we introduce VisionUnite, a novel vision-language foundation model for ophthalmology enhanced with clinical knowledge. VisionUnite has been pretrained on an extensive dataset comprising 1.24 million image-text pairs, and further refined using our proposed MMFundus dataset, which includes 296,379 high-quality fundus image-text pairs and 889,137 simulated doctor-patient dialogue instances. Our experiments indicate that VisionUnite outperforms existing generative foundation models such as GPT-4V and Gemini Pro. It also demonstrates diagnostic capabilities comparable to junior ophthalmologists. VisionUnite performs well in various clinical scenarios including open-ended multi-disease diagnosis, clinical explanation, and patient interaction, making it a highly versatile tool for initial ophthalmic disease screening. VisionUnite can also serve as an educational aid for junior ophthalmologists, accelerating their acquisition of knowledge regarding both common and rare ophthalmic conditions. VisionUnite represents a significant advancement in ophthalmology, with broad implications for diagnostics, medical education, and understanding of disease mechanisms.
Introduction
Ophthalmology faces a massive global challenge: over 2.2 billion people suffer from vision impairment, yet there is a critical shortage of specialists, particularly in low-income regions. While AI has shown promise, existing models suffer from three major "blind spots": a narrow focus on single diseases, poor user interaction, and a lack of clinical interpretability (the "black box" problem). These models often provide a diagnosis without identifying the underlying clinical signs, such as hemorrhages or lesions, making them difficult for clinicians to trust.
VisionUnite addresses these limitations by providing an open-ended, interactive platform capable of multi-disease diagnosis and detailed clinical explanation, mimicking the diagnostic workflow of a human expert.
Methodology
VisionUnite utilizes a multimodal architecture designed to align visual features with clinical knowledge. Its core components include a Transformer-based Vision Encoder (EVA02) for high-level feature extraction and a specialized Vision Adapter. This adapter is explicitly trained to classify six key clinical sign categories: Vascular, Macular, Fundus Boundary Color, Optical Cup Disc, Hemorrhages/Exudation, and Others. By forcing the model to identify these signs, it transitions from simple pattern matching to clinical reasoning.
The model is integrated with LLaMA-7B via a vision projector, allowing it to generate fluent diagnostic reports. Training follows a three-stage objective: image-text contrastive learning, sign-level supervised learning (using Binary Cross-Entropy loss), and text-generation fine-tuning (using Cross-Entropy loss).
Datasets
The model's performance is rooted in the MMFundus dataset, a pioneering collection for ophthalmic fine-tuning. It aggregates data from 53 open-source datasets and a large private clinical dataset, resulting in 296,379 high-quality fundus image-text pairs and 889,137 simulated doctor-patient dialogue instances. This diverse dataset ensures the model can handle a wide range of conditions, from common glaucoma to rare ocular vascular occlusive disorders.
Results and Discussion
VisionUnite demonstrates significant improvements over general-purpose models. In open-ended clinical tasks, it achieved a diagnostic accuracy of over 45%, outperforming GPT-4V (~40%) and Gemini Pro (~19%). Furthermore, in a multiple-choice setting covering 25 different conditions, it reached a 94.36% overall accuracy.
Key insights from the evaluation include:
- Clinical Reasoning: Identifying specific lesions (the "sign bottleneck") significantly improved final diagnostic accuracy.
- Prompt Robustness: Unlike general models, VisionUnite maintains consistent accuracy regardless of how a question is phrased.
- Clinical Impact: When used as an assistant, the model helped primary care doctors reduce diagnostic time by 27% while increasing their accuracy by 29%.
Conclusion
VisionUnite represents a major step toward accessible, expert-level ophthalmic care. While currently focused on fundus photography, future work will expand to include other modalities like OCT and slit-lamp imaging. By providing clinical interpretations alongside diagnoses, VisionUnite serves as both a screening tool and an educational aid for junior ophthalmologists.
Summary updated