VisionUnite: A Vision-Language Foundation Model for Ophthalmology Enhanced With Clinical Knowledge

Zihan Li, Diping Song, Zefeng Yang, Deming Wang, Fei Li, Xiulan Zhang, Paul E. Kinahan, Yu Qiao

TL;DR

VisionUnite is a specialized vision-language foundation model designed to bring expert-level ophthalmic diagnosis to underserved regions. By integrating a dedicated "vision adapter" for clinical sign detection with a large language model (LLaMA-7B), it outperforms general-purpose giants like GPT-4V and Gemini Pro in diagnosing complex eye diseases from fundus images.

What Problem Does This Paper Address?

Ophthalmology faces a massive global challenge: over 2.2 billion people suffer from vision impairment, yet there is a critical shortage of specialists, particularly in low-income regions. While AI has shown promise, existing models suffer from three major "blind spots":

  1. Narrow Focus: Most models are "single-taskers," trained to detect only one specific disease (e.g., just glaucoma) rather than providing a comprehensive assessment.
  2. Poor Interaction: They lack the ability to engage in a dialogue with clinicians or patients to explain their reasoning.
  3. The "Black Box" Problem: They often provide a diagnosis without identifying the underlying clinical signs (like hemorrhages or lesions), making them difficult for doctors to trust.

Methods

The authors propose VisionUnite, a multimodal architecture that mimics the diagnostic workflow of a human expert. Its core components include:

  • Transformer-based Vision Encoder: Uses the EVA02 model to extract high-level visual features from fundus photographs.
  • The Vision Adapter: This is the "secret sauce." Unlike general models, this adapter is explicitly trained to classify six key clinical sign categories (Vascular, Macular, Fundus Boundary Color, Optical Cup Disc, Hemorrhages/Exudation, and Others). This forces the model to "look" for clinical evidence before jumping to a conclusion.
  • Vision Projector & LLaMA-7B: These components align the visual features with linguistic tokens, allowing the model to generate fluent, context-aware diagnostic reports and engage in multi-turn conversations.

The model was trained using a three-stage objective: image-text contrastive learning (aligning pictures with descriptions), sign-level supervised learning, and text-generation fine-tuning.

Datasets

The researchers curated a massive data pipeline to train VisionUnite:

  • Pre-training: 1.24 million image-text pairs, including natural images (COCO) and biomedical data (PMC-OA).
  • MMFundus (The Core): A newly introduced, pioneering dataset for fine-tuning. It contains 296,379 high-quality fundus image-text pairs and 889,137 simulated doctor-patient dialogue instances. It aggregates data from 53 open-source datasets and a large private clinical dataset.
  • Evaluation: The model was tested against a 2,233-case multiple-choice benchmark and a 180-sample multi-round VQA (Visual Question Answering) set.

Key Insights and Ideas

  • Clinical Sign Bottleneck: The researchers found that forcing the model to identify specific lesions (the "signs") significantly improved the accuracy of the final diagnosis. It transforms the model from a pattern-matcher into a clinical reasoner.
  • Prompt Robustness: General-purpose models like GPT-4V often provide inconsistent answers if a question is phrased slightly differently. VisionUnite demonstrates high "prompt robustness," maintaining consistent diagnostic accuracy regardless of how the user asks the question.
  • Problem Sensitivity: In multi-turn dialogues, VisionUnite showed a superior ability to "self-correct." If it missed a detail in the first round, it was much more likely than GPT-4V to recalibrate its diagnosis when prompted with follow-up questions.

Key Results

  • Outperforming the Giants: In diagnostic accuracy, VisionUnite scored over 45%, significantly higher than Gemini Pro (~19%) and GPT-4V (~40%) on open-ended clinical tasks.
  • Junior Doctor Equivalent: The model’s performance was found to be comparable to junior ophthalmologists with one year of clinical experience.
  • Clinical Utility: When used as an assistant, VisionUnite helped primary care doctors reduce diagnostic time by 27% while increasing their accuracy by 29%.
  • Zero-Shot Versatility: While most medical AI is limited to 1-2 diseases, VisionUnite successfully handled 25 different ophthalmological conditions in a multiple-choice setting with a 94.36% overall accuracy.

Limitations and Future Work

Despite its success, the authors note several areas for growth:

  • Modality Constraints: The model is currently focused on fundus photography. Future versions need to incorporate other essential imaging modalities like OCT (Optical Coherence Tomography) and slit-lamp photos.
  • Data Granularity: While the model identifies six broad sign categories, clinical practice requires even more granular labeling of specific lesion types.
  • Real-world Validation: While the retrospective results are strong, the model still needs prospective clinical trials to ensure safety and efficacy in live hospital environments.

Summary updated

Abstract

The need for improved diagnostic methods in ophthalmology is acute, especially in the less developed regions with limited access to specialists and advanced equipment. Therefore, we introduce VisionUnite, a novel vision-language foundation model for ophthalmology enhanced with clinical knowledge. VisionUnite has been pretrained on an extensive dataset comprising 1.24 million image-text pairs, and further refined using our proposed MMFundus dataset, which includes 296,379 high-quality fundus image-text pairs and 889,137 simulated doctor-patient dialogue instances. Our experiments indicate that VisionUnite outperforms existing generative foundation models such as GPT-4V and Gemini Pro. It also demonstrates diagnostic capabilities comparable to junior ophthalmologists. VisionUnite performs well in various clinical scenarios including open-ended multi-disease diagnosis, clinical explanation, and patient interaction, making it a highly versatile tool for initial ophthalmic disease screening. VisionUnite can also serve as an educational aid for junior ophthalmologists, accelerating their acquisition of knowledge regarding both common and rare ophthalmic conditions. VisionUnite represents a significant advancement in ophthalmology, with broad implications for diagnostics, medical education, and understanding of disease mechanisms.
BibTeX
@article{li2025visionunite,
  author = {Zihan Li and Diping Song and Zefeng Yang and Deming Wang and Fei Li and Xiulan Zhang and Paul E. Kinahan and Yu Qiao},
  title = {{VisionUnite: A Vision-Language Foundation Model for Ophthalmology Enhanced With Clinical Knowledge}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  doi = {10.1109/tpami.2025.3598734},
}

Loading related content...

Figure