Abstract
The rapid emergence of foundation models (FMs) in ophthalmology has created a need for systematic benchmarking and integration strategies. We present FusionFM, a framework that evaluates four state-of-the-art ophthalmic FMs—RETFound, VisionFM, RetiZero, and DINORET—and introduces Mixture-of-Experts (MoE) strategies to fuse their complementary strengths. Our results demonstrate that model fusion significantly outperforms any single model, particularly in complex diagnostic tasks like glaucoma detection and systemic disease prediction from retinal images.
Introduction
Foundation models, trained on vast datasets, have become "jacks-of-all-trades" in medical imaging. In ophthalmology, several specialized FMs have been developed using different architectures (e.g., MAE, DINOv2) and training data. However, the field lacks head-to-head comparisons, and individual models often exhibit task-specific biases. FusionFM addresses these challenges by providing a unified benchmarking suite and exploring whether a "super-model" can be created by combining these diverse experts.
The FusionFM Framework
The FusionFM framework consists of two primary stages: systematic benchmarking of individual models and the implementation of fusion strategies.
The researchers evaluated four representative models: RETFound, VisionFM, RetiZero, and DINORET. To integrate these models, two Mixture-of-Experts (MoE) strategies were developed:
- Gating-based MoE: A lightweight neural network analyzes the input image to dynamically assign weights to the features extracted from each foundation model.
- Top-K Router-based MoE: An efficiency-focused approach that selects only the $K$ most relevant models for a specific task, reducing computational overhead while maintaining high accuracy.
Both strategies utilize a frozen backbone approach, where the core parameters of the foundation models remain unchanged, and only a small task-specific "head" is trained to interpret the combined feature set.
Experimental Evaluation
The study utilized a diverse set of international datasets to ensure robust validation across different populations and camera types:
- Ocular Disease Detection: Included datasets from India (Chaksu) and Spain (PAPILA) for glaucoma, China (SUSTech-SYSU) for diabetic retinopathy, and the UK (UK Biobank) for age-related macular degeneration.
- Systemic Disease Detection (Oculomics): Used the SEED dataset (Singapore) for internal validation and the UK Biobank for external validation of diabetes and hypertension prediction.
Key Findings
The Power of Fusion
The primary finding was that fusion consistently outperforms individual models. In glaucoma detection (Chaksu dataset), the gating-based MoE achieved a 4% improvement in AUC and an 18% increase in F1 score compared to the best-performing single model. This success is attributed to the complementary nature of the models; for example, RetiZero (vision-language pre-trained) and DINORET (DINOv2-based) capture different visual scales and features.
Individual Model Performance
Among the standalone models, DINORET and RetiZero were the most consistent performers. RetiZero demonstrated exceptional generalization on external datasets, likely due to its diverse pre-training corpus which included medical literature alongside images.
The Oculomics Challenge
While the models performed well on internal datasets for systemic disease prediction (diabetes and hypertension), performance dropped significantly during external validation. This highlights a persistent "generalization gap" in oculomics, where retinal markers for systemic conditions may not translate perfectly across different ethnicities or imaging hardware.
Conclusion
FusionFM demonstrates that the future of AI in ophthalmology lies not just in building better individual models, but in effectively combining existing ones. By leveraging Mixture-of-Experts strategies, FusionFM provides a scalable way to improve diagnostic accuracy across a wide range of ocular and systemic conditions. Future work will focus on improving the interpretability of these "expert" decisions and optimizing model efficiency for real-world clinical deployment.
Summary updated