FusionFM: Fusing Eye-specific Foundational Models for Optimized Ophthalmic Diagnosis

Ke Zou, Jocelyn Hui Lin Goh, Yukun Zhou, Tian Lin, Samantha Min Er Yew, Sahana Srinivasan, Meng Wang, Rui Santos, Gábor Márk Somfai, Huazhu Fu, Haoyu Chen, Pearse A. Keane, Ching-Yu Cheng, Yih Chung ThamarXiv 2025 · DOI · arXiv

Abstract

The rapid emergence of foundation models (FMs) in ophthalmology has created a need for systematic benchmarking and integration strategies. We present FusionFM, a framework that evaluates four state-of-the-art ophthalmic FMs—RETFound, VisionFM, RetiZero, and DINORET—and introduces Mixture-of-Experts (MoE) strategies to fuse their complementary strengths. Our results demonstrate that model fusion significantly outperforms any single model, particularly in complex diagnostic tasks like glaucoma detection and systemic disease prediction from retinal images.

Introduction

Foundation models, trained on vast datasets, have become "jacks-of-all-trades" in medical imaging. In ophthalmology, several specialized FMs have been developed using different architectures (e.g., MAE, DINOv2) and training data. However, the field lacks head-to-head comparisons, and individual models often exhibit task-specific biases. FusionFM addresses these challenges by providing a unified benchmarking suite and exploring whether a "super-model" can be created by combining these diverse experts.

The FusionFM Framework

The FusionFM framework consists of two primary stages: systematic benchmarking of individual models and the implementation of fusion strategies.

Overview of the FusionFM framework, illustrating the benchmarking of four foundation models and their fusion via Gating-based and Router-based Mixture-of-Experts (MoE) for downstream ocular and systemic disease detection.

The researchers evaluated four representative models: RETFound, VisionFM, RetiZero, and DINORET. To integrate these models, two Mixture-of-Experts (MoE) strategies were developed:

  1. Gating-based MoE: A lightweight neural network analyzes the input image to dynamically assign weights to the features extracted from each foundation model.
  2. Top-K Router-based MoE: An efficiency-focused approach that selects only the $K$ most relevant models for a specific task, reducing computational overhead while maintaining high accuracy.

Both strategies utilize a frozen backbone approach, where the core parameters of the foundation models remain unchanged, and only a small task-specific "head" is trained to interpret the combined feature set.

Experimental Evaluation

The study utilized a diverse set of international datasets to ensure robust validation across different populations and camera types:

  • Ocular Disease Detection: Included datasets from India (Chaksu) and Spain (PAPILA) for glaucoma, China (SUSTech-SYSU) for diabetic retinopathy, and the UK (UK Biobank) for age-related macular degeneration.
  • Systemic Disease Detection (Oculomics): Used the SEED dataset (Singapore) for internal validation and the UK Biobank for external validation of diabetes and hypertension prediction.

Key Findings

The Power of Fusion

The primary finding was that fusion consistently outperforms individual models. In glaucoma detection (Chaksu dataset), the gating-based MoE achieved a 4% improvement in AUC and an 18% increase in F1 score compared to the best-performing single model. This success is attributed to the complementary nature of the models; for example, RetiZero (vision-language pre-trained) and DINORET (DINOv2-based) capture different visual scales and features.

Individual Model Performance

Among the standalone models, DINORET and RetiZero were the most consistent performers. RetiZero demonstrated exceptional generalization on external datasets, likely due to its diverse pre-training corpus which included medical literature alongside images.

The Oculomics Challenge

While the models performed well on internal datasets for systemic disease prediction (diabetes and hypertension), performance dropped significantly during external validation. This highlights a persistent "generalization gap" in oculomics, where retinal markers for systemic conditions may not translate perfectly across different ethnicities or imaging hardware.

Conclusion

FusionFM demonstrates that the future of AI in ophthalmology lies not just in building better individual models, but in effectively combining existing ones. By leveraging Mixture-of-Experts strategies, FusionFM provides a scalable way to improve diagnostic accuracy across a wide range of ocular and systemic conditions. Future work will focus on improving the interpretability of these "expert" decisions and optimizing model efficiency for real-world clinical deployment.

Summary updated

Abstract

Foundation models (FMs) have shown great promise in medical image analysis by improving generalization across diverse downstream tasks. In ophthalmology, several FMs have recently emerged, but there is still no clear answer to fundamental questions: Which FM performs the best? Are they equally good across different tasks? What if we combine all FMs together? To our knowledge, this is the first study to systematically evaluate both single and fused ophthalmic FMs. To address these questions, we propose FusionFM, a comprehensive evaluation suite, along with two fusion approaches to integrate different ophthalmic FMs. Our framework covers both ophthalmic disease detection (glaucoma, diabetic retinopathy, and age-related macular degeneration) and systemic disease prediction (diabetes and hypertension) based on retinal imaging. We benchmarked four state-of-the-art FMs (RETFound, VisionFM, RetiZero, and DINORET) using standardized datasets from multiple countries and evaluated their performance using AUC and F1 metrics. Our results show that DINORET and RetiZero achieve superior performance in both ophthalmic and systemic disease tasks, with RetiZero exhibiting stronger generalization on external datasets. Regarding fusion strategies, the Gating-based approach provides modest improvements in predicting glaucoma, AMD, and hypertension. Despite these advances, predicting systemic diseases, especially hypertension in external cohort remains challenging. These findings provide an evidence-based evaluation of ophthalmic FMs, highlight the benefits of model fusion, and point to strategies for enhancing their clinical applicability.
BibTeX
@misc{zou2025fusionfm,
  author = {Ke Zou and Jocelyn Hui Lin Goh and Yukun Zhou and Tian Lin and Samantha Min Er Yew and Sahana Srinivasan and Meng Wang and Rui Santos and Gábor Márk Somfai and Huazhu Fu and Haoyu Chen and Pearse A. Keane and Ching-Yu Cheng and Yih Chung Tham},
  title = {{FusionFM: Fusing Eye-specific Foundational Models for Optimized Ophthalmic Diagnosis}},
  year = {2025},
  eprint = {2508.11721},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.11721},
}
Figure