Ensemble learning of foundation models for precision oncology

Xiangde Luo, Xiyue Wang, Feyisope Eweje, Xiaoming Zhang, Sen Yang, Ryan Quinton, Jinxi Xiang, Yuchen Li, Yuanfeng Ji, Zhe Li, Yijiang Chen, Colin Bergstrom, Ted Kim, Francesca Maria Olguin, Kelley Yuan, Matthew Abikenari, Andrew Heider, Sierra Willens, Sanjeeth Rajaram, Robert West, Joel Neal, Maximilian Diehn, Ruijiang LiarXiv 2025 · DOI · arXiv

Abstract

Precision oncology relies on the accurate interpretation of histopathology slides to guide treatment decisions. While various pathology foundation models (FMs) have been developed, their performance is often inconsistent across different clinical tasks. We present ELF (Ensemble Learning of Foundation models), a framework that integrates five state-of-the-art foundation models into a unified slide-level representation. By leveraging the complementary strengths of diverse architectures, ELF achieves superior performance across 115 clinical tasks, including disease subtyping, biomarker detection, and treatment response prediction.

Introduction

In digital pathology, foundation models are large-scale AI systems trained on millions of tissue patches to extract generalizable visual features. However, the rapid proliferation of these models has created a "conundrum of choice." Models like GigaPath, CONCH, or Virchow2 are trained on different datasets with varying objectives, leading to specialized strengths and weaknesses. Furthermore, most FMs operate at the "tile" level (small patches), requiring significant labeled data to aggregate features into slide-level clinical predictions—a resource often unavailable in specialized clinical trials.

ELF addresses these challenges by shifting the paradigm from training increasingly larger individual models to strategically ensembling existing ones. This approach captures a broader spectrum of biological signals and provides a robust, "plug-and-play" solution for clinical research.

Methodology

The ELF framework incorporates five leading pathology FMs: GigaPath, CONCH, Virchow2, H-Optimus-0, and UNI.

Architecture and Aggregation

ELF utilizes a multi-head attention-based multi-instance learning (AbMIL) aggregator. Unlike simple feature concatenation, ELF learns to weigh the importance of specific slide regions for each constituent model, fusing them into a unified 768-dimensional slide-level embedding.

Pretraining Strategy

The model was pretrained on a massive dataset of 53,699 whole-slide images (WSIs) across 20 anatomical sites. The training employed a dual-objective approach:

  1. Contrastive Learning (MoCoV3): This aligns features across different foundation models to ensure cross-view consistency.
  2. Weakly Supervised Learning: Clinical metadata (e.g., organ type and cancer vs. normal status) provided semantic guidance during the pretraining phase.

This slide-level pretraining allows ELF to be fine-tuned for specific clinical tasks using very small datasets—sometimes as few as 32 patients—making it highly applicable to rare diseases and specialized clinical trials.

Experimental Results

ELF was evaluated across 115 clinical tasks, demonstrating consistent improvements over state-of-the-art slide-level models such as Prov-GigaPath, CHIEF, and TITAN.

Disease Subtyping and Biomarkers

On the challenging BRACS breast cancer dataset, ELF achieved a 16.3% relative improvement in balanced accuracy over the next best model. In biomarker detection, ELF reached an AUC > 0.85 for critical mutations, including BRAF in thyroid and colorectal cancers and KRAS in lung cancer. It also significantly outperformed existing methods in predicting Microsatellite Instability (MSI) status across multiple colorectal cancer cohorts.

Predicting Therapy Response

The framework's clinical utility was further validated across 21 independent cohorts involving 1,592 patients. In predicting immunotherapy response across eight cancer types, ELF achieved a mean AUC of 0.731, representing a 10–17% improvement over baseline models. Notably, combining ELF scores with traditional biomarkers like CD8 or PD-L1 further enhanced predictive accuracy, suggesting that ELF can augment existing clinical workflows.

Discussion: The Power of Complementarity

A key insight from this research is that different foundation models focus on non-overlapping morphological regions. Analysis of attention maps revealed low pairwise Pearson correlations between the five base models. This confirms that ELF’s performance stems from its ability to capture diverse biological signals that any single model would likely overlook. Unlike "knowledge distillation" methods that often result in an "average" feature set, ELF preserves and integrates the unique expertise of each constituent model.

Conclusion and Future Directions

ELF represents a significant step forward in precision oncology by providing a robust, high-performance framework for slide-level analysis. While the computational cost of running five models for inference is higher than a single-model approach, the gains in accuracy and data efficiency are substantial. Future work will focus on integrating multimodal data—including genomics and radiology—and validating the framework in prospective clinical settings to support real-time treatment decisions.

Source Code: https://github.com/lilab-stanford/ELF

Summary updated

BibTeX
@misc{luo2025ensemble,
  author = {Xiangde Luo and Xiyue Wang and Feyisope Eweje and Xiaoming Zhang and Sen Yang and Ryan Quinton and Jinxi Xiang and Yuchen Li and Yuanfeng Ji and Zhe Li and Yijiang Chen and Colin Bergstrom and Ted Kim and Francesca Maria Olguin and Kelley Yuan and Matthew Abikenari and Andrew Heider and Sierra Willens and Sanjeeth Rajaram and Robert West and Joel Neal and Maximilian Diehn and Ruijiang Li},
  title = {{Ensemble learning of foundation models for precision oncology}},
  year = {2025},
  eprint = {2508.16085},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.16085},
}
Figure