Evaluating Vision and Pathology Foundation Models for Computational Pathology: A Comprehensive Benchmark Study

Rohan Bareja, Francisco Carrillo‐Pérez, Yuanning Zheng, Marija Pizurica, Tarak Nandi, Jeanne Shen, Ravi Madduri, Olivier GevaertmedRxiv 2025 · DOI

TL;DR

This study presents PathBench, a massive benchmarking effort evaluating 31 AI foundation models across 41 computational pathology tasks. The results crown Virchow2 as the most consistent performer, reveal that pathology-specific vision models currently outperform vision-language models, and challenge the "bigger is always better" scaling dogma in medical AI.

What Problem Does This Paper Address?

The field of Computational Pathology (CPath) has seen an explosion of "foundation models"—large-scale models trained on millions of tissue images using self-supervised learning. However, there is no "industry standard" for comparing them.

Most models are evaluated on the same public datasets they were trained on (like TCGA), leading to potential overfitting. We don't know how these models handle "out-of-domain" data—images from different hospitals, scanners, or staining protocols. This paper seeks to provide a definitive, standardized ranking to help researchers choose the right tool for specific diagnostic or prognostic tasks.

Methods

The researchers conducted a head-to-head comparison of 31 models, categorized into four groups:

  1. VM (General Vision): Models like DINOv2 trained on natural images (e.g., cats, cars).
  2. VLM (General Vision-Language): Models like CLIP that understand both images and text.
  3. Path-VM (Pathology Vision): Models trained specifically on pathology slides (e.g., Virchow2, UNI).
  4. Path-VLM (Pathology Vision-Language): Pathology models that incorporate medical text (e.g., CONCH, QuiltNet).

To ensure a fair fight, they used Linear Probing: they "froze" the foundation models (using them only as feature extractors) and trained a simple linear layer on top for each specific task. This measures the inherent quality of the representations the model learned during its massive pre-training phase.

Datasets

The study utilized a staggering 53 datasets encompassing over 17,500 samples:

  • TCGA (The Cancer Genome Atlas): 19 tasks (subtyping, grading, mutation prediction).
  • CPTAC: 7 pan-cancer tasks used as an independent validation set.
  • External Benchmarks: 8 public datasets (e.g., BACH for breast cancer, MHIST for colorectal).
  • Out-of-Domain (OOD): Proprietary datasets from Stanford, NLST, and DHMC to test real-world generalizability on unseen clinical data.

The authors also launched PathBench, a live dashboard for real-time visualization of these results.

Key Insights and Ideas

  • The Scaling Plateau: In general AI, more data and bigger models usually equal better performance. In pathology, this isn't strictly true. Models trained on 100k slides (like UNI) often performed as well as or better than those trained on millions of slides. Diversity and quality of data seem to trump sheer volume once a certain threshold is reached.
  • Vision Trumps Language (For Now): Pathology-specific vision models (Path-VM) consistently beat vision-language models (Path-VLM). The authors suggest that current medical text annotations might be too "noisy" or low-resolution to help the model learn the intricate visual patterns required for pathology.
  • Specialization vs. Generalization: Some models are "specialists." For example, H-optimus-0 excelled at brain and breast cancer, while Prov-GigaPath was a powerhouse for lung cancer.
  • The Wisdom of the Crowd: The researchers created a Fusion Model by aggregating predictions from the top 5 models. While it wasn't always #1 on familiar data, it was the most robust and highest-performing model when faced with completely new, "out-of-domain" datasets.

Key Results

  • Virchow2 emerged as the overall winner, delivering the highest performance across TCGA, CPTAC, and external tasks (Mean performance ~0.706 on TCGA).
  • Path-VM dominance: The top 10 rankings were almost entirely populated by pathology-specific vision models. Only one general vision model (iBOT-L16) managed to break into the top tier.
  • Statistical Significance: While Path-VMs are the practical leaders, the performance gap between them and general vision models (VM) was not always statistically significant, suggesting that general-purpose features are surprisingly resilient in medical contexts.

Limitations and Future Work

  • TCGA Leakage: The authors acknowledge that because almost all pathology models use TCGA for training, evaluating them on TCGA tasks might lead to "inflated" performance.
  • Multimodal Integration: The underperformance of Path-VLMs suggests a need for better architectures that can truly fuse high-resolution visual details with complex medical vocabulary.
  • Task Complexity: The study focused on classification and subtyping; future benchmarks could expand into more complex areas like spatial omics integration or pixel-level segmentation.

Summary updated

Abstract

Abstract To advance precision medicine in pathology, robust AI-driven foundation models are increasingly needed to uncover complex patterns in large-scale pathology datasets, enabling more accurate disease detection, classification, and prognostic insights. However, despite substantial progress in deep learning and computer vision, the comparative performance and generalizability of these pathology foundation models across diverse histopathological datasets and tasks remain largely unexamined. In this study, we conduct a comprehensive benchmarking of 31 AI foundation models for computational pathology, including general vision models (VM), general vision-language models (VLM), pathology-specific vision models (Path-VM), and pathology-specific vision-language models (Path-VLM), evaluated over 41 tasks sourced from TCGA, CPTAC, external benchmarking datasets, and out-of-domain datasets. Our study demonstrates that Virchow2, a pathology foundation model, delivered the highest performance across TCGA, CPTAC, and external tasks, highlighting its effectiveness in diverse histopathological evaluations. We also show that Path-VM outperformed both Path-VLM and VM, securing top rankings across tasks despite lacking a statistically significant edge over vision models. Our findings reveal that model size and data size did not consistently correlate with improved performance in pathology foundation models, challenging assumptions about scaling in histopathological applications. Lastly, our study demonstrates that a fusion model, integrating top-performing foundation models, achieved superior generalization across external tasks and diverse tissues in histopathological analysis. These findings emphasize the need for further research to understand the underlying factors influencing model performance and to develop strategies that enhance the generalizability and robustness of pathology-specific vision foundation models across different tissue types and datasets.
BibTeX
@misc{bareja2025evaluating,
  author = {Rohan Bareja and Francisco Carrillo‐Pérez and Yuanning Zheng and Marija Pizurica and Tarak Nandi and Jeanne Shen and Ravi Madduri and Olivier Gevaert},
  title = {{Evaluating Vision and Pathology Foundation Models for Computational Pathology: A Comprehensive Benchmark Study}},
  year = {2025},
  doi = {10.1101/2025.05.08.25327250},
  howpublished = {medRxiv},
}
Figure