Benchmarking Pathology Foundation Models: Adaptation Strategies and Scenarios

Jeaung Lee, Jeewoo Lim, Keunho Byeon, Jin Tae KwakComputers in Biology and Medicine 2024 · DOI · OpenReview

TL;DR

This paper benchmarks four leading pathology-specific foundation models to understand how best to adapt them to diverse downstream tasks. They find that Parameter-Efficient Fine-Tuning (PEFT) is a robust and efficient strategy for consistency across tasks, while few-shot learning methods like Baseline/Baseline++ are crucial for adapting models to new tasks with limited data.

What Problem Does This Paper Address?

Computational pathology is rapidly adopting foundation models (FMs) for tasks like image classification and segmentation. However, adapting these powerful, pre-trained models to the specific needs of different downstream tasks and datasets remains a significant challenge. This is especially true when dealing with data from various sources, varying acquisition conditions, and limited labeled data, which are common in real-world clinical settings. The authors aim to bridge this gap by systematically evaluating how different adaptation strategies (fine-tuning and few-shot learning) perform across a range of pathology-specific FMs and diverse datasets, providing practical guidance for their deployment.

Methods

The study employs a two-pronged approach:

  1. Consistency Assessment Scenario: This scenario evaluates how well FMs adapt to different datasets within the same downstream task. They test five fine-tuning methods:
  2. Fully Supervised Learning: Training from scratch (baseline).
  3. Linear Probing: Freezing FM weights and training only the final classifier.
  4. Full Fine-tuning: Updating all FM weights.
  5. Partial Fine-tuning: Updating weights in the bottom 50% of layers.
  6. Parameter-Efficient Fine-Tuning (PEFT): Specifically using LoRA (Low-Rank Adaptation) to inject trainable low-rank matrices into self-attention layers, significantly reducing trainable parameters.
  7. Flexibility Assessment Scenario: This scenario assesses the FMs' ability to adapt to new tasks with limited data (few-shot learning). They explore three adaptation tasks (near-domain, middle-domain, out-domain) and evaluate five few-shot learning (FSL) methods:
  8. KNN (K-Nearest Neighbors): A standard baseline.
  9. MatchingNet: Learns a similarity metric between support and query sets.
  10. ProtoNet: Computes class prototypes (means) from support sets.
  11. Baseline: Fine-tunes the final linear layer during testing.
  12. Baseline++: An improved version of Baseline, generating class prototypes.

Datasets

The study utilizes a comprehensive set of 14 publicly accessible pathology datasets from five organs: breast, prostate, colon, gastric, and lung. These datasets exhibit significant variability in acquisition environments, digital scanners, and color distribution, leading to domain shifts.

  • For Consistency Assessment: Datasets were used for four tasks: Breast Cancer Detection (PCam, BRACS, BACH), Colorectal Cancer Detection (KBSMC Colon I, KBSMC Colon II, DigestPath, Chaoyang), Colorectal Sub-Typing (Kather19, Kather16, CRC-TP), and Prostate Cancer Grading (PANDA, UBC, AGGC22).
  • For Flexibility Assessment:
    • Meta-train/validation: CRC-TP and Kather19 (colorectal tissues).
  • Meta-test: KBSMC Colon, LC25000 (lung/colon), PANDA (prostate), KBSMC Gastric (gastric), and BACH (breast).

Key Insights and Ideas

  • PEFT is King for Consistency: Parameter-Efficient Fine-Tuning (PEFT) consistently outperformed other fine-tuning methods across various datasets and tasks in the consistency assessment. This highlights its efficiency and effectiveness in adapting FMs without sacrificing performance, making it ideal for resource-constrained environments.
  • UNI's Size Advantage (and Drawback): The UNI model, with its larger size and extensive pre-training, showed the most stable performance in the consistency assessment. However, this comes at the cost of significantly higher computational resource requirements.
  • FSL for Data Scarcity: In data-limited scenarios (flexibility assessment), few-shot learning methods are essential. Baseline and Baseline++ emerged as top performers, demonstrating the value of specialized FSL techniques for adapting FMs to new tasks with minimal examples.
  • KNN's Niche: While Baseline/Baseline++ generally excelled with more shots, KNN sometimes outperformed them in the 1-shot out-domain scenario, suggesting that for extremely limited data, the inherent knowledge of the FM might be sufficient, and simpler methods can be competitive.
  • Adaptation Strategy Matters: The choice of adaptation strategy (fine-tuning vs. FSL) and the specific method within each category significantly impacts performance, underscoring the need for careful selection based on the task and data availability.

Key Results

  • Consistency Assessment: PEFT achieved the highest win rates across datasets for most models, outperforming fully supervised learning by 100%, full fine-tuning by 85%, and partial fine-tuning by 75%. UNI showed the best overall performance among the foundation models.
  • Breast Cancer Detection: PEFT consistently improved Accuracy (Acc) and F1 scores across all models and datasets, with improvements ranging from 1.88% to 37.27% in Acc.
  • Colorectal Cancer Detection: PEFT also showed significant improvements, especially on the DigestPath dataset, with F1 gains up to 0.324.
  • Prostate Cancer Grading: Partial fine-tuning showed strong performance on the PANDA dataset, while PEFT and linear probing were competitive on out-of-domain datasets. Lunit with PEFT achieved the best Acc and F1 simultaneously for this task.
  • Flexibility Assessment:
    • Near-Domain: Baseline generally achieved the best results, with Phikon and UNI showing competitive performance.
  • Middle-Domain: Baseline++ was generally superior, though KNN and Baseline also performed well.
  • Out-Domain (Prostate, Gastric, Breast): Baseline and Baseline++ consistently performed well, especially with more shots. UNI paired with Baseline++ achieved top results in several out-domain scenarios.
  • Computational Complexity: PEFT added minimal parameters (~1%) but required more memory than linear probing. Full fine-tuning was the most computationally expensive. For FSL, Baseline and Baseline++ were significantly slower during testing due to their iterative nature, while KNN, ProtoNet, and MatchingNet were much faster.

Limitations and Future Work

  • Patch-Level Focus: The study is limited to patch-level pathology image classification. Future work should expand to whole slide-level classification, cell segmentation, and other tasks.
  • Limited Methods: While five fine-tuning and five FSL methods were explored, further investigation into optimizing specific methods for particular datasets/tasks is needed.
  • Dataset Relationships: The flexibility assessment used colorectal tissues for meta-training/validation and a mix of organs for meta-testing. The influence of these relationships on performance warrants further study.
  • Single Dataset per Adaptation Task: The flexibility assessment used a single dataset for each adaptation task. An extended validation study with multiple external datasets is recommended.
  • Future Directions: The authors suggest developing more advanced fine-tuning and FSL methods tailored to pathology FMs and optimizing them for performance and computational complexity. They also note the growing size of FMs and their impact on clinical deployment.

Summary updated

Abstract

In computational pathology, several foundation models have recently emerged and demonstrated enhanced learning capability for analyzing pathology images. However, adapting these models to various downstream tasks remains challenging, particularly when faced with datasets from different sources and acquisition conditions, as well as limited data availability. In this study, we benchmark four pathology-specific foundation models across 14 datasets and two scenarios-consistency assessment and flexibility assessment-addressing diverse adaptation scenarios and downstream tasks. In the consistency assessment scenario, involving five fine-tuning methods, we found that the parameter-efficient fine-tuning approach was both efficient and effective for adapting pathology-specific foundation models to diverse datasets within the same downstream task. In the flexibility assessment scenario under data-limited environments, utilizing five few-shot learning methods, we observed that the foundation models benefited more from the few-shot learning methods that involve modification during the testing phase only. These findings provide insights that could guide the deployment of pathology-specific foundation models in real clinical settings, potentially improving the accuracy and reliability of pathology image analysis. The code for this study is available at: https://github.com/QuIIL/BenchmarkingPathologyFoundationModels.
BibTeX
@article{lee2024benchmarking,
  author = {Jeaung Lee and Jeewoo Lim and Keunho Byeon and Jin Tae Kwak},
  title = {{Benchmarking Pathology Foundation Models: Adaptation Strategies and Scenarios}},
  journal = {Computers in Biology and Medicine},
  year = {2024},
  doi = {10.1016/j.compbiomed.2025.110031},
}
Figure