TL;DR
This paper benchmarks four leading pathology-specific foundation models to understand how best to adapt them to diverse downstream tasks. They find that Parameter-Efficient Fine-Tuning (PEFT) is a robust and efficient strategy for consistency across tasks, while few-shot learning methods like Baseline/Baseline++ are crucial for adapting models to new tasks with limited data.
What Problem Does This Paper Address?
Computational pathology is rapidly adopting foundation models (FMs) for tasks like image classification and segmentation. However, adapting these powerful, pre-trained models to the specific needs of different downstream tasks and datasets remains a significant challenge. This is especially true when dealing with data from various sources, varying acquisition conditions, and limited labeled data, which are common in real-world clinical settings. The authors aim to bridge this gap by systematically evaluating how different adaptation strategies (fine-tuning and few-shot learning) perform across a range of pathology-specific FMs and diverse datasets, providing practical guidance for their deployment.
Methods
The study employs a two-pronged approach:
- Consistency Assessment Scenario: This scenario evaluates how well FMs adapt to different datasets within the same downstream task. They test five fine-tuning methods:
- Fully Supervised Learning: Training from scratch (baseline).
- Linear Probing: Freezing FM weights and training only the final classifier.
- Full Fine-tuning: Updating all FM weights.
- Partial Fine-tuning: Updating weights in the bottom 50% of layers.
- Parameter-Efficient Fine-Tuning (PEFT): Specifically using LoRA (Low-Rank Adaptation) to inject trainable low-rank matrices into self-attention layers, significantly reducing trainable parameters.
- Flexibility Assessment Scenario: This scenario assesses the FMs' ability to adapt to new tasks with limited data (few-shot learning). They explore three adaptation tasks (near-domain, middle-domain, out-domain) and evaluate five few-shot learning (FSL) methods:
- KNN (K-Nearest Neighbors): A standard baseline.
- MatchingNet: Learns a similarity metric between support and query sets.
- ProtoNet: Computes class prototypes (means) from support sets.
- Baseline: Fine-tunes the final linear layer during testing.
- Baseline++: An improved version of Baseline, generating class prototypes.
Datasets
The study utilizes a comprehensive set of 14 publicly accessible pathology datasets from five organs: breast, prostate, colon, gastric, and lung. These datasets exhibit significant variability in acquisition environments, digital scanners, and color distribution, leading to domain shifts.
- For Consistency Assessment: Datasets were used for four tasks: Breast Cancer Detection (PCam, BRACS, BACH), Colorectal Cancer Detection (KBSMC Colon I, KBSMC Colon II, DigestPath, Chaoyang), Colorectal Sub-Typing (Kather19, Kather16, CRC-TP), and Prostate Cancer Grading (PANDA, UBC, AGGC22).
- For Flexibility Assessment:
- Meta-train/validation: CRC-TP and Kather19 (colorectal tissues).
- Meta-test: KBSMC Colon, LC25000 (lung/colon), PANDA (prostate), KBSMC Gastric (gastric), and BACH (breast).
Key Insights and Ideas
- PEFT is King for Consistency: Parameter-Efficient Fine-Tuning (PEFT) consistently outperformed other fine-tuning methods across various datasets and tasks in the consistency assessment. This highlights its efficiency and effectiveness in adapting FMs without sacrificing performance, making it ideal for resource-constrained environments.
- UNI's Size Advantage (and Drawback): The UNI model, with its larger size and extensive pre-training, showed the most stable performance in the consistency assessment. However, this comes at the cost of significantly higher computational resource requirements.
- FSL for Data Scarcity: In data-limited scenarios (flexibility assessment), few-shot learning methods are essential. Baseline and Baseline++ emerged as top performers, demonstrating the value of specialized FSL techniques for adapting FMs to new tasks with minimal examples.
- KNN's Niche: While Baseline/Baseline++ generally excelled with more shots, KNN sometimes outperformed them in the 1-shot out-domain scenario, suggesting that for extremely limited data, the inherent knowledge of the FM might be sufficient, and simpler methods can be competitive.
- Adaptation Strategy Matters: The choice of adaptation strategy (fine-tuning vs. FSL) and the specific method within each category significantly impacts performance, underscoring the need for careful selection based on the task and data availability.
Key Results
- Consistency Assessment: PEFT achieved the highest win rates across datasets for most models, outperforming fully supervised learning by 100%, full fine-tuning by 85%, and partial fine-tuning by 75%. UNI showed the best overall performance among the foundation models.
- Breast Cancer Detection: PEFT consistently improved Accuracy (Acc) and F1 scores across all models and datasets, with improvements ranging from 1.88% to 37.27% in Acc.
- Colorectal Cancer Detection: PEFT also showed significant improvements, especially on the DigestPath dataset, with F1 gains up to 0.324.
- Prostate Cancer Grading: Partial fine-tuning showed strong performance on the PANDA dataset, while PEFT and linear probing were competitive on out-of-domain datasets. Lunit with PEFT achieved the best Acc and F1 simultaneously for this task.
- Flexibility Assessment:
- Near-Domain: Baseline generally achieved the best results, with Phikon and UNI showing competitive performance.
- Middle-Domain: Baseline++ was generally superior, though KNN and Baseline also performed well.
- Out-Domain (Prostate, Gastric, Breast): Baseline and Baseline++ consistently performed well, especially with more shots. UNI paired with Baseline++ achieved top results in several out-domain scenarios.
- Computational Complexity: PEFT added minimal parameters (~1%) but required more memory than linear probing. Full fine-tuning was the most computationally expensive. For FSL, Baseline and Baseline++ were significantly slower during testing due to their iterative nature, while KNN, ProtoNet, and MatchingNet were much faster.
Limitations and Future Work
- Patch-Level Focus: The study is limited to patch-level pathology image classification. Future work should expand to whole slide-level classification, cell segmentation, and other tasks.
- Limited Methods: While five fine-tuning and five FSL methods were explored, further investigation into optimizing specific methods for particular datasets/tasks is needed.
- Dataset Relationships: The flexibility assessment used colorectal tissues for meta-training/validation and a mix of organs for meta-testing. The influence of these relationships on performance warrants further study.
- Single Dataset per Adaptation Task: The flexibility assessment used a single dataset for each adaptation task. An extended validation study with multiple external datasets is recommended.
- Future Directions: The authors suggest developing more advanced fine-tuning and FSL methods tailored to pathology FMs and optimizing them for performance and computational complexity. They also note the growing size of FMs and their impact on clinical deployment.
Summary updated