FAST: A Dual-tier Few-Shot Learning Paradigm for Whole Slide Image Classification

Kexue Fu, Xiaoyuan Luo, Linhao Qu, Shuo Wang, Ying Xiong, Ilias Maglogiannis, Longxiang Gao, Manning Wang

Abstract

The expensive fine-grained annotation and data scarcity have become the primary obstacles for the widespread adoption of deep learning-based Whole Slide Images (WSI) classification algorithms in clinical practice. Unlike few-shot learning methods in natural images that can leverage the labels of each image, existing few-shot WSI classification methods only utilize a small number of fine-grained labels or weakly supervised slide labels for training in order to avoid expensive fine-grained annotation. They lack sufficient mining of available WSIs, severely limiting WSI classification performance. To address the above issues, we propose a novel and efficient dual-tier few-shot learning paradigm for WSI classification, named FAST. FAST consists of a dual-level annotation strategy and a dual-branch classification framework. Firstly, to avoid expensive fine-grained annotation, we collect a very small number of WSIs at the slide level, and annotate an extremely small number of patches. Then, to fully mining the available WSIs, we use all the patches and available patch labels to build a cache branch, which utilizes the labeled patches to learn the labels of unlabeled patches and through knowledge retrieval for patch classification. In addition to the cache branch, we also construct a prior branch that includes learnable prompt vectors, using the text encoder of visual-language models for patch classification. Finally, we integrate the results from both branches to achieve WSI classification. Extensive experiments on binary and multi-class datasets demonstrate that our proposed method significantly surpasses existing few-shot classification methods and approaches the accuracy of fully supervised methods with only 0.22% annotation costs.

Introduction

In computational pathology, Whole Slide Images (WSIs) are gigapixel-scale digital files containing tens of thousands of individual patches. Training deep learning models for WSI classification typically faces a "dual obstacle": fine-grained annotation is prohibitively expensive, and clinical data is often scarce. Traditional approaches rely either on full supervision, requiring experts to annotate thousands of patches, or weak supervision (Multiple Instance Learning), which requires massive amounts of slide-level data to identify relevant features. Existing few-shot methods often fail to fully exploit the information within available WSIs, either ignoring unlabeled patches or relying on generic features that lack domain specificity.

The FAST Paradigm

The proposed FAST (Few-shot Annotation-efficient Slide Task) framework introduces a dual-tier approach to both data annotation and model architecture to overcome these limitations.

Dual-Level Annotation Strategy

FAST optimizes the labeling process by combining slide-level (bag) and patch-level (instance) information. Instead of labeling every patch or just the slide, FAST selects a few slides and labels an extremely small number of patches within them. This provides a strong supervisory signal without the massive time investment required for full annotation.

Comparison of few-shot learning paradigms: (a) Instance-level, (b) Bag-level, and (c) the proposed FAST dual-tier approach.

Dual-Branch Classification Framework

The model processes patches through two parallel paths to maximize information extraction from limited data:

  1. Cache Branch: This branch utilizes a frozen CLIP image encoder to extract features. It maintains a cache where labels for unlabeled patches are treated as learnable parameters. Through a knowledge retrieval mechanism, the model uses the few labeled patches to "teach" the unlabeled ones, effectively mining the structural context of the entire slide.
  2. Prior Knowledge Branch: To provide a domain-specific head start, FAST uses GPT-4V to generate descriptive text prompts for specific pathologies. These are processed by CLIP’s text encoder. Learnable prompt vectors (prompt tuning) are used to refine these descriptions, ensuring they align closely with the visual features of the pathology slides.
The FAST framework architecture featuring the Cache Branch for mining unlabeled patches and the Prior Branch utilizing GPT-4V guided knowledge.

Experimental Results

FAST was evaluated on two major benchmarks: CAMELYON16 (breast cancer metastasis) and TCGA-RENAL (renal cancer subtypes). The results demonstrate that FAST consistently outperforms state-of-the-art few-shot methods like Tip-Adapter, particularly in extreme scenarios with only 1 or 2 slides.

On the CAMELYON16 dataset, using only 16 slides and 16 labeled patches per slide, FAST achieved an instance-level AUC of 0.9151, approaching the 0.9532 achieved by a fully supervised model. Remarkably, this performance is attained at only 0.22% of the annotation cost compared to full supervision.

Conclusion

FAST demonstrates that unlabeled data in few-shot scenarios is a valuable resource when treated as structural context. By integrating learnable image caches with vision-language priors guided by LLMs, FAST provides a highly efficient path toward clinical deployment of WSI classification. Future work will focus on more sophisticated spatial aggregation methods to better capture tissue architecture beyond simple pooling.

Summary updated

Abstract

The expensive fine-grained annotation and data scarcity have become the primary obstacles for the widespread adoption of deep learning-based Whole Slide Images (WSI) classification algorithms in clinical practice. Unlike few-shot learning methods in natural images that can leverage the labels of each image, existing few-shot WSI classification methods only utilize a small number of fine-grained labels or weakly supervised slide labels for training in order to avoid expensive fine-grained annotation. They lack sufficient mining of available WSIs, severely limiting WSI classification performance. To address the above issues, we propose a novel and efficient dual-tier few-shot learning paradigm for WSI classification, named FAST. FAST consists of a dual-level annotation strategy and a dual-branch classification framework. Firstly, to avoid expensive fine-grained annotation, we collect a very small number of WSIs at the slide level, and annotate an extremely small number of patches. Then, to fully mining the available WSIs, we use all the patches and available patch labels to build a cache branch, which utilizes the labeled patches to learn the labels of unlabeled patches and through knowledge retrieval for patch classification. In addition to the cache branch, we also construct a prior branch that includes learnable prompt vectors, using the text encoder of visual-language models for patch classification. Finally, we integrate the results from both branches to achieve WSI classification. Extensive experiments on binary and multi-class datasets demonstrate that our proposed method significantly surpasses existing few-shot classification methods and approaches the accuracy of fully supervised methods with only 0.22$\%$ annotation costs. All codes and models will be publicly available on https://github.com/fukexue/FAST.
BibTeX
@inproceedings{fu2024fast,
  author = {Kexue Fu and Xiaoyuan Luo and Linhao Qu and Shuo Wang and Ying Xiong and Ilias Maglogiannis and Longxiang Gao and Manning Wang},
  title = {{FAST: A Dual-tier Few-Shot Learning Paradigm for Whole Slide Image Classification}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=9vcqleAHPl},
}
Figure