The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification

Linhao Qu, Xiaoyuan Luo, Kexue Fu, Manning Wang, Zhijian Song

Abstract

This paper introduces the novel concept of few-shot weakly supervised learning for pathology Whole Slide Image (WSI) classification, denoted as FSWC. A solution is proposed based on prompt learning and the utilization of a large language model, GPT-4. Since a WSI is too large and needs to be divided into patches for processing, WSI classification is commonly approached as a Multiple Instance Learning (MIL) problem. In this context, each WSI is considered a bag, and the obtained patches are treated as instances. The objective of FSWC is to classify both bags and instances with only a limited number of labeled bags. Unlike conventional few-shot learning problems, FSWC poses additional challenges due to its weak bag labels within the MIL framework. Drawing inspiration from the recent achievements of vision-language models (V-L models) in downstream few-shot classification tasks, we propose a two-level prompt learning MIL framework tailored for pathology, incorporating language prior knowledge. Specifically, we leverage CLIP to extract instance features for each patch, and introduce a prompt-guided pooling strategy to aggregate these instance features into a bag feature. Subsequently, we employ a small number of labeled bags to facilitate few-shot prompt learning based on the bag features. Our approach incorporates the utilization of GPT-4 in a question-and-answer mode to obtain language prior knowledge at both the instance and bag levels, which are then integrated into the instance and bag level language prompts. Additionally, a learnable component of the language prompts is trained using the available few-shot labeled data. We conduct extensive experiments on three real WSI datasets encompassing breast cancer, lung cancer, and cervical cancer, demonstrating the notable performance of the proposed method in bag and instance classification.

Introduction

In clinical pathology, Whole Slide Images (WSIs) are gigapixel-scale digital files that present a significant computational challenge. Due to their immense size, they are typically processed using Multiple Instance Learning (MIL), where a slide is treated as a "bag" containing thousands of small image "patches" or "instances."

Traditional MIL approaches are notoriously data-hungry, often requiring thousands of labeled slides to achieve clinical-grade accuracy. Furthermore, they suffer from weak supervision: while a diagnosis may be known for the entire slide, the specific patches containing the pathology are often unknown. This creates a bottleneck for rare diseases or emerging pathogens where data is scarce. This paper addresses Few-Shot Weakly-supervised WSI Classification (FSWC), proposing a framework that enables accurate diagnosis and patch identification with as few as one to sixteen labeled examples.

Methodology: The TOP Framework

The proposed TwO-level Prompt Learning (TOP) framework shifts away from traditional "black box" image processing toward a language-guided approach. By bridging the gap between Large Language Models (LLMs) and Vision-Language models (like CLIP), TOP allows AI to "read" pathology slides by combining visual patterns with descriptive medical knowledge.

Two-Level Prompting and GPT-4 Priors

TOP utilizes prompts at two distinct scales: the Instance level (identifying specific tissue types like glandular epithelium) and the Bag level (identifying the overall disease, such as Lung Adenocarcinoma). Rather than relying on simple manual prompts, the framework uses GPT-4 to generate detailed visual descriptions of pathological features. These descriptions serve as "language priors" that guide the vision model.

The TOP framework architecture, detailing the integration of GPT-4 generated medical descriptions into bag-level and instance-level prompt groups to guide feature extraction and pooling.

Prompt-Guided Pooling

A critical innovation in TOP is the prompt-guided pooling strategy. Standard MIL models often use simple averaging or max-pooling, which can be easily skewed by irrelevant patches. TOP uses instance-level text descriptions to query the patches, calculating similarity weights between patch features and text-based "prototypes." This allows the model to selectively attend to patches that align with established medical descriptions, effectively filtering noise from the signal.

Experimental Evaluation

The researchers evaluated TOP on three real-world WSI datasets: Camelyon 16 (breast cancer), TCGA-Lung Cancer, and an in-house Cervical Cancer dataset. The framework was tested across various "shot" counts (1, 2, 4, 8, and 16 labeled bags).

Performance and Robustness

TOP consistently outperformed state-of-the-art baselines, including Linear Probing and CoOp. On the Camelyon 16 dataset, TOP achieved a significant 0.8301 AUC in the 16-shot setting. Notably, the model demonstrated remarkable robustness in extreme low-data scenarios; in the 1-shot setting, TOP maintained an AUC of ~0.67, whereas baseline methods often approached random guessing.

Performance comparison on the Camelyon 16 dataset, showing TOP's superior AUC across all few-shot settings compared to standard pooling and prompt learning methods.

Across the Lung and Cervical cancer datasets, TOP showed an average improvement of 6.3% over the next best method, confirming its ability to generalize across different pathological domains.

Discussion and Conclusion

The success of TOP highlights the potential of using medical literature as a bridge for few-shot learning. By converting existing knowledge into visual descriptions via LLMs, models can be "pre-informed" about what to look for before seeing a single training image.

Future work may involve utilizing vision-language backbones specifically pre-trained on medical imagery rather than general internet data. Additionally, while prompt-guided pooling offers some interpretability, further research is required to ensure these models align perfectly with human diagnostic reasoning. Ultimately, TOP represents a significant step toward deploying AI in data-scarce clinical environments.

Summary updated

Abstract

This paper introduces the novel concept of few-shot weakly supervised learning for pathology Whole Slide Image (WSI) classification, denoted as FSWC. A solution is proposed based on prompt learning and the utilization of a large language model, GPT-4. Since a WSI is too large and needs to be divided into patches for processing, WSI classification is commonly approached as a Multiple Instance Learning (MIL) problem. In this context, each WSI is considered a bag, and the obtained patches are treated as instances. The objective of FSWC is to classify both bags and instances with only a limited number of labeled bags. Unlike conventional few-shot learning problems, FSWC poses additional challenges due to its weak bag labels within the MIL framework. Drawing inspiration from the recent achievements of vision-language models (V-L models) in downstream few-shot classification tasks, we propose a two-level prompt learning MIL framework tailored for pathology, incorporating language prior knowledge. Specifically, we leverage CLIP to extract instance features for each patch, and introduce a prompt-guided pooling strategy to aggregate these instance features into a bag feature. Subsequently, we employ a small number of labeled bags to facilitate few-shot prompt learning based on the bag features. Our approach incorporates the utilization of GPT-4 in a question-and-answer mode to obtain language prior knowledge at both the instance and bag levels, which are then integrated into the instance and bag level language prompts. Additionally, a learnable component of the language prompts is trained using the available few-shot labeled data. We conduct extensive experiments on three real WSI datasets encompassing breast cancer, lung cancer, and cervical cancer, demonstrating the notable performance of the proposed method in bag and instance classification. All codes will be made publicly accessible.
BibTeX
@inproceedings{qu2023rise,
  author = {Linhao Qu and Xiaoyuan Luo and Kexue Fu and Manning Wang and Zhijian Song},
  title = {{The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=mSDfBXr8Py},
}
Figure