The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification

Linhao Qu, Xiaoyuan Luo, Kexue Fu, Manning Wang, Zhijian Song

TL;DR

This paper introduces TOP (Two-level Prompt Learning), a novel framework that tackles the challenging task of classifying gigapixel pathology images with very few labeled examples. By leveraging the medical knowledge of GPT-4 to guide a vision-language model (CLIP), TOP achieves state-of-the-art classification at both the whole-slide and individual-patch levels using as few as one to sixteen training samples.

What Problem Does This Paper Address?

In digital pathology, doctors work with Whole Slide Images (WSIs)—gigapixel-resolution scans of tissue samples. Because these images are too massive to feed directly into deep learning models, researchers typically chop them into thousands of smaller patches.

Standard practice treats WSI classification as a Multiple Instance Learning (MIL) problem: the entire slide is a "bag," and the patches are "instances." Usually, only the slide has a label (e.g., "cancerous"), while the individual patches remain unlabeled.

Figure 1: A. Existing WSI classification tasks. B. Our proposed FSWC task based on language models. C. Existing prompt learning paradigm using pre-trained V-L models, where the parameters of the V-L models are always frozen. D. Our proposed Two-level Prompt Learning paradigm.

While MIL works well with massive datasets, clinical reality often presents a data scarcity problem due to rare diseases or privacy concerns. The authors define a new, highly challenging task: Few-shot Weakly-supervised WSI Classification (FSWC). The goal is to accurately classify both the entire slide and the individual patches using only a handful of labeled slides (e.g., 1, 2, 4, 8, or 16 slides per class) and zero patch-level labels.

Methods

To solve the FSWC problem, the authors propose TOP (TwO-level Prompt Learning), which integrates the visual feature extraction of CLIP with the rich medical knowledge of GPT-4.

The architecture of the proposed TOP framework, showing instance-level and bag-level prompt learning streams.

The framework operates on two distinct levels:

  1. Instance-Level (Patch) Guidance: Instead of using standard mathematical pooling (like max or mean pooling) to combine patch features into a single slide feature, TOP introduces prompt-guided pooling. The authors ask GPT-4 to describe the visual characteristics of various tissue phenotypes (e.g., squamous epithelium, lymphocytes). These descriptions form "instance prototypes." The model calculates the similarity between each patch and these prototypes, using the scores to intelligently weight and aggregate the patches into a single "bag feature."
  2. Bag-Level (Slide) Classification: At the slide level, GPT-4 is queried to provide detailed visual descriptions of specific diseases (e.g., Lung Adenocarcinoma vs. Lung Squamous Cell Carcinoma). These descriptions are combined with learnable prompt tokens (similar to CoOp) to form a "bag-level prompt group." The aggregated bag feature is then matched against these prompts to make the final slide classification.

To prevent the learned instance prototypes from blending together and becoming identical during training, the authors introduce a unique auxiliary diversity loss that forces the prototypes to remain distinct.

Datasets

The authors evaluated their method on three real-world WSI datasets representing different cancer types and medical centers:

  • Camelyon 16: A breast cancer dataset containing true labels for both slides and individual patches (used to verify patch-level accuracy).
  • TCGA-Lung Cancer: A public dataset for differentiating lung cancer subtypes.
  • Cervical Cancer Dataset: An in-house dataset used to evaluate the model on challenging cervical cancer classifications.

Key Insights and Ideas

  • LLMs as Clinical Priors: Instead of relying purely on data-driven features, TOP uses GPT-4 as an automated, highly detailed medical textbook. This injects expert-level domain knowledge into the vision model without requiring tedious manual prompt engineering by human pathologists.
  • Two-Level Alignment is Crucial: Simply applying natural-image few-shot techniques (like CoOp) directly to the slide level yields poor results. Aligning language prompts at both the patch level (for aggregation) and the slide level (for classification) is what unlocks strong few-shot performance.
  • Prototype Diversity: The inclusion of an auxiliary loss to keep instance prototypes distinct ensures that the model captures a wide variety of tissue structures across the slide, rather than hyper-focusing on just one visual feature.

Key Results

The TOP framework consistently outperformed all baseline methods (including Linear-Probe and standard CoOp) across all three datasets, often by a wide margin.

  • On Camelyon 16 (Breast Cancer): In a 16-shot setting, TOP achieved a slide-level Area Under the Curve (AUC) of 0.8301 compared to CoOp's 0.8012. More impressively, its patch-level classification AUC reached 0.8896, significantly outperforming other methods at identifying exact cancerous regions.
  • On TCGA-Lung Cancer: TOP achieved a 16-shot slide AUC of 0.8235 (outperforming CoOp at 0.7840) and maintained a high AUC of 0.7123 even in the extreme 1-shot scenario.
  • On Cervical Cancer: On this highly challenging dataset, TOP demonstrated an average improvement of 6.3% over the second-best method across all shot configurations.

Limitations and Future Work

The primary limitation of TOP is its dependency on the quality of the large language model. If the LLM generates inaccurate or vague visual descriptions for a specific rare disease, the quality of both the prompt-guided pooling and the final classification will degrade.

Moving forward, the authors suggest exploring the integration of TOP with emerging, domain-specific pathology foundation models rather than general-domain vision-language models like standard CLIP.

Summary updated

Abstract

This paper introduces the novel concept of few-shot weakly supervised learning for pathology Whole Slide Image (WSI) classification, denoted as FSWC. A solution is proposed based on prompt learning and the utilization of a large language model, GPT-4. Since a WSI is too large and needs to be divided into patches for processing, WSI classification is commonly approached as a Multiple Instance Learning (MIL) problem. In this context, each WSI is considered a bag, and the obtained patches are treated as instances. The objective of FSWC is to classify both bags and instances with only a limited number of labeled bags. Unlike conventional few-shot learning problems, FSWC poses additional challenges due to its weak bag labels within the MIL framework. Drawing inspiration from the recent achievements of vision-language models (V-L models) in downstream few-shot classification tasks, we propose a two-level prompt learning MIL framework tailored for pathology, incorporating language prior knowledge. Specifically, we leverage CLIP to extract instance features for each patch, and introduce a prompt-guided pooling strategy to aggregate these instance features into a bag feature. Subsequently, we employ a small number of labeled bags to facilitate few-shot prompt learning based on the bag features. Our approach incorporates the utilization of GPT-4 in a question-and-answer mode to obtain language prior knowledge at both the instance and bag levels, which are then integrated into the instance and bag level language prompts. Additionally, a learnable component of the language prompts is trained using the available few-shot labeled data. We conduct extensive experiments on three real WSI datasets encompassing breast cancer, lung cancer, and cervical cancer, demonstrating the notable performance of the proposed method in bag and instance classification. All codes will be available.
BibTeX
@inproceedings{qu2023rise,
  author = {Linhao Qu and Xiaoyuan Luo and Kexue Fu and Manning Wang and Zhijian Song},
  title = {{The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=mSDfBXr8Py},
}
Figure